Hello,
Is it possible to use the IPFS HTTP API add a directory?
The fact that there is a recursive flag leads me to think it is possible, but I cannot figure out how.
My simple curl attempt fails (where stuff is my directory):
curl "https://ipfs.infura.io:5001/api/v0/add?recursive=true" \
--verbose \
-X POST \
-H "Content-Type: multipart/form-data" \
-F 'file=@"stuff"'IPFS add function works for files and directories, check out the documentation here, https://infura.io/docs/ipfs/post/add
Yes, but I am having trouble with the actual way to do this.
Looking at the docs, I should be able to provide a directory to file parameter, like in my curl example above.
However, when I run the command above, I get:
curl: (43) A libcurl function was given a bad argument
(It works as expected with a file however.)
What am I doing wrong?
Copy and pasted and tried this on our side and it works, this appears to be a curl error with the parameters you are using.
It seems I was using the “default” version of curl that comes with MacOS (7.54.0).
The error I was seeing appears to be a known issue on some level.
After upgrading to the latest version of curl (7.63.0) via homebrew, I am able to successfully make the API call.
Thank you for the help!
Top!
I am trying to pin a directory and going through this command
curl "https://ipfs.infura.io:5001/api/v0/add?recursive=true" \
-X POST \
-H "Content-Type: multipart/form-data" \
-F file=@"address-book"
And I get this output:
{“Name”:“address-book”,“Hash”:“QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH”,“Size”:“6”}
https://ipfs.infura.io/ipfs/QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH
It looks like you’re trying to upload a directory using Curl. Is there a reason you aren’t using the ipfs command-line tool for this? I don’t think curl will work for you without you manually crafting the multi-part upload of the directory yourself. These people seem to have run into your problem too: https://ethereum.stackexchange.com/questions/60260/ipfs-add-a-directory-recursively-with-an-http-api
I wanted to “add” to IPFS during the deploy step of a build (without a running IPFS node). This is the best I’ve been able to put together so far. It’s not pretty, but will upload a small directory. It also doesn’t work with filenames/dirs that have spaces. I’ll update this post if I am able to improve it. Hopefully it helps someone else out, or someone can improve upon it.
cd /your/chosen/path
FILES=$(find * -type f | grep -v ' ' | sed -e 's/ /\\ /g' | awk -v q="'" '{print " -F " q "file=@\"" $0 "\";filename=\"" $0 "\"" q}')
curl "https://ipfs.infura.io:5001/api/v0/add?pin=true&recursive=true&wrap-with-directory=true" -X POST -H "Content-Type: multipart/form-data" $FILES
Having been using this - I can also see that for an unknown reason there is a limit to the number of uploads. I’ve yet to debug this, but if anyone has any info on this - I would very much appreciate it!
Here is a log to the last few builds of my personal site:
https://api.travis-ci.org/v3/job/644132678/log.txt - here the uploads stopped at a festival article.
https://api.travis-ci.org/v3/job/644246234/log.txt - whereas this got stopped even earlier and very short of adding all files.
Following my last post, if all files don’t add (usually the case), sometimes running the exact same command again seems to get further down the list of files to add.
Could anyone at Infura please confirm if this is currently expected behaviour during your free phase?
Here is the curl output from 50 attempts: https://api.travis-ci.org/v3/job/644637963/log.txt
Hi @craigmayhew, how big was the original upload and subsequent uploads? I suspect it worked through a number of the files up to the 100mb limit and then didn’t upload the successful files in the following attempts until it had worked through all the files.
Hi @mike, the total size is 43MB if it were to add everything first time. However it looks to be managing at most a few MBs per attempt.
$ curl "https://ipfs.infura.io:5001/api/v0/add?pin=true&recursive=true&wrap-with-directory=true" -vv -X POST -H "Content-Type: multipart/form-data" $FILES
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 64:ff9b::36ae:5bac...
* TCP_NODELAY set
* Trying 54.172.69.22...
* TCP_NODELAY set
* Connected to ipfs.infura.io (64:ff9b::36ae:5bac) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.infura.io
* start date: Apr 25 00:00:00 2019 GMT
* expire date: May 25 12:00:00 2020 GMT
* subjectAltName: host "ipfs.infura.io" matched cert's "*.infura.io"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffe4341580)
> POST /api/v0/add?pin=true&recursive=true&wrap-with-directory=true HTTP/2
> Host: ipfs.infura.io:5001
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 40411928
> Content-Type: multipart/form-data; boundary=------------------------91a9f6aea0847183
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Sat, 01 Feb 2020 14:54:14 GMT
< content-type: application/json
< content-length: 1460
< access-control-allow-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< access-control-expose-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< server: go-ipfs/0.4.22
< vary: Origin
< vary: Origin
< x-chunked-output: 1
<
{"Name":"404/index.html","Hash":"QmPJzfnqWoiXpJtdeYhQTnifx4YaQzLuXLdWcZzqr3qrHL","Size":"3903"}
{"Name":"articles/blockchain/index.html","Hash":"QmYmVr5d2AQLBTUqLQTKiTqTAEoLgWYegqF2do4aoQMk3E","Size":"10305"}
{"Name":"articles/copacity/index.html","Hash":"QmcUkN8mwWnRsLkD3jXymzxXT9mxemfmUcWtgri6wgTzBY","Size":"9193"}
{"Name":"articles/index.html","Hash":"QmS5A3TgrqszVrhjX9Govfh1W4JpPibhjHADxMMherwKsd","Size":"5882"}
{"Name":"articles/livelab/index.html","Hash":"QmRCkAdB8RKyAjTqMU2yo6fg5JYD6Z2vhdUY9Zv5BXrQfr","Size":"6476"}
{"Name":"articles/serverless/index.html","Hash":"QmdyapZx98NDZYMaqmgixJfEK8ZpVd2XxuGjgYHa4WUjFt","Size":"8198"}
{"Name":"blog/2560x1600-desktop-backgrounds/index.html","Hash":"QmeHay6NWXzv4tmhettaueN7UcCpVWWfLmP9AeCxnRBKs9","Size":"7114"}
{"Name":"blog/25th-birthday/index.html","Hash":"QmXuarxeDRBqejdqFU8wyhYD3uPew7cNtGF5Jih5nN3BUo","Size":"6773"}
{"Name":"blog/360-degree-movies/index.html","Hash":"QmSRTjcosrCC7XvhCs519HaYtiEEVXsBd2MER526ZzXcmk","Size":"6001"}
{"Name":"blog/9/index.html","Hash":"Qma4CuJBJ3k9AxFJPgQVo3w4rVsFFjSoSJtKu4cGwvQcqU","Size":"6102"}
{"Name":"blog/a-solution-to-over-priced-ebooks/index.html","Hash":"QmcmRzF1jvNKwBG7sZNYTBMXQ1KxVHETKRhxKy4di2yR4Q","Size":"7012"}
{"Name":"blog/apache-htpasswd-files-and-windows/index.html","Hash":"QmNgYToia2z8SWMbGvccabij1puifDWX3RuevcjbgEsFLn","Size":"5445"}
{"Name":"blog/archive/index.html","Hash":"QmTSYbp7Hy2Z7KDUY2L2QL4FjudadpNFF35bU13EHCyPQi","Size":"24298"}
* Connection #0 to host ipfs.infura.io left intact
My latest attempt is showing promise at working around the issue with the addition of -C - to the end of the command to allow it to start exactly where it left off each time the command is run.
curl "https://ipfs.infura.io:5001/api/v0/add?pin=true&recursive=true&wrap-with-directory=true" -vv -X POST -H "Content-Type: multipart/form-data" $FILES -C -
Thanks for the added info, will review this with the team and get back to you.
Thanks again @craigmayhew, we dug into this further, can you try this using the parameter progress = false in your request?
Hi @mike,
I’ve added &progress=false onto the url.
Latest log output here: https://travis-ci.org/craigmayhew/craig.mayhew.io/jobs/645691943#L1846
Curl command is here: https://github.com/craigmayhew/craig.mayhew.io/blob/master/.travis.yml#L85
Can I provide any more info? Happy to jump on screen sharing if that helps?
Hi @mike
I wondered if this is being worked on? If so, is there any more debug I can provide? If it’s not a priority, please let me know and I’ll move onto something else.