Uploading 10k files to IPFS

Hello

I have a simple Python script that uploads all the files under a directory to IPFS, wrapped in a directory.
Basically:

files = {“file1”: “file1contents”, “file2”: “file2contents”}
response = requests.post(
https://ipfs.infura.io:5001/api/v0/add?recursive=true&wrap-with-directory=true”,
auth=(“xxx”, “yyy”),
files=files
)

Output is for example:
{“Name”:“file1”,“Hash”:“QmZn…”,“Size”:“10”}
{“Name”:“file2”,“Hash”:“QmZn…”,“Size”:“10”}
{“Name”:"",“Hash”:“QmaG2T9wsrTsA7kH9K4HZLWXnEzpRyYjMDhesZTJfBoEgS”,“Size”:“20”}

So if you go to the hash of the last line, it will take you to a directory with all the files uploaded.
This works perfectly fine with a few hundred files. However, when I load it up with 10,000 files, the output is:

{“Name”:“file1”,“Hash”:“QmZn…”,“Size”:“10”}
{“Name”:“file2”,“Hash”:“QmZn…”,“Size”:“10”}

{“Name”:“file9999”,“Hash”:“QmZn…”,“Size”:“10”}
{“Name”:“file10000”,“Hash”:“QmZn…”,“Size”:“10”}

And DOES NOT SHOW the last line (“Name”: “”, the one with the directory CID).

Also, I do not see it in the explorer under my account …

Am I doing something wrong?

Thanks!

I’m having this exact issue using the ipfs-http-client node library.

I believe there’s some sort of folder size limitation in Infura’s side but it’d be great if they could confirm. Adding the files without a wrapping directory seems to work fine.

Mac.

Welcome and thank you for reaching out! For the second run, where you uploaded 10k files, can you share any one of those hashes? I want to try to see if a trace one hash to see if we can find the root directory.

Kind regards,
Alex | Infura | Consensys

Hi @MacP , We extended our write timeout for the /api/v0/add endpoint from 10 to 30 mins this week! You should be able to upload bigger directories now.