Takes too long time to add 10k json files

Questions
1

Hi, there.
I’m developing PFP NFT projects and concerning to use the Infura service to upload metadata JSON files to the IPFS.
In usual cases, 10k json files are batch uploaded with wrapping directory, so that URIs for metadatas can be linked with the same root hash.
After that we just update baseURI our smart contract to “ipfs://{root hash}” and uri will be {base_uri}/{token_id}.json, which is compatible with ERC1155 protocols.

For this, I write below codes, (nodejs)

const axios_config = {
method: “post”,
url: “https://ipfs.infura.io:5001/api/v0/add”,
params: {
“wrap-with-directory”: true,
“cid-version”: 1,
hash: “sha2-256”,
},
maxBodyLength: “Infinity”,
headers: {
authorization: auth,
…NFT_metadatas_form_data.getHeaders(),
“Content-Type”: “application/json”,

  accept: "application/json",
},
data: NFT_metadatas_form_data,

};
return await axios(axios_config)
.then((res) => Promise.resolve(res.data))
.catch((e) => Promise.reject(e));

, where NFT_metadatas_from_data is created using new FormData() with appending 10 k json metadatas.

It works, try to test with cid: bafybeihkqz4psu7i36ipwqv652oh35i7go76sa3z6ggztzwxpkwbyowbm4/{id}.json
, where id is hexadecimal number of 64 hex string from 0x0 to 0x270f (0 ~ 9999 decimal)

ex) https://bafybeihkqz4psu7i36ipwqv652oh35i7go76sa3z6ggztzwxpkwbyowbm4.ipfs.infura-ipfs.io/000000000000000000000000000000000000000000000000000000000000270f.json

But 2 problems I encounter.

  1. It takes 10 ~ 15 minutes to add all 10k metadata JSON files. I think it is related to the rate limit (10 req/sec) for adding API.
    Is there any better way to upload as fast as I can?

  2. When I query all uploaded files with root hash like https://bafybeihkqz4psu7i36ipwqv652oh35i7go76sa3z6ggztzwxpkwbyowbm4.ipfs.infura-ipfs.io
    gateway timeout error is responded. And I think it is due to too many blocks inside. How can I query all the files in the directory using a public gateway?

2

Hi @ddevkim

  1. There is nothing we can do at this point, since you want to have all of your 10k metadata JSON files under the same directory.

  2. Could you please let me know if you use an authorized request when querying the files? (project id + project secret)

3

Hi, Flaviu. Thanks for the reply.

How can I let you know the secret key?

I’m concerned it is used from anonymous.