IPFS Multi-File Add Bug

We are looking to upload a directory (e.g. a website) to IPFS using the ipfs-http-client library. We’re able to upload the files as a directory and, generally, view them correctly (say 100 files totaling about 10MB). Our issue is that, sometimes, the files are being lost or corrupted on upload. Specifically, if we upload files A, B, and C, it looks like file B bleeds into file A (and sometimes file C, too). Here’s a simple reproduction scheme:

First, generate some semi-large data into a few files (it doesn’t matter if this data is random, zero, ASCII, etc). In our case, we use a “build” directory and have filled it with 100 files f000 through f100, each with an escalating 1KB of random data (so the first file is 1KB, the second is 2KB, …). E.g. in bash:

mkdir build
for i in {001..100}; do head -c $(($i * 1000)) </dev/urandom > build/f$i; done

Then, run this JavaScript snippet with valid Infura IPFS credentials:

const IpfsHttpClient = require('ipfs-http-client');
let ipfs = IpfsHttpClient.create({
  host: 'ipfs.infura.io',
  port: 5001,
  protocol: 'https',
  headers: {
    authorization: process.env['IPFS_AUTH']
  }
});
function progress(size, path) {
  console.log("Sent " + Math.round(size / 1000) + "KB for " + path);
}
await ipfs.add(ipfsClient.globSource('build', { recursive: true }), { timeout: 600000, progress });

Note: IPFS_AUTH here is Basic with the key:secret already encoded into base64. Authorization or connectivity do not appear to be the issue and the cids do appear in our dashboard.

The result of this upload will look like this:

Sent 1KB for build/f001 # these are good
Sent 2KB for build/f002
Sent 3KB for build/f003
Sent 4KB for build/f004
Sent 5KB for build/f005
Sent 6KB for build/f006
Sent 7KB for build/f007
Sent 8KB for build/f008
Sent 9KB for build/f009
Sent 10KB for build/f010
Sent 11KB for build/f011
Sent 12KB for build/f012
Sent 18KB for build/f018 # f013-f017 missing
Sent 33KB for build/f019 # this should be 19KB
Sent 53KB for build/f023 # we've skipped 20-22 and this is too large
Sent 54KB for build/f028 # we've skipped 24-27 and this is too large
Sent 63KB for build/f031 # we've skipped 29 and 30
Sent 34KB for build/f034 # this is good
Sent 152KB for build/f035 # this is too large
...snip

Note: these results are non-deterministic but consistent.

Specifically, if you look above, f020, f021 and f022 are missing (and correspondingly, f019 is exceedingly large).

I was able to dig a lot deeper and crafted files that had more self-explanatory text. These files have the name of the file and the hex line number on each line. That is, f000.txt looks like:

f000.txt: 00000000
f000.txt: 00000001
f000.txt: 00000002
...

Here, it became very obvious what was happening.

As seen here from a real upload: https://bafybeibwg3eyzsclut3zzw3zhtp7a4baqbqhrc74sjop2i3o2hsdlkh4de.ipfs.infura-ipfs.io/f024.txt (search for 000006AE)

...snip

f024.txt: 000006AB
f024.txt: 000006AC
f024.txt: 000006AD
f024.txt: 000006AE
000078              # f024 jumps to line 0x78 of f030.txt (!)
f030.txt: 00000079
f030.txt: 0000007A
f030.txt: 0000007B

...snip

The files are, in fact, bleeding together. After digging into the request (specifically, the HTTP multipart form-data, which is being generated in the IPFS http client library), the request is being sent correctly, but the server seems to be randomly dropping data from the request body. For instance, imagine you had the following request:

-------boundary
Content-Disposition: file; filename="file1.txt"
Content-Type: text/plain
file 1 data
-------boundary
Content-Disposition: file; filename="file2.txt"
Content-Type: text/plain
file 2 data
-------boundary
Content-Disposition: file; filename="file3.txt"
Content-Type: text/plain
file 3 data
------boundary

But then due to dropped data, the IPFS client saw only:

-------boundary
Content-Disposition: file; filename="file1.txt"
Content-Type: text/plain
file 1 data
-------boundary
Content-Disposition: file; filename="file2.txt"
Content-Type: text/plain
file 2 data
Content-Type: text/plain # note the drop here, and the lack of boundary separator
file 3 data
------boundary

From here, the server would effectively only see two files, file 1 with its correct data, and file 2, with a blend of file 2 and file 3 data (and possibly some metadata [which I have not observed in the wild]).

I have tested this locally against an IPFS server and have not been able to reproduce the issue. The issue is, alternatively, consistently occurring when pushing to Infura from multiple different machines.

My core suspicion is that Infura is dropping some data on the large streaming HTTP requests. That would be 1) consistent with the results being seen, 2) explain the non-determinism of the bug, 3) explain why it only occurs when pushing to Infura, and 3) explain why the large requests are important to trigger the bug.

It’s possible that we could work-around this issue by uploading files one-by-one, but it would be convenient to dig into this issue and whether the fault lies with our code, the IPFS client code, the IPFS server code, or the Infura infrastructure.

Sorry for making a verbose post, but I wanted to make clear the debugging we’ve done so far. Happy to help dig in more.

~~ Geoff

1 Like

Hey Geoff,

Thank you for the detailed report, it was really helpful.

After some long digging it appears that your suspicion were correct, there is something going on with the HTTP multipart/form-data request and files are indeed corrupted/merged. However it’s not due to Infura’s infrastructure.

It turns out that the origin of the problem is … NodeJS 15. This particular version is producing broken requests, possibly at the TCP level. Those requests even makes wireshark choke and display Warn Dissector bug, protocol HTTP, in packet 170: ./epan/packet.c:775: failed assertion "saved_layers_len < 500". Somehow go-ipfs is not sensible to those broken requests, but our internal proxy is, as well as the golang’s standard library.

As NodeJS 15 is not a supported release, I would advise to migrate to either 14 (LTS) or 15 (Current).

1 Like

Michael,

Thanks for looking into this. I’m currently seeing the issue when I try Node 14, 15 and 16 (earlier versions aren’t supported by the libraries). Could you think of any other work-around for this bug?

Thanks, -Geoff

Here Be Dragons :world_map: :dragon: :dragon: :dragon:

go-ipfs seems to have buggy HTTP behavior, detailed in go-ipfs#5168. It’s not entirely clear why the behavior exists or if it is fixed in master (see the related issue), but the workaround is to ensure your client sends a Connection: close header with all HTTP requests. By default, the ipfs-http-client does not, as specified in the node http Agent spec. Specifically:

keepAlive Keep sockets around even when there are no outstanding requests, so they can be used for future requests without having to reestablish a TCP connection. Not to be confused with the keep-alive value of the Connection header. The Connection: keep-alive header is always sent when using an agent except when the Connection header is explicitly specified or when the keepAlive and maxSockets options are respectively set to false and Infinity , in which case Connection: close will be used. Default: false.

Looking at the node ipfs client, the settings by default are:

    agent = opts.agent || new Agent({
      keepAlive: true,
      // Similar to browsers which limit connections to six per host
      maxSockets: 6
    })

Thus, the client does not send Connection: close as a header and thus we’re trip the ongoing go-ipfs bug.

The solution is thus to set the settings for the Agent (as specified above) such that we always send the header. This can be accomplished by passing in the following settings when creating the client:

const ipfsClient = require('ipfs-http-client');
const { Agent } = require('https');

return ipfsClient.create({
    host: ipfsHost,
    port: ipfsPort,
    protocol: ipfsProtocol,
    headers: {
      authorization: ipfsAuth
    },
    apiPath: '/api/v0',
    agent: new Agent({
      keepAlive: false,
      maxSockets: Infinity
    })
  });

With these headers set for the Agent and thus the connection header being passed to work around the go ipfs bug, we’re able to consistently upload files to IPFS.

Again, this does not, per se, uncover the root cause, but it is a viable path forward for any other intrepid travelers.

~~ Geoff