Timing out when adding data to IPFS node

stellarmagnet · May 15, 2019, 8:54pm

We are getting a lot of timeouts when trying to pin data to Infura’s ipfs endpoint via our dapps and are losing data in multiple cases.

Is there any background for what is going on and estimated time to resolution?

mike · May 15, 2019, 9:19pm

Sorry to hear you are having issues, can you provide the code for your requests and the full error response?

stellarmagnet · May 15, 2019, 9:42pm

It used to work fine 1 week ago, and we haven’t changed any of our code. We are getting 504 gateway timeouts.

Reference code though:

github.com

AutarkLabs/planning-suite/blob/27a69db81add0f8be2ac079811571f6febd64382/apps/projects/app/components/App/App.js#L22


import { hot } from 'react-hot-loader'
import styled from 'styled-components'
import { map } from 'rxjs/operators'
import ApolloClient from 'apollo-boost'
import { ApolloProvider } from 'react-apollo'
import { Issues, Overview, Settings } from '../Content'
import PanelManager, { PANELS } from '../Panel'
import { STATUS } from '../../utils/github'
import ErrorBoundary from './ErrorBoundary'
import BigNumber from 'bignumber.js'
import { ipfsAdd, computeIpfsString } from '../../utils/ipfs-helpers'
import {
networkContextType,
AppTitle,
AppTitleButton,
} from '../../../../../shared/ui'
import {
REQUESTED_GITHUB_TOKEN_SUCCESS,
REQUESTED_GITHUB_TOKEN_FAILURE,
} from '../../store/eventTypes'
import { CURRENT_USER } from '../../utils/gql-queries'

github.com

AutarkLabs/planning-suite/blob/dev/apps/projects/app/utils/ipfs-helpers.js

import axios from 'axios'
import ipfsClient from 'ipfs-http-client'

let infuraIpfs = ipfsClient({ host: 'ipfs.infura.io', port: '5001', protocol: 'https' })
const infura = 'https://ipfs.infura.io:5001/api/v0'

const bufferFile = (content) => infuraIpfs.types.Buffer.from(JSON.stringify(content))

export const ipfsAdd = async (content) => {
  const file = bufferFile(content)
  try {
    const result = await infuraIpfs.add(file)
    return result[0].hash
  } catch (err) {
    console.error('Error pinning file to IPFS', err)
  }
}

// compute the hashes individually, then join them together into one string
// this is specific for our bounty smart contract interface

This file has been truncated. show original

michael.mure · May 16, 2019, 3:55pm

Does that happen every time or is that occasional ? Could you provide the exact request that is made (cURL equivalent if possible) ?

stellarmagnet · May 16, 2019, 6:26pm

@michael.mure this seems to happen like 50% of the time lately. It was not like this a week ago.

Here is what i get in the browser console when i trigger an ipfsGet action via open-enterprise/ipfs-helpers.js at dev · AutarkLabs/open-enterprise · GitHub and open-enterprise/issues.js at 783a3f67e5657a6f862e68d5628fa1f2c84731d4 · AutarkLabs/open-enterprise · GitHub

michael.mure · May 17, 2019, 1:40pm

It looks like you are using chrome, could you give me the cURL request ?

Baquedano · May 17, 2019, 2:48pm

I’m getting the same issue, this is my add request:

curl -H 'Host: ipfs.infura.io:5001' -H 'accept: */*' -H 'content-type: multipart/form-data; boundary=jrAYs1aeI3vG5z99gHcM5mnf27cdvyQm' -H 'user-agent: Swift IPFS Client' -H 'accept-language: en-us' --data-binary '--jrAYs1aeI3vG5z99gHcM5mnf27cdvyQm

content-disposition: file; name="file";

content-type: application/octet-stream

content-transfer-encoding: binary

cmon just pin me already

--jrAYs1aeI3vG5z99gHcM5mnf27cdvyQm--

' --compressed 'https://ipfs.infura.io:5001/api/v0/add?stream-channels=true'

and this is my pin request:

curl -H 'Host: ipfs.infura.io:5001' -H 'user-agent: my-proj/21 CFNetwork/897.15 Darwin/17.5.0' -H 'accept: */*' -H 'accept-language: en-us' --compressed 'https://ipfs.infura.io:5001/api/v0/pin/add?stream-channels=true&arg=QmPTExpvLpQstJDJb8icrqQZ5LHEqGe2G98gpm2qymgMeH'

it happens like 20% of the time when pinning.

jamesmorgan · May 17, 2019, 7:58pm

I am also seeing issues with one off our service - “Gateway Timeout” on some of the pinned images.

Moonwilles · May 19, 2019, 5:35pm

Same here, and it is doing it intermittently. I think it has something to do with Infura itself rather than the code.
Thank you.

mike · May 20, 2019, 11:15pm

Thanks everyone, we are looking into this, we will update once we have more information.

thibautvdu · May 24, 2019, 3:08pm

Hello,

I’m sharing the issue here as this thread seems more relevant, we are encountering the same issue when retrieving pinned data. That is the IPFS.add function correctly returned the hash without any exception, but when trying to retrieve the file a minute after and (even now, hours after), the gateway times out. Here are two hashes that aren’t working :
Qmf7yvoeCSfLcMZtfT9UaPkz383VJvXkkL8rYpz4oRRufk
QmYTWr3Qh1KQf6XQ2s7uM1YcbvJHVcukTS1MCkb7aYzQgU

Is this data lost ?
Many thanks,

mike · May 24, 2019, 5:32pm

Would you provide us with your code that is generating these requests and the full error response?

thibautvdu · May 24, 2019, 6:18pm

Hello Mike and thanks for your reply.

This is the upload code, which executed successfully with photoNearHash and photoFarHash being filled in with the values of my previous post :

  var photoNearFileContent = await Buffer.from(await LoadFileAsync(photoNearFile));
  var photoFarFileContent = await Buffer.from(await LoadFileAsync(photoFarFile));

  var photoNearHash = null;
  var photoFarHash = null;

  try {
    var res = await ipfs.add(photoNearFileContent);
    if(!res) throw new Error("uploading photoNearFileContent failed");

    photoNearHash = res[0].hash;
    dispatch(actions.receiveIpfsHash({photoNearHash: photoNearHash}));

    res = await ipfs.add(photoFarFileContent);
    if(!res) throw new Error("uploading photoFarFileContent failed")

    photoFarHash = res[0].hash;
    return dispatch(actions.receiveIpfsHash({photoFarHash: photoFarHash}));
  } catch(err) {
    dispatch(actions.fail());
    alert(err);
    return;
  }

As for the code generating the error, it is simply a get request :

  const photoNearUrl = ifpfsGatewayAddress + "/" + photoNearHash;
  const photoFarUrl = ifpfsGatewayAddress + "/" + photoFarHash;
  //const photoNearLocalPath = "./public/pdfAssets/ipfsAssets/" + photoNearHash + ".jpeg";
  //const photoFarLocalPath = "./public/pdfAssets/ipfsAssets/" + photoFarHash + ".jpeg";
  var photoNearBuffer = null;
  var photoFarBuffer = null;
  var tries = 1;
  while(!photoNearBuffer || !photoFarBuffer) {
    try {
      if(!photoNearBuffer) photoNearBuffer = await request({url: photoNearUrl, encoding: null});
      if(!photoFarBuffer) photoFarBuffer = await request({url: photoFarUrl, encoding: null});
    } catch(e) {
      if(tries > 6) throw "error while downloading files from ipfs: " + e.toString();

      await delay(tries * 30000);
      tries++;
    }
  }

The exact error being

error while downloading files from ipfs: StatusCodeError: 504 - {“type”:“Buffer”,“data”:[71,97,116,101,119,97,121,32,84,105,109,101,111,117,116,10]}

I got the same error in my browser, which now seems to have changed to a 502 bad gateway (after 10 minutes of loading or so) :
https://ipfs.infura.io/ipfs/Qmf7yvoeCSfLcMZtfT9UaPkz383VJvXkkL8rYpz4oRRufk
https://ipfs.infura.io/ipfs/QmYTWr3Qh1KQf6XQ2s7uM1YcbvJHVcukTS1MCkb7aYzQgU

As you can notice, I put a loop with a delay in my code when getting the files. That’s because from times to times, the pinned files aren’t available before 5 minutes or so after the IPFS.add, but this time it’s been about 8 hours already.

Any help would be much appreciated, especially regarding the state of those files. We completely relied on your node for storing files so I’d really need to know whether or not those two uploads are definitely lost (it’s user generated content )

( to answer your question on the other thread, that issue was lastly experienced today, about 8 hours ago )

mike · May 24, 2019, 7:08pm

Thanks for the added info! We pushed a fix that should resolve the 504 errors you’re seeing, can you test it now?

thibautvdu · May 25, 2019, 6:12am

Hello Mike,
A new upload worked fine, but the two hashes I gave you are still timing out

thibautvdu · May 27, 2019, 8:36am

Follow up : I ended up asking my client to send its file by e-mail for re-upload.

While Infura is awesome for Ethereum, I wouldn’t say the IPFS service is ready for production yet. There is currently no warranty that a successful call to IPFS.add will actually upload & pin your file, as you might find out later when retrieving your file with the generated hash that it was a false positive. I’ll use a second node provider as a backup on our D-App for now, even though I’d have preferred to stick with Infura even if I had to pay for it. Thanks for your help @mike

vishesh_malhotra · October 9, 2019, 7:06pm

I am getting the same issue

egalano · October 16, 2019, 7:57am

Hi vishesh,

Are you still experiencing any issues with our IPFS endpoint?

vishesh_malhotra · October 16, 2019, 6:06pm

Now its working fine. Thanks

john-osullivan · January 8, 2020, 10:55pm

I’ve been using Infura to upload builds of static sites and getting intermittent issues which look kind of like timeouts. We always get a response in about a minute, but more often than not, it is only has a handful of file hashes, and no directories. Most importantly, no root hash.

For a while I thought this was an issue with how I was using IPFS, but it looks like it’s really related to Infura behavior – it works fine with a local daemon. Here’s the issue I made on the IPFS-JS repo documenting the behavior more thoroughly: https://github.com/ipfs/js-ipfs-http-client/issues/1212

I was able to work around it by simply retrying the request – it eventually succeeds with a complete response. Thought this still might be worth reporting to y’all, though – @egalano !