Web3js subscriptions stop firing events after a while when using Infura

Iglou · March 25, 2021, 2:59pm

Hello!

I have been developing a project using Infura, and during longer tests I have noticed something that makes Infura unusable for my project.

I am using Web3js and am connecting to my Infura node through Websockets, and while the connection stays alive, some individual subscriptions stop firing events.

I know that the connection is alive, because:

Neither the web3.providers.WebsocketProvider nor the wrapped Websocket are firing any error/close event
My ‘newBlockHeaders’ subscription on the same connection is still firing events with correct data
I use a single Websocket connexion for all my subscriptions

The subscriptions that are subject to the issue do not fire an error event either. This is one of the subscription that cause that issue:

factoryContract.events.PairCreated()
    .on("connected", () => {
        console.log("Connected");
    }
    ).on("data", (event) =>{
        console.log(event);

    }).on("error", (error) => {
        console.log(error);
    });

With factoryContract being an instance of web3.eth.Contract, properly instanciated with the Uniswap V2 Factory Contract at 0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f, and its ABI. Nothing special really!

Everything works fine, but after a while (hard to pinpoint because of the scarcity of the events, but usually takes up to an hour), I can see on Etherscan that a new PairCreated event has been recorded, but no event has been fired on that subscription. However, I keep receiving events from my newBlockHeaders subscription.

My program, on start, retrieves the pairs I missed using the allPairs(uint256) method of the contract, still on Infura. When restarting, I can see that all the pairs I missed the event of have indeed been created.

I have then tried to use LinkPool’s public ethereum node, and have not had that issue at all, despite several hours of uninterrupted connexion to their Websocket endpoint. I assume it is an issue specific to Infura, and since it is not documented, I guess it is not an intended feature.

I have also configured my Websocket client with a keepalive firing every 60s, so this can’t be the issue.

Thank you for your time and help!

Leiya_Kenney · March 25, 2021, 9:58pm

Hey @Iglou, and welcome to the Infura community! Thanks for bringing this to our attention - we’re looking into it and will let you know what we’re able to find!

Iglou · March 25, 2021, 11:14pm

Thank you for taking an interest in this!

I thought it could be useful to have a summary of the issue: Subscriptions with scarce events, like PairCreated on the Uniswap Factory, or Swap on some low volume pairs, seem to silently stop firing events, and it only happens when using Infura.

If there is any more information I can provide to help, I would be happy to do so.

Leiya_Kenney · March 25, 2021, 11:40pm

Thanks so much! We love extra context, so if you notice anything else while we look into it, please let us know on this thread!

Iglou · March 26, 2021, 1:34am

To support my report, I have written and ran a small script using web3js that reproduces and demonstrates the issue.

The “abi.json” file contains the official contract ABI from Uniswap available at this address: https://unpkg.com/@uniswap/v2-core@1.0.1/build/IUniswapV2Factory.json
The websocket address contains my token, redacted for privacy.

const Web3 = require('web3');

const factoryAbi = require('./abi.json').abi;
const factoryAddress = "0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f";

const provider = new Web3.providers.WebsocketProvider(
    "wss://mainnet.infura.io/ws/v3/<REDACTED>",
    {
        clientConfig: {
            keepalive: true,
            keepaliveInterval: 60000
        },
        reconnect: {
            auto: true,
            delay: 5000,
            maxAttempts: 5,
            onTimeout: false
        }
    }
);

provider.on('connect', () => {
    console.log("Websocket connected.");
});

provider.on('close', (event) => {
    console.log(event);
    console.log("Websocket closed.");
});

provider.on('error', (error) => {
    console.error(error);
});

const web3 = new Web3(provider);

const factoryContract = new web3.eth.Contract(factoryAbi, factoryAddress);

web3.eth.subscribe("newBlockHeaders")
    .on('data', (data) => {
        console.log(
            `Received block header for block number ${data.number}.`
        );
    }).on('error', (error) => {
        console.error(error);
        console.error(
            "An error occured on the new blocks subscription."
        );
    }).on('connected', (id) => {
        console.log(
            `NewBlockHeaders subscription connected (${id})`
        );
    });

factoryContract.events.PairCreated()
    .on("connected", (id) => {
        console.log(`PairCreated subscription connected (${id})`);
    }
    ).on("data", (event) => {
        console.log(event);

    }).on("error", (error) => {
        console.log(error);
    })

I started the script at 12:25:32 AM UTC, and stopped it at 01:21:39 AM UTC. This is the output of that script:

Websocket connected.
NewBlockHeaders subscription connected (0x1)
PairCreated subscription connected (0x1feed340bd5e027678d08194c8a3fbbc)
Received block header for block number 12111375.
Received block header for block number 12111376.
Received block header for block number 12111377.
Received block header for block number 12111378.
Received block header for block number 12111379.
Received block header for block number 12111380.
Received block header for block number 12111381.
Received block header for block number 12111382.
Received block header for block number 12111383.
Received block header for block number 12111384.
Received block header for block number 12111385.
Received block header for block number 12111386.
Received block header for block number 12111387.
Received block header for block number 12111388.
Received block header for block number 12111389.
Received block header for block number 12111390.
Received block header for block number 12111391.
Received block header for block number 12111391.
Received block header for block number 12111392.
Received block header for block number 12111393.
Received block header for block number 12111394.
Received block header for block number 12111395.
Received block header for block number 12111396.
Received block header for block number 12111397.
Received block header for block number 12111398.
Received block header for block number 12111399.
Received block header for block number 12111400.
Received block header for block number 12111401.
Received block header for block number 12111402.
Received block header for block number 12111403.
{
removed: false,
logIndex: 176,
transactionIndex: 74,
transactionHash: '0x93017d13260c54c95851582f02da80e2d42327be27e040771dac843c1e8e4022',
blockHash: '0x1f25cdfd28132567704f3884870e73c6d3fdbbbccdbe2ec1372f7c0f24766260',
blockNumber: 12111403,
address: '0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f',
id: 'log_10fb5d4b',
returnValues: Result {
    '0': '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2',
    '1': '0xd5D84D6524b5A59494C46996c187C2468b0Dc3e9',
    '2': '0xC9d99c00CC8e86b0DCe3256616a93F9805d2419E',
    '3': '31529',
    token0: '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2',
    token1: '0xd5D84D6524b5A59494C46996c187C2468b0Dc3e9',
    pair: '0xC9d99c00CC8e86b0DCe3256616a93F9805d2419E'
},
event: 'PairCreated',
signature: '0x0d3648bd0f6ba80134a33ba9275ac585d9d315f0ad8355cddefde31afa28d0e9',
raw: {
    data: '0x000000000000000000000000c9d99c00cc8e86b0dce3256616a93f9805d2419e0000000000000000000000000000000000000000000000000000000000007b29',
    topics: [
    '0x0d3648bd0f6ba80134a33ba9275ac585d9d315f0ad8355cddefde31afa28d0e9',
    '0x000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2',
    '0x000000000000000000000000d5d84d6524b5a59494c46996c187c2468b0dc3e9'
    ]
}
}
Received block header for block number 12111404.
Received block header for block number 12111405.
Received block header for block number 12111406.
Received block header for block number 12111407.
Received block header for block number 12111408.
Received block header for block number 12111409.
Received block header for block number 12111410.
Received block header for block number 12111411.
Received block header for block number 12111412.
Received block header for block number 12111413.
Received block header for block number 12111414.
Received block header for block number 12111415.
Received block header for block number 12111416.
Received block header for block number 12111417.
Received block header for block number 12111418.
Received block header for block number 12111419.
Received block header for block number 12111420.
Received block header for block number 12111421.
Received block header for block number 12111422.
Received block header for block number 12111423.
Received block header for block number 12111424.
Received block header for block number 12111425.
Received block header for block number 12111426.
Received block header for block number 12111427.
Received block header for block number 12111428.
Received block header for block number 12111429.
Received block header for block number 12111430.
Received block header for block number 12111431.
Received block header for block number 12111432.
Received block header for block number 12111433.
Received block header for block number 12111434.
Received block header for block number 12111435.
Received block header for block number 12111436.
Received block header for block number 12111437.
Received block header for block number 12111438.
Received block header for block number 12111439.
Received block header for block number 12111440.
Received block header for block number 12111441.
Received block header for block number 12111442.
Received block header for block number 12111443.
Received block header for block number 12111444.
Received block header for block number 12111445.
Received block header for block number 12111446.
Received block header for block number 12111447.
Received block header for block number 12111448.
Received block header for block number 12111449.
Received block header for block number 12111450.
Received block header for block number 12111451.
Received block header for block number 12111452.
Received block header for block number 12111453.
Received block header for block number 12111454.
Received block header for block number 12111455.
Received block header for block number 12111456.
Received block header for block number 12111457.
Received block header for block number 12111458.
Received block header for block number 12111459.
Received block header for block number 12111460.
Received block header for block number 12111461.
Received block header for block number 12111462.
Received block header for block number 12111463.
Received block header for block number 12111464.
Received block header for block number 12111465.
Received block header for block number 12111466.
Received block header for block number 12111467.
Received block header for block number 12111468.
Received block header for block number 12111469.
Received block header for block number 12111470.
Received block header for block number 12111471.
Received block header for block number 12111472.
Received block header for block number 12111473.
Received block header for block number 12111474.
Received block header for block number 12111475.
Received block header for block number 12111476.
Received block header for block number 12111477.
Received block header for block number 12111478.
Received block header for block number 12111478.
Received block header for block number 12111479.
Received block header for block number 12111480.
Received block header for block number 12111481.
Received block header for block number 12111482.
Received block header for block number 12111483.
Received block header for block number 12111484.
Received block header for block number 12111485.
Received block header for block number 12111486.
Received block header for block number 12111487.
Received block header for block number 12111488.
Received block header for block number 12111489.
Received block header for block number 12111490.
Received block header for block number 12111491.
Received block header for block number 12111492.
Received block header for block number 12111493.
Received block header for block number 12111493.
Received block header for block number 12111494.
Received block header for block number 12111495.
Received block header for block number 12111496.
Received block header for block number 12111497.
Received block header for block number 12111498.
Received block header for block number 12111499.
Received block header for block number 12111500.
Received block header for block number 12111501.
Received block header for block number 12111502.
Received block header for block number 12111503.
Received block header for block number 12111504.
Received block header for block number 12111505.
Received block header for block number 12111506.
Received block header for block number 12111507.
Received block header for block number 12111508.
Received block header for block number 12111509.
Received block header for block number 12111510.
Received block header for block number 12111511.
Received block header for block number 12111512.
Received block header for block number 12111513.
Received block header for block number 12111514.
Received block header for block number 12111515.
Received block header for block number 12111516.
Received block header for block number 12111517.
Received block header for block number 12111518.
Received block header for block number 12111519.
Received block header for block number 12111520.
Received block header for block number 12111521.
Received block header for block number 12111522.
Received block header for block number 12111523.
Received block header for block number 12111524.
Received block header for block number 12111525.
Received block header for block number 12111526.
Received block header for block number 12111527.
Received block header for block number 12111528.
Received block header for block number 12111529.
Received block header for block number 12111530.
Received block header for block number 12111531.
Received block header for block number 12111532.
Received block header for block number 12111533.
Received block header for block number 12111534.
Received block header for block number 12111535.
Received block header for block number 12111536.
Received block header for block number 12111537.
Received block header for block number 12111538.
Received block header for block number 12111539.
Received block header for block number 12111540.
Received block header for block number 12111541.
Received block header for block number 12111542.
Received block header for block number 12111543.
Received block header for block number 12111544.
Received block header for block number 12111545.
Received block header for block number 12111546.
Received block header for block number 12111547.
Received block header for block number 12111548.
Received block header for block number 12111549.
Received block header for block number 12111550.
Received block header for block number 12111551.
Received block header for block number 12111552.
Received block header for block number 12111553.
Received block header for block number 12111554.
Received block header for block number 12111555.
Received block header for block number 12111556.
Received block header for block number 12111557.
Received block header for block number 12111558.
Received block header for block number 12111559.
Received block header for block number 12111560.
Received block header for block number 12111561.
Received block header for block number 12111562.
Received block header for block number 12111563.
Received block header for block number 12111564.
Received block header for block number 12111565.
Received block header for block number 12111566.
Received block header for block number 12111567.
Received block header for block number 12111568.
Received block header for block number 12111569.
Received block header for block number 12111570.
Received block header for block number 12111571.
Received block header for block number 12111572.
Received block header for block number 12111573.
Received block header for block number 12111574.
Received block header for block number 12111575.
Received block header for block number 12111576.
Received block header for block number 12111577.
Received block header for block number 12111578.
Received block header for block number 12111579.
Received block header for block number 12111580.
Received block header for block number 12111581.
Received block header for block number 12111582.
Received block header for block number 12111583.
Received block header for block number 12111584.
Received block header for block number 12111585.
Received block header for block number 12111586.
Received block header for block number 12111587.
Received block header for block number 12111588.
Received block header for block number 12111589.
Received block header for block number 12111590.
Received block header for block number 12111591.
Received block header for block number 12111592.
Received block header for block number 12111593.
Received block header for block number 12111594.
Received block header for block number 12111594.
Received block header for block number 12111595.
Received block header for block number 12111596.
Received block header for block number 12111597.
Received block header for block number 12111598.
Received block header for block number 12111599.
Received block header for block number 12111600.
Received block header for block number 12111601.
Received block header for block number 12111602.
Received block header for block number 12111603.
Received block header for block number 12111604.
Received block header for block number 12111605.
Received block header for block number 12111606.
Received block header for block number 12111607.
Received block header for block number 12111608.
Received block header for block number 12111609.
Received block header for block number 12111610.
Received block header for block number 12111611.
Received block header for block number 12111612.
Received block header for block number 12111613.
Received block header for block number 12111614.
Received block header for block number 12111615.
Received block header for block number 12111616.
Received block header for block number 12111617.
Received block header for block number 12111618.
Received block header for block number 12111619.

You can see here that only one pair was received on bloc 12111403.
At bloc 12111607, another pair has been created and the proper event emitted on the blockchain, but no event has been received on my side. No error either.

I let the script run for a dozen more blocs to make sure data wouldn’t arrive with a bit of delay. On my main program, when I noticed the issue, it never arrived.

I hope this can help!

Ryan_Schneider · April 5, 2021, 4:42pm

Thanks for the information @Iglou, I’ve been looking into the issue and haven’t been able to come up with a potential cause yet, but wanted to let you know I’ve been looking.

Iglou · April 7, 2021, 10:37pm

Thank you for letting me know!

romain.griffiths · April 21, 2021, 11:36am

Hello Iglou,

I have experienced the same issue yesterday.
I have also several components connected to Infura and some of them exhibit the missed Event and some not (same code used).

What could be useful is to bypass Web3 to exclude the library. We should directly emit the Subscription to the WebSocket and watch for missing Events.

Iglou · April 21, 2021, 12:02pm

Hello Romain,

Before writing this bug report, I have read the source code of the WebsocketProvider Web3js uses, and attempted to listen to the events directly from the actual WebSocket object without receiving anything either, no data, no error.
Of course, I am definitely not infallible, and ditching Web3js to work with raw websockets might produce results, but as far as I am concerned, I find it highly unlikely.
I have also been running the same code with a different node-as-a-service provider, which I won’t tell the name of because it is not the appropriate place for that, and I did not encounter a single issue.

My guess is that the source of the issue is Infura, and I can’t investigate further than that by myself!

romain.griffiths · April 21, 2021, 4:56pm

Hi,

I have the paid plan and I made a sound bug report on their support.

Best Regards,

sarfraz · June 7, 2021, 10:10pm

I am experiencing a similar issue using ethers.js. Any update on this?

Leiya_Kenney · June 8, 2021, 6:42pm

Hi @sarfraz, and welcome to the Infura community!
Wanted to let you know that we are still looking into this, but don’t have a solution yet. We will update this thread when we have something to share!

influenceth · July 9, 2021, 2:32pm

I have also been experiencing the same issue. It has required modification of our listeners to occasionally do a call to reach back and grab all events over the past 24 hours to ensure none were missed. But the eventual consistency with substantial lag is certainly not ideal.

Any updates as to whether this is fixed / when it will be fixed?

Leiya_Kenney · August 11, 2021, 4:27pm

Hi @influenceth, and welcome to the Infura community! This hasn’t been fixed yet, but it’s on our docket to work on shortly. We will be sure to update this thread when we have one!

omadalosso · August 23, 2021, 11:46am

I’ve experienced the same behavior on my application… But only when running it locally…
The same code running on a EC2 instance never failed to receive event updates from Infura.
I wonder if the others who experienced the error where running their app locally or not. Might be a clue…

Leiya_Kenney · August 25, 2021, 10:57pm

Hey @omadalosso, and welcome to the Infura community!

Thank you so much for the additional context! I’ve passed that information along to our devs.

JimLynchCodes · November 27, 2021, 9:24pm

Any update on this?

I am using ether.js, and actually not infura… maybe this is an issue with ethereum itself? Or the way we are establishing the websocket connection?

I too am seeing that when I deploy my [node.js] server I can listen to events, and I hear them all as expected.

After a few hours of no events being emitted though it seems like the subscribe just stops listening altogether- no events that are emitted are picked up by the listener… when I restart the server running the listener it all works fine, until a few hours go by…

The whole thing is just maddeningly frustrating… if I have a subscription that does have any events coming through for a while SHOULD it disconnect? It shouldn’t right? What is the point of that? and is there any way that it should never disconnect just from being idle??

two.bits · December 26, 2021, 2:57pm

We had the same issue with Web3.js as well. This usually happens when the subscription becomes idle for quite some time, i.e. no event for a few hours. We perform health checks every 30 seconds using web3.eth.net.isListening. Whenever this happens, seems like the connection is still up. At the same time, there are no exception thrown. Might be something with the subscription itself? Not sure if there are any methods to check subscription status by the subscription id on the node?

We only had a work around to auto reestablish the connection every 6 hours. This is certainly not ideal. Looking forward to better solutions to deal with this.

surinder.k · January 3, 2023, 11:08am

I’m also facing the same issue with NestJs + Web3. Service remains up however stops listening to the contract events. Below is the sample code from our service.

import { Injectable } from '@nestjs/common';
import Web3 from 'web3';
import { RabbitMqService } from '../../common/helper/rabbitmq/rabbit-mq.service';
import { RedisService } from '../../common/helper/redis/redis.service';
import { ABI } from '../../constants/contract-abi';
// Ref: https://community.infura.io/t/web3-js-how-to-track-nft-erc-721-1155-transfers-and-mints-specific-address-nft/5500

@Injectable()
export class EthereumService {
  providerObj: any = {};
  web3: any = {};
  myContract: any = {
    status: false,
    instance: {},
  };
  provider: any = {
    status: false,
    instance: {},
  };
  counter = 1;
  eventsList = [
    'RewardClaimed',
    'RaffleCreated',
    'BuyTicket',
    'WinnerDeclared',
  ];
  constructor(
    private readonly rabbitMqService: RabbitMqService,
    private readonly redisService: RedisService,
  ) {
    this.initiateProvider();
  }

  /**
   * This code will initiate Web3 provider,
   * If Connected: InitiateContractInsance()
   * If Error/End: call retry function retryProviderConnection()
   * Ref: https://github.com/web3/web3.js/tree/1.x/packages/web3-providers-ws#usage
   * timeout,delay in Milliseconds
   */
  initiateProvider() {
    // const options = {
    //   timeout: 3000,
    //   reconnect: {
    //     auto: true,
    //     delay: 1000,
    //     maxAttempts: 5,
    //     onTimeout: false,
    //   },
    // };
    this.providerObj = new Web3.providers.WebsocketProvider(
      process.env.PROVIDER_URL,
    );
    this.providerObj.on('connect', (data) => {
      this.provider.status = true;
      this.provider.instance = this.providerObj;
      this.initiateContractInstance();
    });
    this.providerObj.on('error', () => {
      this.retryProviderConnection();
    });
    this.providerObj.on('end', () => {
      this.retryProviderConnection();
    });
  }

  initiateContractInstance(): void {
    this.web3 = new Web3(this.provider.instance);
    this.myContract.status = true;
    this.myContract.instance = new this.web3.eth.Contract(
      JSON.parse(ABI),
      process.env.CONTRACT_ADDRESS,
    );
    this.subscribeContractEvents();
  }

  /**
   * Retry connection max 5 times, if not connected send an alert email to
   * support team and admin and stop retrying
   */
  retryProviderConnection() {
    if (this.counter <= 5) {
      setTimeout(() => {
        this.initiateProvider();
      }, 5000);
      this.counter++;
    } else if (this.counter == 6) {
      // Send email to support team and admin, pending
      console.log('Web3 provider socket disconnected after 5 retries');
    }
  }

  /**
   * Subscribe multiple events
   * fetch all past events since the service stopped/restarted
   */
  subscribeContractEvents(): void {
    this.getPastEventsIfAny();
    this.subscribeContractEventsList();
  }

  /**
   * Subscribe array of events
   */
  subscribeContractEventsList(): void {
    this.eventsList.forEach((eventName) => {
      this.myContract.instance.events[eventName]({ fromBlock: 'latest' })
        .on('connected', (subscriptionId) => {
          console.table([eventName, subscriptionId]);
        })
        .on('data', (event) => {
          this.sendToQueue(event.event, event);
          this.updateRedisBlockNo(event.blockNumber);
        })
        // .on('changed', (event) => {
        //   console.log('event changed: ', event);
        // })
        .on('error', (error, receipt) => {
          console.log('error: ', error, receipt);
        });
    });
  }

  async getLatestBlockNumber() {
    return await this.web3.eth.getBlockNumber();
  }

  async getPastEventsIfAny(): Promise<void> {
    const currentBlockNo = await this.getLatestBlockNumber();
    let lastBlockNo: any = await this.redisService.get('eventBlockNo');
    if (isNaN(lastBlockNo)) {
      lastBlockNo = currentBlockNo;
    }
    this.myContract.instance
      .getPastEvents('allEvents', {
        fromBlock: lastBlockNo,
        toBlock: 'latest',
      })
      .then(async (events) => {
        if (events.length) {
          for (const el of events) {
            if (this.eventsList.includes(el.event)) {
              this.sendToQueue(el.event, el);
              this.updateRedisBlockNo(el.blockNumber);
            }
          }
        }
      });
  }
  /**
   * Send event data to queue
   * patteren = name of the event
   * @param pattern, @param data
   */
  async sendToQueue(pattern: string, data: any) {
    try {
      this.rabbitMqService.send(pattern, data);
      console.table([data.event, data.transactionHash]);
    } catch (error) {
      throw new Error(error);
    }
  }
  /**
   * Update last processed event's block no in Redis
   * ttl 7 days = 604800
   * @param blockNo
   */
  async updateRedisBlockNo(blockNo: number) {
    try {
      this.redisService.set('eventBlockNo', blockNo, {
        ttl: 604800,
      });
    } catch (error) {
      throw new Error(error);
    }
  }
}

Does the issue is with Infura WSS?