Ropsten websocket event timeout (with web3.py)

When getting events for my contract deployed on Ropsten, fromBlock=6807500 will succeed, however, fromBlock=6807400 will result in timeout –

“”"
fromBlock=6807500, Both BrickTransfer and EthUpdate succeed
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef

EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef

fromBlock=6807400, BrickTransfer succeeds, but EthUpdate will timeout
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19

“”"

Please checkout the below package to reproduce this issue (tested in Ubuntu 18.04, Python 3.6.8) –

Thanks.

Hi @Jerry_Ji could you provide the full error output?

The second round of EthUpdate part will timeout, so basically –

“”"
fromBlock=6807500, Both BrickTransfer and EthUpdate succeed
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
BrickTransfer event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef

EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef
EthUpdate event transactionHash 0xee9fffbf547c2aed22df404c5a9ee247fb030f92027c18a08f2532fb8d1c60ef

fromBlock=6807400, BrickTransfer succeeds, but EthUpdate will timeout
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19
BrickTransfer event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19

Traceback (most recent call last):
File “ropsten_wss_timeout_demo.py”, line 42, in
for event in event_filter.get_all_entries()[:3]:
File “/usr/local/lib/python3.6/dist-packages/web3/_utils/filters.py”, line 128, in get_all_entries
log_entries = self._filter_valid_entries(self.web3.eth.getFilterLogs(self.filter_id))
File “/usr/local/lib/python3.6/dist-packages/web3/eth.py”, line 407, in getFilterLogs
“eth_getFilterLogs”, [filter_id],
File “/usr/local/lib/python3.6/dist-packages/web3/manager.py”, line 94, in request_blocking
response = self._make_request(method, params)
File “/usr/local/lib/python3.6/dist-packages/web3/manager.py”, line 81, in _make_request
return request_func(method, params)
File “cytoolz/functoolz.pyx”, line 232, in cytoolz.functoolz.curry.call
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py”, line 50, in apply_formatters
response = make_request(method, params)
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/gas_price_strategy.py”, line 18, in middleware
return make_request(method, params)
File “cytoolz/functoolz.pyx”, line 232, in cytoolz.functoolz.curry.call
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py”, line 50, in apply_formatters
response = make_request(method, params)
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/attrdict.py”, line 18, in middleware
response = make_request(method, params)
File “cytoolz/functoolz.pyx”, line 232, in cytoolz.functoolz.curry.call
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py”, line 50, in apply_formatters
response = make_request(method, params)
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/normalize_errors.py”, line 9, in middleware
result = make_request(method, params)
File “cytoolz/functoolz.pyx”, line 232, in cytoolz.functoolz.curry.call
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py”, line 50, in apply_formatters
response = make_request(method, params)
File “cytoolz/functoolz.pyx”, line 232, in cytoolz.functoolz.curry.call
File “/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py”, line 50, in apply_formatters
response = make_request(method, params)
File “/usr/local/lib/python3.6/dist-packages/web3/providers/websocket.py”, line 119, in make_request
return future.result()
File “/usr/lib/python3.6/concurrent/futures/_base.py”, line 432, in result
return self.__get_result()
File “/usr/lib/python3.6/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
File “/usr/local/lib/python3.6/dist-packages/web3/providers/websocket.py”, line 107, in coro_make_request
timeout=self.websocket_timeout
File “/usr/lib/python3.6/asyncio/tasks.py”, line 362, in wait_for
raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
“”"

May I know if anyone has succeeded in running the code and reproducing the issue? It should be easy as the only third-party dependent is web3.py

Hi @Jerry_Ji we are looking into this, investigating how web3.py handles subscriptions. We will get back to you asap.

Thanks @mike, I’ve traced down to the two requests web3.py makes –

b'{"jsonrpc": "2.0", "method": "eth_newFilter", "params": [{"topics": ["0xdd459839ac0a0ca84e4f1ba88385faf17fe078dfc7570213a8e11e72115e45ab"], "fromBlock": "0x67df68", "toBlock": "latest", "address": ["0xe517fdC6ccceaf902b085fBEd66ADcc47D799BFa"]}], "id": 0}'

b'{"jsonrpc": "2.0", "method": "eth_getFilterLogs", "params": ["0x286ee012b947f4402b23c65dbad1ac35"], "id": 1}'

Though I don’t see anything wrong with them.

This is very likely an issue with web3.py in how it manages the request via filters. Since this is working for you in web3.js I would recommend using that, or managing your websocket connection using the python websockets library instead. Logs subscription through the websocket library should be much more controllable on your side.

Hi @mike,

Well, it’s not that web3.py does not work at all, it does work to certain extent.

Here’s an additional piece of evidence which shows with toBlock. It works when toBlock - fromBlock <= 112, but timeout when toBlock - fromBlock > 112

Doesn’t that mean there’s something with Infura that should be fixed / improved?

BTW, I’ve updated my github repository ( link in my original post ) with the new piece of code –

$ python ropsten_wss_timeout_demo2.py 
fromBlock=6807400, works when toBlock-fromBlock <= 112, timeout when toBlock-fromBlock > 112
when toBlock-fromBlock <= 112
EthUpdate event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19
EthUpdate event transactionHash 0xb0e40b17a73fc8fd32b80e90b18257d490c96328fe55d3a7514276bff888ab19

when toBlock-fromBlock > 112
^CTraceback (most recent call last):
  File "ropsten_wss_timeout_demo2.py", line 44, in <module>
    all_entries = event_filter.get_all_entries()
  File "/usr/local/lib/python3.6/dist-packages/web3/_utils/filters.py", line 125, in get_all_entries
    log_entries = self._filter_valid_entries(self.web3.eth.getFilterLogs(self.filter_id))
  File "/usr/local/lib/python3.6/dist-packages/web3/eth.py", line 407, in getFilterLogs
    "eth_getFilterLogs", [filter_id],
  File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 109, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 96, in _make_request
    return request_func(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 51, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 51, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 51, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/normalize_errors.py", line 9, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 51, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 51, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/providers/websocket.py", line 122, in make_request
    return future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 427, in result
    self._condition.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt

Unfortunately filters can be a heavy request on the node and a timeout can occur. We recommend using websocket subscriptions in place of filters, https://infura.io/docs/ethereum/wss/eth_subscribe.

Can you give this a try for the data you’re trying to pull down and let me know if it helps?

Hi @mike ,

We recommend using websocket subscriptions in place of filters

OK I get it. For new events we should use eth_subscribe

But for past events there is only getPastEvents, for which I’m curious to understand –

  • Why does Infura behave differently for web3.js and web3.py, is that an inherent feature or bug?

  • Why is there a cut off for toBlock - fromBlock >= 113, which is not a particularly large range?

Any information would be much appreciated.

eth_getLogs is your best HTTP method for past events, https://infura.io/docs/ethereum/json-rpc/eth_getLogs, where eth_subscribe will be your best WSS method for current events.

We do not have any specific handling for any Web3 library, the difference in behavior would be entirely because of how the library is written to work with our APIs.

The issue you’re seeing with the 113 block range on the filters method is likely do to the number of logs inside of that range. Filters are specifically heavy on the nodes and smaller block ranges is to be expected.