I would like to receive all event logs for a particular topic of an event log. I know there is a limit of 10,000 records, but the getPastLogs function times-out when less than 2,500 records need to be retrieved:
Hi @tester welcome to the community and thanks for reaching out.
Is it failing at exactly 2,500 records? We generally recommend iterating through blocks as we have set the limit so that no single block exceeds the limit.
Are there any different conditions that occur when it works without error? Possibly a different contract/address?
Thank you for your reply. To answer your questions:
Using the address and topic shown in my example, it should return about 2800 event logs. So no, it doesn’t fail at exactly 2500 records.
There are no other conditions that consistently produce success, or an error. It is currently producing an error every time. Yesterday, it consistently retrieved all ~2700 records for several hours.
To show how intermittent this problem is, when I finished writing this reply, it returned all 2793 records without error.
The following function will replicate this problem:
My client program can’t do anything while the request is being processed by your server. It’s not timing-out in the conventional sense (which normally takes at least 30 seconds). Instead, your server sends a “query timeout exceeded” message within about 10 seconds. The solution might be that you need to increase the time (to at least 30 seconds), before your server sends the “query timeout exceeded” message.
Apologies, my last message wasn’t applicable. Have you tried iterating through ranges of blocks, since setting the from block at 8,000,000 helped, this is typically our recommendation.
Your suggestion would require 9 message calls to receive the event logs for just one contract/topic, which is not feasible
This appears to be a problem with your system. Many users would like to receive all event logs for a contract/topic, and they don’t want to limit their search to just the past few months (1,000,000 blocks is about 5 months).
Can you possibly escalate this, so a search that returns less than 10,000 records will not receive a “query timeout exceeded” error?
What is the status of this issue?
It is difficult to audit contracts (such as the new MakerADO version that will be released in 6 days), without being able to receive all event logs.
I provided a function which demonstrates that your system issues a “query timeout exceeded” error when a contract contains less than 3,000 records. This is obviously a bug that needs to be fixed.
@tester we investigated the issue further and have found a potential way to speed up the query in two places that could resolve the issue on your side. We will keep you updated as we progress with the fix.
At this time, the best way to do this to avoid the issue is querying over ranges of blocks, you could do a dynamic backoff on block ranges to reduce the number of queries you have to do.
Apologies for the inconvenience and we thank you for helping us find this issue.
@mike Please provide a revised version of the DoGetInfuraLogs() function that I provided, to exemplify how the “dynamic backoff on block ranges” that you referred to can be used to receive all the event logs for that contract/topic (which has ~2800 event logs), without incurring additional message calls.
@mike Is there any progress on this bug? I would like to know that the getPastLogs function is reliable through Infura, before the big MakerDAO release on Monday (11/18/19)
Thanks for staying on top of this! The current query you are using fetches ~ 6MB of JSON data which is directly affecting the time to serve the request. The size will only grow as new data is added, since the historical data is immutable you can optimize by not including all the data in every query. We recommend only retrieving newer block data and appending that to a cache of previous results.
This problem is not caused by the data size, it is caused by the number of blocks that need to be searched, regardless of the data size.
The previous example that I provided retrieves less than 3,000 records, and your API allows retrieval of up to 10,000 records.
The following example shows that the data size is not causing the problem:
console.log(await web3.eth.getPastLogs({
address: “0x89d24a6b4ccb1b6faa2625fe562bdd9a23260359”,
topics: [“0x0f6798a560793a54c3bcfe86a93cde1e73087d944c0ea20544137d4121396885”],
fromBlock: 8850000
}))
This example retrieves more than 4,600 records with no problem, because it is only searching from block 8850000.
The next example retrieves the same records as the original example, except it retrieves from block 5000000 (instead of block 1):
console.log(await web3.eth.getPastLogs({
address: “0x75228dce4d82566d93068a8d5d49435216551599”,
topics: [“0xb2e65de73007eef46316e4f18ab1f301b4d0e31aa56733387b469612f90894df”],
fromBlock: 5000000
}))
This example does not display “query timeout exceeded”, because it is not retrieving from block 1.
Your API needs to retrieve up to 10,000 event logs from block 1. Otherwise, it is not returning an accurate report. Users need to retrieve all event logs from each contract, and they have no way of knowing the lowest block number of each contract.
Your system must either allow for a longer query time before throwing an error, or it must process faster.
Hiya tester, I get that this is not the most pleasant way to do this, but grabbing the logs in smaller chunks and merging them on that contract/topic works for me. Give this a try?
let logs, chunks;
chunks = 7000; // Even 10,000 chunks have a few segments that error out
logs = [];
for (let i=0; i < 10000000; i+=chunks) {
web3.eth.getPastLogs({
address: "0x89d24a6b4ccb1b6faa2625fe562bdd9a23260359",
topics: ["0x0f6798a560793a54c3bcfe86a93cde1e73087d944c0ea20544137d4121396885"],
fromBlock: i,
toBlock: i+chunks-1,
}).then(r => { logs = logs.concat(r); console.log("success:", i) }, err => { console.error(i + " failed", err) }).catch(console.log);
}
You may also need to re-sort the logs array, depending on how you’re using it.
I agree we need to come up with a more ergonomic way to do this, and document it better. It’s tricky because long running queries are a common attack vector, so getting rid of timeouts is not an easy solution. Will think about this more!
You’re correct, it requires a lot of requests, that is the nature of the workaround.
And yes, our load varies, sometimes queries run a little faster than other times and it can take time for the system to respond to increased load. That is the nature of having a system with many users.
You will hopefully have many more users, if your system runs efficiently and performs as represented (10,000 records per query). You might want to increase your capabilities to allow for more users than you currently have.
For what its worth, your system has been running better the last two days.I just ran the original example that I described, and it worked without error twice.
