Internal logs subscription error

kaleb-keny · May 26, 2021, 8:57am

Hi,

I have websockets setup on infura, and I keep getting every once in a while an error thrown Internal logs subscription error… This started happening around 2 weeks ago, and i haven’t changed anything in my setup. Is there some update happening in the background? I’ve raised an issue with the dev on websockets and I think he is pointing me to this problem being from the infura server…

github.com/aaugustin/websockets

Exception code = 1011

opened 07:32PM - 25 May 21 UTC

closed 04:13PM - 26 May 21 UTC

kaleb-keny

Hi @aaugustin I really tried looking all over for a solution to this issue i've recently started facing and i'm hoping you could help me out. I have the below script in a class that connects to a server and process messages from many sockets that are setup. The program needs to kill the sockets and generate new ones every `timerTopics` minutes: ``` async def wssSubscribe(self,payload): link,data=payload async with websockets.connect(link) as ws: #setup subscription await ws.send(data) message = await ws.recv() message = json.loads(message) subscriptionID = message["result"] try: #Get & process messages async for message in ws: message = json.loads(message)["params"]["result"] print(message) self.dumpToSql(responses=message) #On cancellation, cancel subscription except asyncio.CancelledError: message = self.baseProvider.encode_rpc_request(method='eth_unsubscribe', params=[subscriptionID]) await ws.send(message) await ws.recv() await ws.close() #On other exception except Exception as e: #Inform user print(f"Exception {e} seen while gathering topics wss") #Triggers task regensis self.activateTopicsHeal = True print(traceback.format_exc()) #Unsubscribe message = self.baseProvider.encode_rpc_request(method='eth_unsubscribe', params=[subscriptionID]) await ws.send(message) await ws.close() #Regenerate new tasks automatically async def taskGenerator(self): loop = asyncio.get_running_loop() try: while True: await asyncio.sleep(60 * (self.timerTopics - 1 )) self.tasksToKill = self.wssTasks.copy() self.wssTasks = list() for payload in self.payloads: task = self.wssSubscribe(payload) task = asyncio.ensure_future(task,loop=loop) self.wssTasks.append(task) await self.cancelWSSTasks(self.tasksToKill) except asyncio.CancelledError: print("cancellation happened!") ``` Now every now and then since the last 3 weeks I started getting this, showing up under general exception catch... ``` Traceback (most recent call last): File "C:\kaleb\data_manager\utils\gatherTopicsWSS.py", line 60, in wssSubscribe async for message in ws: File "C:\kaleb\miniconda\envs\keeper\lib\site-packages\websockets\protocol.py", line 439, in __aiter__ yield await self.recv() File "C:\kaleb\miniconda\envs\keeper\lib\site-packages\websockets\protocol.py", line 509, in recv await self.ensure_open() File "C:\kaleb\miniconda\envs\keeper\lib\site-packages\websockets\protocol.py", line 812, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: code = 1011 (unexpected error), reason = Internal logs subscription error ``` I've looked all over the web and stackoverflow and all i could find is that someone saying the ping/pong is killing my socket, tried setting it to None, the problem persists... do you think i have something deep in my script causing this that i should change... Forgot to mention, I am on ubuntu with websocket version 8.1 the latest... Thank you for your help 🙏🏼

sdan · June 2, 2021, 12:14am

I have been running into this too! websocket: close 1011 (internal server error): Internal logs subscription error

whenever i leave a websocket connection open with infura for a few hours (that i regularly ping)

Leiya_Kenney · June 7, 2021, 4:47pm

Hi @kaleb-keny and welcome to the Infura community, @sdan!

One thing to check is if your library supports automatic/manual Pong, and to make sure your context doesn’t cancel the client by timeout, particularly if you’re using Golang. If you can let us know what language you’re using, we can likely help find a more specific solution, but it sounds like the most likely cause is either the ping/pong isn’t set up correctly, or that something is being blocked by a firewall.

kaleb-keny · June 7, 2021, 5:16pm

Hi Leiya,

yeah i’m using python websockets library and the main snippet of my script is pretty simply:

import json
import websockets
import asyncio
import nest_asyncio
nest_asyncio.apply()

async def wssSubscribe(payload):
    
    link,data=payload
     
    async with websockets.connect(link) as ws:
        #setup subscription
        await ws.send(data)
        message = await ws.recv()
        message = json.loads(message)
        subscriptionID = message["result"]

        async for message in ws:
            message = json.loads(message)["params"]["result"]
            dumpToSql(responses=message)

where
link = 'wss://mainnet.infura.io/ws/v3/XXXXXXXX'
and
data is the subscription topic

This has happened mostly I think during gas intense times… And it is a fairly recent issue, where it didn’t occur throughout the time I used it in the past. But so far since gas went down, maybe infura infrastructure became less burdoned with these request and the error decreased considerably. I have asked the author of the websockets library and he told me that it is mainly coming from the server: "Your connection handler is throwing an exception. The exception message is: Internal logs subscription error"

Krasi_Georgiev · November 1, 2021, 6:35am

Also getting the same using golang so interested to get a reply from the support team.

In my definitely not a context cancelation.

Using the geth WatchLogs binding.