Hashes provided by newHeads subscription not lining up on Kovan

ngmiller · January 28, 2019, 9:20pm

Hey all,

Hoping I can get some clarity around why I’m seeing some odd behavior on the Kovan network.

Using go-ethereum, I’ve set up a websocket subscription for newHeads, and am simply printing the hash and parent hash of each header that comes through. But, when connected to kovan.infura.io, none of the hashes actually “line up”, i.e. the parent hash of a header is not equal to the reported hash of that parent. Hopefully this console output illustrates what I’m running into:

INFO[0000] 2019-01-28 21:10:42.9135011 +0000 UTC m=+0.076496001
INFO[0000] Subscribed to new block headers from wss://kovan.infura.io/ws/v3/MY-PROJECT-ID ...
INFO[0001] new block: 10215772
INFO[0001] (header) hash: 0xf45092d2a98576e5a8aad891535402d396b49f2ec47ca01cfb32e88037839fb2
INFO[0001] (header) parent hash: 0x2661ddd9185d6d8800ff3c041c2bc3c5dafac100a5ca7d11de57000cd222ac37
INFO[0002]
INFO[0018] new block: 10215773
INFO[0018] (header) hash: 0xe7f3493e179ff207a37a51987db7226e742d64da259f53b68d345cc58e438264
INFO[0018] (header) parent hash: 0xbc08c3fa91bf866abee2f6a7c838665bf4f97a32e7d0359e6549c8c149cc6b4b
INFO[0018]
INFO[0021] new block: 10215774
INFO[0021] (header) hash: 0xa484e90beacf3585ea4262e602b51c2d0c2ead4e373d04ccc9d50aeee7d18e18
INFO[0021] (header) parent hash: 0xcd931916f6213985b3a98ff0ae4d866c65a1090e994b7941d8647c99dd546fb6
INFO[0021]
INFO[0025] new block: 10215775
INFO[0025] (header) hash: 0x95ecf03250bedd9b360c5bed9e02dbef0140b989e8099879b501842323b292cf
INFO[0025] (header) parent hash: 0x73c01438b77345f3336a8d9a85b39cef08fff7bae7312edd783806972efb2d82
INFO[0025]

I suspect this has something to do with the fact that Kovan is PoA and not PoW? I posted here just in case this needs to be on Infura’s radar. I don’t think it’s a bug, I’m sure there’s just something I’m missing. Any help in understanding what’s going on here would be greatly appreciated.

Thanks!

egalano · January 28, 2019, 9:45pm

Thanks for reporting. Our team will look in to this immediately. Are you pulling data from kovan over our websocket interface? Or via the JSON-RPC API.

ngmiller · January 28, 2019, 10:16pm

Hey egalano. I’m listening over a websocket connection. Sounds like this is unexpected behavior?

ngmiller · January 28, 2019, 10:18pm

If it’s helpful, here’s the Go code I’m using for this test

package main

import (
    "context"

    "github.com/ethereum/go-ethereum/core/types"
    "github.com/ethereum/go-ethereum/ethclient"
    "github.com/ethereum/go-ethereum/rpc"

    log "github.com/sirupsen/logrus"
)

const HOST = "kovan.infura.io"
const PROJECT_ID = "MY-PROJECT-ID"

const WS_HOST = "wss://" + HOST
const WS_ENDPOINT = WS_HOST + "/ws/v3/" + PROJECT_ID

func main() {
    // Connect to node

    wsRpc, err := rpc.DialWebsocket(context.Background(), WS_ENDPOINT, WS_HOST)
    if err != nil {
        log.Error(err.Error())
        return
    }

    // Subscribe to new block headers
    wsClient := ethclient.NewClient(wsRpc)
    headers := make(chan *types.Header)

    sub, err := wsClient.SubscribeNewHead(context.Background(), headers)
    if err != nil {
        log.Error(err.Error())
        return
    }

    log.Info("Subscribed to new block headers from ", WS_ENDPOINT, " ...")

    for {
        select {
        case newHead := <-headers:
            // new block
            if err != nil {
                log.Error(err.Error())
            } else {
                log.Info("new block: ", newHead.Number.String())
                log.Info("(header) hash: ", newHead.Hash().String())
                log.Info("(header) parent hash: ", newHead.ParentHash.String())
            }

        case subError := <-sub.Err():
            // subscription error
            log.Error(subError)
        }
    }
}

mike · January 28, 2019, 11:32pm

Unfortunately the functionality in clients currently for a newHeads sub is they will not notify the you of a reorg, unless you match up the parenthash of the previous block to the new incoming block hash and detect it yourself.

We will be releasing an update in February that will fix this issue when using the Infura API. We will raise the issue to the client teams so they can address the issue for people not using the Infura API.

ngmiller · January 28, 2019, 11:48pm

So in the case I showed initially, there is a re-org happening every block? I’ve let that program run for 15+ minutes and none of the hashes ever lined up as I expected them to.

jee.choi · January 29, 2019, 12:24am

Hello,

I am currently running a newHeads subscription on the kovan network

XXXXX-Jee$ wscat -c wss://kovan.infura.io/ws
connected (press CTRL+C to quit)
> {"id": 1, "method": "eth_subscribe", "params": ["newHeads"]}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{"author":"0x00a0a24b9f0e5ec7aa4c7389b8302fd0123194de","difficulty":"0xfffffffffffffffffffffffffffffffc","extraData":"0xde830202078f5061726974792d457468657265756d86312e33312e31826c69","gasLimit":"0x7a1200","gasUsed":"0x2bdaf0","hash":"0x3e1527497799c36a2a5979996590c35ea3327785dcad9c42f17d7263e6b27f2c","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x00a0a24b9f0e5ec7aa4c7389b8302fd0123194de","number":"0x9be620","parentHash":"0x9c7f21b2c5edf682d45b2f89890e4bbd28b574bdd18ca88eb4d537c714ac7adc","receiptsRoot":"0xf562f8a55ebefc25412e64e049d302e0941ef43deed1655ac81f0cb726e6c887","sealFields":["0x1713e6ae","0xaa4528ef0c2b404ead84737b944372a710469bd0d1180509c1ad9fc99f098bcb2e0de5f45d9cdc59a201060435b213772a8bd1cdd0b4577bfcef978840bd5a0801"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","signature":"aa4528ef0c2b404ead84737b944372a710469bd0d1180509c1ad9fc99f098bcb2e0de5f45d9cdc59a201060435b213772a8bd1cdd0b4577bfcef978840bd5a0801","size":"0x24a","stateRoot":"0x7c8196ac997deea571fc8df56d729d8f70d2aafedb0fb15beb6f76c1a10927a6","step":"387180206","timestamp":"0x5c4f9ab8","transactionsRoot":"0x5b8991f92f90eac19ac60682420925aeee17d2671eb343024e18f59425aeec1f"},"subscription":"0x4bce26303bb7265b"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{"author":"0x0020ee4be0e2027d76603cb751ee069519ba81a1","difficulty":"0xfffffffffffffffffffffffffffffffb","extraData":"0xde830203008f5061726974792d457468657265756d86312e33312e31826c69","gasLimit":"0x7a1200","gasUsed":"0x50f656","hash":"0x0538a12781006bafb5823e81cfac91150ede62ed305e2a9695da0958575ec7a7","logsBloom":"0x00000000002000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000","miner":"0x0020ee4be0e2027d76603cb751ee069519ba81a1","number":"0x9be621","parentHash":"0x3e1527497799c36a2a5979996590c35ea3327785dcad9c42f17d7263e6b27f2c","receiptsRoot":"0x774469c24ad8d167b3d97cb77d529c4206d538fdaf62dd5ed61ede07462d0578","sealFields":["0x1713e6b2","0x43c2317c83b71b9bd00512a9bc0ca4884f467de78de0cb2e571a48e10671ccbc51a416b3014d1dcb0f2c36fa3244053bb7f8f6534e7607e2f3e38003b4d2b23601"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","signature":"43c2317c83b71b9bd00512a9bc0ca4884f467de78de0cb2e571a48e10671ccbc51a416b3014d1dcb0f2c36fa3244053bb7f8f6534e7607e2f3e38003b4d2b23601","size":"0x24a","stateRoot":"0x0fb2e704dd4124672cedb9effa8dc60d86c11101bc2f3aefadf5f17a0fc73cdc","step":"387180210","timestamp":"0x5c4f9ac8","transactionsRoot":"0x0e1b62258abaef03c9e0c60482ae716003376b48e0a55b5f33bb0245c8fc662c"},"subscription":"0x4bce26303bb7265b"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{"author":"0x0010f94b296a852aaac52ea6c5ac72e03afd032d","difficulty":"0xfffffffffffffffffffffffffffffffe","extraData":"0xde830202078f5061726974792d457468657265756d86312e33312e31826c69","gasLimit":"0x7a1200","gasUsed":"0x10ca7b","hash":"0xd12e8427771a2485988db97798a50c96d88571672f38e2cbad1ed5145d0a401b","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0010f94b296a852aaac52ea6c5ac72e03afd032d","number":"0x9be622","parentHash":"0x0538a12781006bafb5823e81cfac91150ede62ed305e2a9695da0958575ec7a7","receiptsRoot":"0xd6c36336640b5ed591262389a9591f373e6e2bbbbac7d21f5ae87a22cb2eca07","sealFields":["0x1713e6b3","0xa0c666f51b4a512c9aa0dbe43a2142a6d69b41407b8dc43ce18f2d95ee1d88554c9346cd0e5489d95b4c6dbb5e3f312645f18a8684f292f5ae8a110aeac2948a01"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","signature":"a0c666f51b4a512c9aa0dbe43a2142a6d69b41407b8dc43ce18f2d95ee1d88554c9346cd0e5489d95b4c6dbb5e3f312645f18a8684f292f5ae8a110aeac2948a01","size":"0x24a","stateRoot":"0x83fa40b3a1f43b77c322d6692c57189bda7b5e8aaf58563a2f26ca6cc1e3d00e","step":"387180211","timestamp":"0x5c4f9acc","transactionsRoot":"0xe16b8786a4866bc175627f142a1a4482f80367cc4ec18c1723d8d135c95c6458"},"subscription":"0x4bce26303bb7265b"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{"author":"0x007733a1fe69cf3f2cf989f81c7b4cac1693387a","difficulty":"0xfffffffffffffffffffffffffffffffe","extraData":"0xde830203008f5061726974792d457468657265756d86312e33312e31826c69","gasLimit":"0x7a1200","gasUsed":"0xf3748","hash":"0x072a16dd26bfb8c073fbb2789a60fffdf4b7b5abfd0c2d1a046a1c766402406a","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x007733a1fe69cf3f2cf989f81c7b4cac1693387a","number":"0x9be623","parentHash":"0xd12e8427771a2485988db97798a50c96d88571672f38e2cbad1ed5145d0a401b","receiptsRoot":"0x3ba6d2ea1043d4cfcd41eb0321829f9465b9ab931407d373885e2d3447914e85","sealFields":["0x1713e6b4","0xddf5b27caea63d0bff302958b04df0b99e607cd371e5b03410c25cb8d5157e6667e6f544098313dd81897a98dc2f1e4b634d1842f12adddfa3e66ce56d4a6bbc00"],"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","signature":"ddf5b27caea63d0bff302958b04df0b99e607cd371e5b03410c25cb8d5157e6667e6f544098313dd81897a98dc2f1e4b634d1842f12adddfa3e66ce56d4a6bbc00","size":"0x24a","stateRoot":"0x1200455d5fe22db7c92b98d265a1a0588220c8737930d84831e24d8a31584a68","step":"387180212","timestamp":"0x5c4f9ad0","transactionsRoot":"0x5befa1bdd0df3f06ed20b5b9a9e6fe726e4e037e42c72c976835b49afa0a4fb3"},"subscription":"0x4bce26303bb7265b"}}

as you can see from the results, the hashes are mapping up correctly
the first 3 blocks from subscription:

“number”:“0x9be620”,
“hash”:“0x3e1527497799c36a2a5979996590c35ea3327785dcad9c42f17d7263e6b27f2c”,
“parentHash”:"0x9c7f21b2c5edf682d45b2f89890e4bbd28b574bdd18ca88eb4d537c714ac7adc”

“number”:“0x9be621”,
“hash”:“0x0538a12781006bafb5823e81cfac91150ede62ed305e2a9695da0958575ec7a7”
“parentHash”:“0x3e1527497799c36a2a5979996590c35ea3327785dcad9c42f17d7263e6b27f2c”

“number”:“0x9be622”,
“hash”:“0xd12e8427771a2485988db97798a50c96d88571672f38e2cbad1ed5145d0a401b”
“parentHash”:“0x0538a12781006bafb5823e81cfac91150ede62ed305e2a9695da0958575ec7a7”

are you running the currently a newHeads subscription and they are not matching up, or are you referring to when you were getting the results above? I will try and check throughout the day, but the random times I have check today, they all seem to match up. (as mentioned by mike, in the case of a reorg, you will not receive the reorg’d block, so in case the multiple blocks have reorgs in a row, you will not get matching hashes)

ngmiller · January 29, 2019, 5:03pm

Hi Jee,

I just ran my test program again this morning for about 5 - 6 minutes and still did not get any header hashes to line up correctly. The only difference I can think of between my test program and your run with wscat is that I am using the new v3 endpoints that require a project ID in the URL path. I suspect those are hitting different sets of nodes? Specifically, I am using wss://kovan.infura.io/ws/v3/PROJECT_ID

I also have not ruled out the possibility that this is something weird in go-ethereum, but when I connect to mainnet.infura.io, or my own node, all the hashes line up correctly.

ngmiller · January 29, 2019, 5:07pm

Just tried the same setup with wss://kovan.infura.io/ws instead (no project ID), and still am not getting correct hashes. Unless the Infura team can confirm otherwise, this is looking like something odd in go-ethereum. I will open an issue with them just in case.

jee.choi · January 29, 2019, 5:45pm

Hello,

Have you tried running running wscat -c wss://kovan.infura.io/ws and subscribing to newHeads and comparing the hashes (like I did above). I just checked right now and I am still getting the correct hashes. Running wscat can confirm if the issue is the Go code you are running

ngmiller · January 29, 2019, 6:02pm

Yeah wscat is fine on both endpoints (with or without project-id). Definitely a go-ethereum issue

Ryan_Schneider · January 30, 2019, 5:23pm

I can confirm this is an issue in go-ethereum/ethclient, specifically that rather than using the return “hash” value from the JSON payload the ethclient is computing it’s own Hash() by re-computing the RLP hash of the recreated block. However, it appears that approach is not compatible w/ Kovan’s Aura POA consensus. I haven’t dug into the details but my guess is that Aura contains one or more additional fields that ethclient is not including when computing the hash.