Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Besu returns empty trace on block 21030627 #7804

Open
dylanbutler1 opened this issue Oct 24, 2024 · 5 comments
Open

Besu returns empty trace on block 21030627 #7804

dylanbutler1 opened this issue Oct 24, 2024 · 5 comments
Labels
bug Something isn't working P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))

Comments

@dylanbutler1
Copy link

Actual behavior
Execute
curl -H "Content-Type: application/json" -d '{"id": 1,"jsonrpc": "2.0","method": "trace_block","params":["0x140e6e3"]}' localhost:8545

Result:
{"jsonrpc":"2.0","id":1,"result":[]}

Expected behavior
Expected response is a list of traces.

cc @joshuafernandes

@macfarla macfarla added bug Something isn't working P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc)) labels Oct 24, 2024
@ahamlat
Copy link
Contributor

ahamlat commented Oct 24, 2024

@dylanbutler1 Which besu node are you using to test ? You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log
Limit of historical layers that can be loaded with BONSAI ...
It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

@racket2000
Copy link

For visibility, there was a similar issue affecting Erigon as well

I am not sure how to resolve this at the moment.

@macfarla
Copy link
Contributor

This is expected behaviour for bonsai (default). You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log
Limit of historical layers that can be loaded with BONSAI ...
It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

@ahamlat
Copy link
Contributor

ahamlat commented Oct 25, 2024

I tested with set_newHead on a bonsai node, to change the head near the block we want to trace, but it didn't work, the node had few issues


{"@timestamp":"2024-10-25T19:06:55,322","level":"WARN","thread":"vert.x-worker-thread-3","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -13956 old
       Old - hash: 0x48a7ab9c62ce82c50c2809475c6ff7077a9577a889e53517991b54ba98785b00, height: 21044611
       New - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655
  Ancestor - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655","throwable":""}
{"@timestamp":"2024-10-25T19:08:01,179","level":"WARN","thread":"vert.x-eventloop-thread-4","class":"EngineQosTimer","message":"Execution engine not called in 120 seconds, consensus client may not be connected","throwable":""}
{"@timestamp":"2024-10-25T19:08:09,504","level":"INFO","thread":"vert.x-worker-thread-7","class":"DiffBasedWorldStateProvider","message":"Archive rolling failed for block hash 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" java.util.NoSuchElementException: No value present\n\tat java.base/java.util.Optional.get(Optional.java:143)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:177)\n\
org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-10-25T19:08:09,505","level":"ERROR","thread":"vert.x-worker-thread-7","class":"BlockchainQueries","message":"failed worldstate query for 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" org.hyperledger.besu.ethereum.trie.MerkleTrieException: invalid\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:249)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}

cc. @garyschulte

@garyschulte
Copy link
Contributor

#7821 adds the capability to move the worldstate along with the blockchain, to a specified block number. Since bonsai is not intended to serve as an archive node, this is a workaround that can be used until bonsai-archive feature is complete.

This should allow a trace of block 21030627 on nodes which have the trielog present (nodes which are not pruning their bonsai history), but it can be a foot-gun if used carelessly or incorrectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))
Projects
None yet
Development

No branches or pull requests

6 participants
@garyschulte @macfarla @ahamlat @racket2000 @dylanbutler1 and others