Besu returns empty trace on block 21030627

[dylanbutler1]

Actual behavior
Execute
curl -H "Content-Type: application/json" -d '{"id": 1,"jsonrpc": "2.0","method": "trace_block","params":["0x140e6e3"]}' localhost:8545

Result:
{"jsonrpc":"2.0","id":1,"result":[]}

Expected behavior
Expected response is a list of traces.

cc @joshuafernandes

[ahamlat]

@dylanbutler1 Which besu node are you using to test ? You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log
Limit of historical layers that can be loaded with BONSAI ...
It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

[racket2000]

For visibility, there was a similar issue affecting Erigon as well

I am not sure how to resolve this at the moment.

[macfarla]

This is expected behaviour for bonsai (default). You need to have a Besu forest archive node otherwise you will get an empty response as a Besu Bonsai full node can trace by default chain height - 512 blocks. You can test again and if you see this log
Limit of historical layers that can be loaded with BONSAI ...
It means that you're using Bonsai and it can not perform a trace_block call on a block that is older than chain height - 512

[ahamlat]

I tested with set_newHead on a bonsai node, to change the head near the block we want to trace, but it didn't work, the node had few issues

{"@timestamp":"2024-10-25T19:06:55,322","level":"WARN","thread":"vert.x-worker-thread-3","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -13956 old
       Old - hash: 0x48a7ab9c62ce82c50c2809475c6ff7077a9577a889e53517991b54ba98785b00, height: 21044611
       New - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655
  Ancestor - hash: 0x7bc8e8f0f548333aae54bc5a061b0dc02d931221ac13d6f0e3278aacc25e94e2, height: 21030655","throwable":""}
{"@timestamp":"2024-10-25T19:08:01,179","level":"WARN","thread":"vert.x-eventloop-thread-4","class":"EngineQosTimer","message":"Execution engine not called in 120 seconds, consensus client may not be connected","throwable":""}
{"@timestamp":"2024-10-25T19:08:09,504","level":"INFO","thread":"vert.x-worker-thread-7","class":"DiffBasedWorldStateProvider","message":"Archive rolling failed for block hash 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" java.util.NoSuchElementException: No value present\n\tat java.base/java.util.Optional.get(Optional.java:143)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:177)\n\
org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-10-25T19:08:09,505","level":"ERROR","thread":"vert.x-worker-thread-7","class":"BlockchainQueries","message":"failed worldstate query for 0xc8758a6bd1b3c1184ceaed1e451f3380698b6114c842f45a1ce9f3cedd41f24c","throwable":" org.hyperledger.besu.ethereum.trie.MerkleTrieException: invalid\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.rollMutableStateToBlockHash(DiffBasedWorldStateProvider.java:249)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.lambda$getMutable$4(DiffBasedWorldStateProvider.java:145)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.trie.diffbased.common.DiffBasedWorldStateProvider.getMutable(DiffBasedWorldStateProvider.java:145)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$getAndMapWorldState$24(BlockchainQueries.java:948)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.query.BlockchainQueries.getAndMapWorldState(BlockchainQueries.java:946)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.lambda$processTracing$1(Tracer.java:49)\n\tat java.base/java.util.Optional.flatMap(Optional.java:289)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.processor.Tracer.processTracing(Tracer.java:47)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.traceBlock(TraceBlock.java:99)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.lambda$resultByBlockNumber$0(TraceBlock.java:86)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.TraceBlock.resultByBlockNumber(TraceBlock.java:86)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.findResultByParamType(AbstractBlockParameterMethod.java:88)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterMethod.response(AbstractBlockParameterMethod.java:102)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)\n\tat org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)\n\tat org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)\n\tat java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)\n\tat org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)\n\tat io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)\n\tat io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}

cc. @garyschulte

[garyschulte]

#7821 adds the capability to move the worldstate along with the blockchain, to a specified block number. Since bonsai is not intended to serve as an archive node, this is a workaround that can be used until bonsai-archive feature is complete.

This should allow a trace of block 21030627 on nodes which have the trielog present (nodes which are not pruning their bonsai history), but it can be a foot-gun if used carelessly or incorrectly.