[HOLESKY PECTRA] onBlockImport accessing a missing state

[rolfyone]

{"@timestamp":"2025-02-26T05:53:00,005","level":"INFO","thread":"TimeTick-1","class":"teku-event-log","message":"Syncing     *** Target slot: 3720565, Head slot: 3717621, Remaining slots: 2944, Connected peers: 4","throwable":""}
{"@timestamp":"2025-02-26T05:53:03,575","level":"ERROR","thread":"forkchoice-async-0","class":"Subscribers","message":"Error in callback: ","throwable":" java.lang.IllegalArgumentException: Cannot get active validator indices from an epoch beyond the seed lookahead period. Requested epoch 116175 from state in epoch 116168\n\tat com.google.common.base.Preconditions.checkArgument(Preconditions.java:448)\n\tat tech.pegasys.teku.spec.logic.common.helpers.BeaconStateAccessors.getActiveValidatorIndices(BeaconStateAccessors.java:110)\n\tat tech.pegasys.teku.spec.logic.common.helpers.BeaconStateAccessors.getCommitteeCountPerSlot(BeaconStateAccessors.java:190)\n\tat tech.pegasys.teku.spec.logic.common.helpers.BeaconStateAccessors.lambda$getBeaconCommitteesSize$7(BeaconStateAccessors.java:325)\n\tat tech.pegasys.teku.infrastructure.collections.cache.LRUCache.get(LRUCache.java:57)\n\tat tech.pegasys.teku.spec.logic.common.helpers.BeaconStateAccessors.getBeaconCommitteesSize(BeaconStateAccessors.java:321)\n\tat tech.pegasys.teku.spec.Spec.getBeaconCommitteesSize(Spec.java:871)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.lambda$getCommitteesSizeUsingTheState$5(AggregatingAttestationPool.java:177)\n\tat java.base/java.util.Optional.map(Optional.java:260)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.getCommitteesSizeUsingTheState(AggregatingAttestationPool.java:177)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.getCommitteesSize(AggregatingAttestationPool.java:138)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.onAttestationIncludedInBlock(AggregatingAttestationPool.java:215)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.lambda$onAttestationsIncludedInBlock$8(AggregatingAttestationPool.java:211)\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\n\tat tech.pegasys.teku.statetransition.attestation.AggregatingAttestationPool.onAttestationsIncludedInBlock(AggregatingAttestationPool.java:211)\n\tat tech.pegasys.teku.statetransition.block.BlockImporter.lambda$notifyBlockOperationSubscribers$12(BlockImporter.java:230)\n\tat tech.pegasys.teku.infrastructure.subscribers.Subscribers.lambda$forEach$0(Subscribers.java:98)\n\tat java.base/java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4783)\n\tat tech.pegasys.teku.infrastructure.subscribers.Subscribers.forEach(Subscribers.java:95)\n\tat tech.pegasys.teku.statetransition.block.BlockImporter.notifyBlockOperationSubscribers(BlockImporter.java:229)\n\tat tech.pegasys.teku.statetransition.block.BlockImporter.lambda$importBlock$3(BlockImporter.java:165)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)\n\tat tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:147)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)\n\tat java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)\n\tat java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:1261)\n\tat java.base/java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1283)\n\tat java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)\n\tat tech.pegasys.teku.infrastructure.async.eventthread.AsyncRunnerEventThread.lambda$asSupplier$2(AsyncRunnerEventThread.java:139)\n\tat tech.pegasys.teku.infrastructure.async.eventthread.AsyncRunnerEventThread.recordEventThreadIdAndExecute(AsyncRunnerEventThread.java:130)\n\tat tech.pegasys.teku.infrastructure.async.eventthread.AsyncRunnerEventThread.lambda$doExecute$1(AsyncRunnerEventThread.java:113)\n\tat tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:80)\n\tat tech.pegasys.teku.infrastructure.async.AsyncRunner.lambda$runAsync$2(AsyncRunner.java:47)\n\tat tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:72)\n\tat tech.pegasys.teku.infrastructure.async.ScheduledExecutorAsyncRunner.lambda$createRunnableForAction$1(ScheduledExecutorAsyncRunner.java:124)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\t

way too much noise here.

[rolfyone]

ok basically we've got a block we've just imported, but we're getting the state from the target of the attestation. its super important we just use a state we have if we can, and in this specific case we're importing a block that contained the attestation so the post state from the attestation should have the data we need already.

It's a bit of jigging around so will come back to this if noone picks up, but its a legitimate issue, not just logging.

potentially onAttestationIncludedInBlock should get the state from the block slot and do everyhing based on that for electra attestations...

[rolfyone]

possibly one of the best candidates is head state if we're processing a block with the attestation, or even if we're close to head...

[tbenr]

I think we can even avoid putting attestations in the pool when they are older than 64 slots (from current slot). this happens while we are syncing and those attestations are not interesting.

[rolfyone]

yep 100%, if we're at head - they probably should fail this:

[IGNORE] aggregate.data.slot is equal to or earlier than the current_slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance) -- i.e. aggregate.data.slot <= current_slot (a client MAY queue future aggregates for processing at the appropriate slot).
[IGNORE] the epoch of aggregate.data.slot is either the current or previous epoch (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance) -- i.e. compute_epoch_at_slot(aggregate.data.slot) in (get_previous_epoch(state), get_current_epoch(state))

if we're outside the propagation slot range, we should ignore it coming in, it should never get processed... (rules as at deneb)
even if we're not at head we can trivially compute current slot and determine if they should just be ignored

[rolfyone]

The path here is different I guess because its onAttestationIncludedInBlock

We would be able to use the same state as the pre-state from the block (or the postState really) to verify that aggregate, as it made it into the block when being built.