Caplin: the epoch transition takes ~1 s under the fork-choice lock and runs on several paths #25136

Summary

On Hoodi (~1.14M active validators, 1.67M registry, Fulu), the epoch transition costs about one second on our test node (Xeon E-2288G), and it runs on several paths in the same slot. Two of these paths hold important locks while they run:

• Block import. The first block of each epoch runs ProcessEpoch and the following state-root hash inside AddChainSegment, under the fork-choice write lock f.mu.
• attestation_data. When the head is still in the previous epoch, the handler copies the head state and runs ProcessSlots across the boundary. It holds attCacheMutex exclusively and the synced-data read lock the whole time.

Most of the cost comes from two inefficiencies inside epoch processing, not from the spec work itself. The remaining cost is the same transition computed again on several state copies.

Code references are on origin/main 292d836.

This already happens on Fulu today (Hoodi, and mainnet has a similar validator count); Gloas makes it worse. One PR will not fix all of it, so this issue tracks the directions below:

• Direction 1: remove the two inefficiencies in rewards/penalties and effective-balance updates. Fixed by cl/transition/impl/eth2/statechange: avoid contended and no-op dirty marks in epoch processing #25144 (release/3.8: [r3.8] cl/transition/impl/eth2/statechange: avoid contended and no-op dirty marks in epoch processing #25163).
• Direction 2: attestation_data without a full state advance. Fixed by cl: answer attestation_data from fork choice without waiting for the head state #25359 (release/3.8: [r3.8] cl: answer attestation_data from fork choice without waiting for the head state #25458).
• Direction 3: gossip validation of the boundary block without a copy and ProcessSlots. Fixed by cl: validate boundary blocks without a state copy #25253 (release/3.8: [r3.8] cl: stop fsyncing fork-graph dumps, validate boundary blocks without a state copy, and republish moved heads and queued envelopes #25287).
• Direction 4: precompute the next-epoch state for the head (evaluate after 1-3). Fixed for block import by cl, cmd/caplin: precompute the next-epoch state off the fork-choice lock #25473 (release/3.8: [r3.8] cl: reduce head latency and fork-choice lock hold time #25704). The other paths do not need it: after cl: answer attestation_data from fork choice without waiting for the head state #25359, attestation_data no longer advances the state at the boundary, and payload preparation already advances its own copy outside f.mu.

Measurements

Test node (Hoodi, Fulu, probe build)

The fill can run twice for one boundary. The cache entry expires after half a slot, so if the head is still in the old epoch one slot later, the handler fills it again (for example slots 4190816 and 4190817, 1.08 s and 1.14 s).

Two inefficiencies cause more than half of the cost

1. Rewards and penalties. processRewardsAndPenaltiesPostAltair (process_rewards_and_penalties.go:106) runs on runtime.NumCPU() workers. Each worker writes balances through SetValidatorBalance → MerkleTree.MarkLeafAsDirty, and every call takes the tree's RWMutex read lock (merkle_tree.go:55). Readers do not block each other, but they all update the same reader count, so about 90% of the loop's CPU is cache-line contention on that counter. This applies when the balances Merkle tree is initialized, which is the normal case for a warm Fulu state; Gloas hashes balances without that tree. cl/transition: restore parallel rewards and inactivity epoch loops #23966 already noted this lock and left it out of scope.
2. Effective balance updates. ProcessEffectiveBalanceUpdates (process_effective_balance_update.go:48) calls SetEffectiveBalanceForValidatorAtIndex every time the hysteresis condition is true, even when the new value equals the old one. Validators that are already at the maximum and hold excess balance meet this condition in every epoch. On this state, 120,912 validator leaves become dirty, but only 9 effective balances change. About 70% of the post-epoch hash is spent rehashing these unchanged validator leaves.

Experiment in a scratch copy, not a PR:

• compute the reward/penalty deltas in parallel and apply them serially;
• skip the effective-balance write when the value is unchanged.

The post-epoch state root stays identical in all four combinations on the Fulu state. On the same state upgraded to Gloas, the root is also identical between the baseline and both fixes together. The state bytes are the same because eligible indices are unique and no worker reads a balance that another worker writes. What changes is callback behaviour: OnNewValidatorEffectiveBalance is no longer called for no-op writes, and OnNewValidatorBalance is called in a different order. The antiquary installs neither callback and computes history from the final state bytes (state_antiquary.go:264, :569).

So the block-import transition would drop from about 296 ms to 112 ms locally, roughly 0.85 s to 0.3 s on the node. With fewer threads (4) the result is the same, so the gain does not depend on the core count.

Where the same transition runs again

For one boundary on one chain, the transition from E to E+1 runs on these paths:

1. Block import, in place on the fork graph's current state, under f.mu (fork_graph_disk.go:335, called from on_block.go:435).
2. Gossip validation of the boundary block. computeBlockValidationContext copies (or reconstructs) the parent state under f.mu.RLock (block_service.go:718, forkchoice.go:831), then runs ProcessSlots after the lock is released (:743). It only needs the expected proposer, latest_block_hash and the latest bid. cl/phase1/network/services: read same-epoch parent state in place for gossip block validation #24523 removed the copy only for same-epoch parents. In Fulu and later, the parent state's proposer_lookahead already contains the proposer for the next epoch, and GetBeaconProposerIndexForSlot already reads it (cache_accessors.go:152). Exceptions: at the Fulu activation epoch the parent has no lookahead, and at the Gloas activation epoch UpgradeToGloas creates latest_block_hash and the initial bid, so the copy is still needed there.
3. attestation_data when the head is in the previous epoch (attestation_producer.go:165-181). It also advances one slot past the epoch start, which adds a full post-epoch hash. The target root is the same for start and start+1, and the source is the same as well. The cache is not checked again after attCacheMutex is taken, so concurrent misses each run the full fill, one after another.
4. Payload preparation and block production when we propose the first slot of an epoch (payload_preparation.go:582, block_production.go:846). These repeat the same work only while the head stays the same.

Gossip validation and import always repeat the boundary transition. The other paths are conditional. Checkpoint states advance only to the epoch start and are singleflighted and cached (forkchoice/utils.go:293), and normal one-epoch attestations reuse the head-state shuffling instead (on_attestation.go:63), so they are rarely an extra cost.

The attestation handler only needs the post-epoch current_justified_checkpoint. Fork choice already computes it for every block: OnBlock runs ProcessJustificationBitsAndFinality on the post-block state and stores the result as unrealizedJustifications[blockRoot] (on_block.go:551-561). When the head is in epoch E and the request is for E+1, this is the same computation on the same data for the same head. Empty slots do not change participation or balances. When more than one boundary is crossed, the value is different, and a fallback is needed. Today the cache is keyed by epoch only, so a cached entry can outlive a head change.

Shuffling and caches

• ProcessEpoch never computes the E+1 attester shuffle. The E+1 shuffle is needed later, for the second block of E+1 and for E+1 attestations. It hits ShuffledIndiciesCacheGlobal through the precache in postForkchoiceOperations (stages/forkchoice.go:533-605). A miss costs about 270 ms CPU locally.
• The proposer lookahead uses the E+2 active set. It cannot use the global cache because the cache key needs the block root at the state's current slot, and GetBlockRootAtSlot rejects that slot (raw/getters.go:360). The cost is small: about 14 ms locally on every path. This is the same key problem as in Caplin: Gloas performance at Hoodi/mainnet scale #25120 item 1. The first GetTotalActiveBalance of E+1 can still scan all validators, because nothing warms it before the boundary.
• Copy drops the per-state caches and rebuilds the pubkey index (copy.go:44). This is about 75% of the copy CPU and is tracked in cl/phase1/forkchoice: race on the fork graph's shared current state between block import and head-state readers #25111.

Proposed directions (ranked by gain/complexity)

1. Remove the two inefficiencies (high gain, low complexity, consensus-neutral).
   • Skip SetEffectiveBalanceForValidatorAtIndex when the value does not change.
   • Do not mark balance leaves dirty through a contended read lock from many goroutines. Either compute deltas in parallel and apply them in one pass, or make dirty marking lock-free or batched per chunk.
   • Risks: OnNewValidatorEffectiveBalance is no longer called for no-op writes and OnNewValidatorBalance order changes; no production consumer depends on either today. Add a regression test that compares post-epoch roots on a large state.
2. attestation_data without a full state advance (high gain for validators, low-medium complexity).
   • When the head is exactly one epoch behind the request, use the head block's unrealized justified checkpoint as the source and (E+1, head root) as the target. Otherwise fall back.
   • In the fallback, advance only to the epoch start, not start+1.
   • Do the work outside the synced-data read lock and outside the exclusive attCacheMutex, for example with singleflight.
   • Key the cached base data by (epoch, head root) instead of a 6 s TTL, and check the cache again after taking the lock.
   • Risks: prove the equivalence with tests against ProcessSlots (epoch-start head block, genesis epochs, multi-epoch gaps). Read the fork-choice value before taking attCacheMutex, so that no new lock-order edge is added (postForkchoiceOperations holds the MDBX write transaction and then takes attCacheMutex).
3. Gossip validation of the boundary block (medium gain, low complexity). In Fulu and later, except at the Fulu and Gloas activation epochs, read the proposer from the parent's proposer_lookahead and skip the copy and ProcessSlots. This extends cl/phase1/network/services: read same-epoch parent state in place for gossip block validation #24523. It saves one copy under f.mu.RLock and one epoch transition before the block is forwarded and imported.
4. Precompute the next-epoch state for the head (largest gain, medium-high complexity).
   • During the last slot of an epoch, advance a copy of the head state across the boundary outside f.mu, including the next state-root hash.
   • Share it with block import (swap it in under the lock when the parent is still that head), the attestation handler, and payload preparation/production.
   • Risks: one more full state in memory; wasted work when the head changes late in the slot; strict ownership, because the fork graph mutates currentState in place; and TransitionState must accept a state that is already at the block slot.

Gloas

Gloas adds ProcessPtcWindow, which reshuffles the full set for E+2 on every state that crosses the boundary. This is about 300 ms CPU locally and the largest single step (#25120 item 1, being fixed separately).

Directions 1-4 apply unchanged to Gloas:

• With the effective-balance fix, the Gloas post-epoch hash drops from 244 to 96 ms CPU locally.
• Every duplicate path in the list above also repeats the PTC shuffle until that fix lands, so removing duplicates is worth more after the fork.
• For gossip validation, the bid is already read before the advance, and ProcessEpoch does not change latest_block_hash. The existing copy path is still needed at the Gloas fork epoch.

Related

• Caplin: Gloas performance at Hoodi/mainnet scale #25120 (Gloas performance tracking; item 1 is the E+2 PTC shuffle; the copy cost is in cl/phase1/forkchoice: race on the fork graph's shared current state between block import and head-state readers #25111)
• cl/phase1/network/services: gossip block validation copies and advances the full parent state, adding ~0.4s to every block #24522 / cl/phase1/network/services: read same-epoch parent state in place for gossip block validation #24523 (gossip validation copied the parent state; fixed only for same-epoch parents)
• cl/transition: restore parallel rewards and inactivity epoch loops #23966 (restored the parallel rewards loop; left the MarkLeafAsDirty read lock out of scope)
• cl: bound payload preparation work and recover post-PTC retries #24119 (payload preparation does expensive state work close to the deadline; benefits from directions 1 and 4)