Repository navigation
[r3.8] cl/phase1/forkchoice: keep vote weights across get_head calls and apply votes when recorded - #25711
Merged
Conversation
## Problem `Signature.VerifyAggregate` used blst `FastAggregateVerify`. It sums the public keys with `P1Aggregate.Aggregate`, which starts GOMAXPROCS goroutines and makes one cgo call plus one `runtime.Gosched` per key. On mainnet, each Electra block carries aggregates with about 25-30k keys in total, and they are verified inside the block state transition under the fork-choice lock. With 32k keys the key aggregation alone takes about 13-16 ms on an M3 Pro, and more when the cores are busy. ## Change - Sum the keys with blst batch addition (`P1Affines.Add`, which calls `blst_p1s_add`), then verify once against the aggregate key. The flags are the same as inside `FastAggregateVerify`: the signature is group-checked, the keys are not validated again (they were validated when decoded). An empty key list and a nil key still return false. - At most 512 keys go into one cgo call. The chunks are summed on up to GOMAXPROCS goroutines, each with its own buffer, and the partial sums are added in Go. Why 512: `blst_p1s_add` puts its scratch buffer on the C stack (`alloca` of `min(n, 1024)` points at 144 bytes each, so up to `SCRATCH_LIMIT` = 144 KiB), see supranational/blst#142. Go starts cgo threads with the libc default thread stack size, which is 128 KiB on musl. 512 points need 72 KiB of scratch. A compile-time check keeps the chunk scratch below the musl thread stack. Which builds are affected: glibc builds (8 MiB thread stacks) are not. Of our images, only `debug.Dockerfile` (Alpine) is musl. The root `Dockerfile` is Debian, CI builds the Hive image from it, hive's own erigon Dockerfiles are Debian too, and hive runs erigon with `--externalcl`. On musl, passing all keys to one `P1Affines.Add` call crashes with SIGBUS at 2048 keys under GOMAXPROCS=1. With more procs, blst splits the call into at most GOMAXPROCS slices of at least 512 points, so 32768 keys still give slices of more than 1024 points (the full 144 KiB of scratch) with fewer than 32 procs. Callers that benefit: block attestation verification, attester slashings (gossip, block, Beacon API pool), and the sync aggregate (512 keys or fewer, so one call without goroutines). Gossip aggregates and sync committee contributions use `AggregatePublickKeys` with `VerifyMultipleSignatures` and are not changed. ## Results `BenchmarkSignatureVerifyAggregate`, 32768 keys, Apple M3 Pro: | | `main` | this PR | |---|---|---| | GOMAXPROCS unset (12) | 15.8 ms | 1.8 ms | | GOMAXPROCS=4 | 11.4 ms | 2.5 ms | Each worker allocates one buffer of up to 512 keys (48 KiB), so a 32768-key call allocates about 200 KB with 4 workers and about 600 KB with 12. The same benchmark on the machine of the mainnet dev node (Intel Xeon E-2288G, 16 threads), run with `nice -n 19` next to the running node. The old and new test binaries ran alternately, six times each; the table shows the median: | Keys | `main` | this PR | |---|---|---| | 1 | 0.77 ms | 0.77 ms | | 128 | 0.90 ms | 0.79 ms | | 513 | 1.10 ms | 0.92 ms | | 2048 | 1.65 ms | 1.15 ms | | 32768 | 12.5 ms | 2.6 ms | For 2048 and 32768 keys the ranges of the six runs do not overlap. With 16 workers, a 32768-key call allocates about 800 KB. Mainnet dev node: block attestation verification takes 9 ms at p50 and 11 ms at p90, for 25.4k keys per block at p50. That run had no baseline with the old code; the benchmark above is the same-machine comparison. The fork-choice lock hold after `newPayload` went from 50 to 44 ms at p50 compared to the previous day, which is within day-to-day noise, so I do not claim more than about 10 ms per block from the node data. ## Tests - The aggregate verification gives the same results for 1 key and for 511, 512, 513, 1024, 1025 and 4096 keys; a wrong message, a missing key, an extra key and a nil key return false; empty key lists keep the existing behaviour. - With 16 or more keys (blst's batch path): an adjacent duplicate pair that reaches the doubling branch (18 keys), an adjacent P/-P pair, a P/-P pair in a 32-key batch whose infinity result goes into the next tree round, and a cancelling pair across a 512-key chunk boundary. - With GOMAXPROCS=2, so that one worker sums several chunks (1025 and 2049 keys). - Benchmarks for `bls.VerifyAggregate` (with the key cache warm) and `Signature.VerifyAggregate`. The tests and benchmarks share one lazily built set of keys, and the 32768-key benchmark size runs only outside `-short`, so `make test-bench` does not sign 32768 keys. These tests pass on `main` as well, because the change does not alter results. The musl stack limit has no test in the repository, because it needs a thread with a small stack. I checked it with a cgo probe that calls `Signature.VerifyAggregate` from a pthread with a 128 KiB stack and GOMAXPROCS=1: a version that passes all keys to one `P1Affines.Add` call crashes with SIGBUS at 2048 keys, while `main` and this PR pass at 1024, 2048 and 8192 keys. I did not run it on a real Alpine host or inside `debug.Dockerfile`.
…ad state (#25539) ## Problem On Gloas the ForkChoice stage runs again for the same head root when the head's payload status changes from EMPTY to FULL (`GetExecutionHead` changes and `sleep_for_slot.go` wakes the stage), and again about once per second while the forkchoice update returns busy. Each run calls `postForkchoiceOperations`, which dumps the head state with `DumpBeaconStateOnDisk`. The envelope does not change the beacon state, so every extra run encodes and rewrites the same state for the same root. A head-state dump takes about 405 ms p50 on a mainnet node with the current dump code (no fsync since #25251), and it holds `stateDumpLock`, which disk state reads (replays) also need. By code reasoning this can add up to 8 redundant dumps per epoch on Gloas (one per dump slot, more while the forkchoice update is busy). There is no Gloas mainnet data for this yet. ## Fix `forkGraphDisk` remembers the block root and slot of its last successful state dump, guarded by `stateDumpLock`. `DumpBeaconStateOnDisk` returns early when both match. The slot is part of the key because a state advanced through empty slots keeps its block root but has different content. The marker is cleared before each write and set only after the write finished and the file closed without error, so a failed or panicked write is retried (stage panics are recovered, so the next run would otherwise skip a truncated file). ## Tests New tests in `fork_graph_test.go` count state-file writes through the existing failing-fs wrapper: - the same root and slot dumped twice writes once (red on main: 2 writes); - a close error is retried (`TestDumpBeaconStateReturnsCloseError`); - a panic during the write is retried (`TestDumpBeaconStateRetriesAfterPanic`); - a different root is still written; - the same root at a different slot is still written, and the file holds the later state; - a failed later-slot write does not block a new dump of the earlier slot. Each test fails when the production branch it targets is removed. ## Out of scope - `stateDumpLock` contention between state dumps and replays in general, and making the state dump atomic (temp file + rename). - Avoiding the repeated ForkChoice stage run itself. Related to #25120 and #25136.
…ock (#25473) Part of #25136 (direction 4, block import only). ## Problem The first block of each epoch runs the epoch transition inside `AddChainSegment`, under the fork-choice write lock `f.mu`. Head updates, attestation processing and `get_head` wait during that time. On a mainnet node on `main` (24 h, 2026-10-27 to 2026-10-28), this transition took 467 ms p50. At the boundary, `f.mu` was held 491 ms after `newPayload`, compared to 61 ms in the slot before. Public Xatu head events for 2026-10-27 show the effect. We compared each slot with the per-slot median of each client: - Normal slots: our node is within about 0.1 s of Lighthouse, Prysm, Teku and Lodestar. - First slot of an epoch: these clients update the head 0.3-0.5 s earlier than our node. That matches the transition time. Directions 1-3 of #25136 are merged. This PR does the remaining part for block import. ## Change - `ForkChoiceStore.RunNextEpochPrecompute` runs in a goroutine started by `RunCaplinService`. At 2/3 of the last slot of each epoch, if the node is synced and the head is in that epoch, it copies the head state, runs `ProcessSlots` to the first slot of the next epoch and hashes the state. None of this holds `f.mu`. - The result goes to the fork graph with `SetNextEpochState(parentRoot, state)`, keyed by the head root. The fork graph keeps one entry. - The first `AddChainSegment` that reaches the state transition with a block at or after the entry's slot takes the entry. It uses it only when the block's parent root matches and full validation is on, and drops it otherwise. An import that stops before the transition (for example `MissingSegment`) leaves the entry in place; it is still correct for any later child of the same parent. - With the entry, the import calls the new `TransitionStateFromPrecomputedEpochState`. It runs only the block part when the state is already at the block slot, and the normal `TransitionState` otherwise. - If that fails, the import falls back to the normal transition on the parent state, so a wrong precomputed state cannot reject a valid block. A failure only on the precomputed path logs `Precomputed epoch state diverged` at Warn. - `AddChainSegment` records the highest block slot that reached the state transition, valid or not. `SetNextEpochState` ignores a state at or below that slot, so a precompute that finishes after the boundary block was imported is dropped. The precompute stops without publishing once ctx is cancelled. - `caplin_next_epoch_state_outcomes_total{outcome}` counts what happens to the precomputed state. At an import: `hit`, `no_entry` (a block that crosses an epoch and finds no entry, also while the node is not synced and no precompute runs), `parent_mismatch`, `no_full_validation`, `diverged`, `invalid`. At publication: `late`. Each outcome is also logged at Debug, except `diverged`, which stays at Warn. - `machine.TransitionState` is split into `ProcessSlots` and `transitionBlock`. Its behavior does not change. Not in this PR: payload preparation, block production, checkpoint states, gossip validation and the Beacon API do not use the precomputed state. After #25359, `attestation_data` no longer advances the state at the boundary. Payload preparation already advances its own copy before the slot, outside `f.mu`. ## Why 2/3 of the last slot From the mainnet node's logs over 87 epochs: - The head at 2/3 (8 s) of the last slot was the parent of the next epoch's first imported block in 87 of 87 epochs. The head never changed between that point and the boundary block. - Starting when the last slot's block becomes the head would have missed one epoch, where the last slot was empty. - The earliest boundary block arrived 4.7 s after the trigger. The precompute takes about 0.5 s. On the canonical chain, 998 of the last 1000 epochs had a block in the last slot. These numbers use the fork-choice head. The precompute reads the head from synced data, which is published at the end of the fork-choice pass, so a last-slot block imported just before the trigger but not yet published is a miss. The node measurement below will show the real hit rate. ## Cost - In the background, once per epoch: one copy of the head state (taken under the synced-data read lock), one epoch transition and one state hash. - One extra beacon state in memory from the trigger until the boundary import, about 4-5 s per epoch. ## Tests - `cl/transition`: - The precomputed path gives the same state, state root and rewards as the normal path at every epoch boundary of the Bellatrix test chain. - A state ahead of the block is rejected. - `TestTransitionStateRejectsBlockAtStateSlot` guards the `TransitionState` split and passes on `main`. - `fork_graph`: the matching entry is used, also when empty slots remain before the block (Electra fixture: entry at 8288, block at 8290); a wrong parent, a missing full validation and an invalid block drop it; a missing segment keeps it; a corrupted entry falls back to the normal transition; a late publication is ignored, also after an invalid block; each import or late publication counts exactly one outcome, and a block that does not cross an epoch counts none. - `forkchoice`: the target epoch calculation, the trigger slot of the loop, the synced and head-epoch gates, the published state is used by `OnBlock`, and no publication after cancellation. - Spectests: `sanity/blocks`, `finality` and the transition `core` vectors that cross an epoch also run the precomputed path. Both paths must agree on whether the block is valid, and when it is valid they must produce the same state. Error messages are not compared, because the two paths can fail with different messages for the same invalid block. ## Measurements Mainnet dev node, same machine and datadir, about 24 h per run: - without: `main` 85258cd + #25359 + #25258 + #25274, 2026-10-27 07:31 to 2026-10-28 11:58 UTC; - with: `main` cdf7da1 + this PR + #25423 + #25274, 2026-10-28 12:00 to 2026-10-29 13:12 UTC. The base moved between the runs, but none of the other changes touch the epoch transition. The first row is measured directly in `AddChainSegment`. | | without | with | |---|---|---| | Epoch transition under `f.mu`, first slot of an epoch, p50 / p90 | 463 / 528 ms | 45 / 56 ms | | EL "head validated", first slot of an epoch, p50 (time after slot start) | 2.58 s | 2.13 s | | First slots of an epoch with "head validated" after 3.5 s | 25% | 12% | | EL "head validated", other slots, p50 | 1.87 s | 1.89 s | | `attestation_data` latency, first slot of an epoch, p90 / p99 | 46 / 470 ms | 0 / 25 ms | Precomputed state: used in 215 of 224 epochs, not available in 9, and never diverged. In most of the 9 misses, the first block of the epoch arrived late and used the entry, and the next block was built on the same parent, so no entry was left for it. Per-slot comparison with other clients, from the public Xatu `beacon_api_eth_v1_events_head` data. For each slot, the value is the client's median head time minus the Caplin node's head time; positive means Caplin is earlier. Without: 2026-10-27 07:32-24:00 UTC; with: 2026-10-28 12:01-24:00 UTC. | Client | first slot of an epoch, without | first slot of an epoch, with | other slots, without | other slots, with | |---|---|---|---|---| | Grandine | -0.676 s | -0.240 s | -0.227 s | -0.231 s | | Teku | -0.454 s | -0.009 s | -0.118 s | -0.095 s | | Lighthouse | -0.395 s | +0.040 s | -0.068 s | -0.067 s | | Lodestar | -0.467 s | -0.021 s | -0.081 s | -0.067 s | | Prysm | -0.261 s | +0.205 s | -0.082 s | -0.074 s | | Nimbus | -0.375 s | +0.062 s | +0.049 s | +0.060 s | On other slots the difference stays almost the same, so the network conditions were similar. On the first slot of an epoch, the node moved from behind all clients to level with Teku and Lodestar.
…ead-state copy (#25543) `postForkchoiceOperations` copies the full head state with `GetOwnedStateAtBlockRoot` before it sends the `head` and `head_v2` events. On a mainnet node this copy takes about 85-90 ms at p50 (`caplin_forkchoice_current_state_copy`), so every head event reaches subscribers about 90 ms after the EL has validated the head block. Public Xatu data for 2026-10-27 shows other consensus clients send `head` almost at the same time as the `block` event. Caplin 3.8.1 sends it about 210 ms later; on `main` most of the remaining gap is this copy. - New `ForkChoiceStore.ViewCurrentStateAtBlockRoot`. It runs the callback on the fork graph's current state under `f.mu.RLock`, and only when that state belongs to the given root. Unlike `ViewStateAtBlockRoot`, it never replays a state from disk. - `postForkchoiceOperations` first tries to send the events before the copy: - take a head snapshot (root, slot, payload status, optimistic flag); - read the current state through the new method: the state root comes from `PeekPreviousStateRoot`, which a verified transition sets to `block.StateRoot`, so no `HashSSZ` is needed; version and dependent roots come from the existing `BuildHeadV2Data`; - send through the existing `emitHeadEventsIfCurrent`, which checks root, slot, payload status and the optimistic flag again under the head-event lock. - If the early events were sent, the post-copy path does not send them again. Otherwise (the head is not the current state, the saved state root is zero, the head changed, or an error) the existing post-copy path runs as before and logs any error once. - `payload_status` now follows the fork of the head block in both paths. The Beacon API says: "For pre-Gloas head blocks, `payload_status` must be `full`." The post-copy path used the state version, so with an advanced checkpoint anchor across the Gloas fork (state at the first Gloas slot, latest block in the last Fulu slot) it reported `empty` or `pending`. - `emitHeadEventsIfCurrent` now also returns whether it sent the events. - The post-copy path now also takes `state` from `PeekPreviousStateRoot` and calls `HashSSZ` only when the saved root is zero. Before, it used `HashSSZ`, which for a Gloas head with an applied envelope is the post-envelope root, while the early path and the envelope path report the block's state root. The view callback does not call fork choice, so the deadlock contract in `interface.go` holds. - State-backed endpoints (`/eth/v1/beacon/states/head/...`, duties) keep serving the previous head for about one copy duration (~85 ms) after the head event. Before this change that window was about 30 ms. Attestation data and block production read the fork-choice head and are not affected. - If the copy fails after the early events were sent, the events are already out. Before, no event was sent in that case. - Gloas only: if the payload status of the head changes between the snapshot and the send, the early events are dropped and the post-copy path sends them. The envelope path can then send the same `head_v2` "full" event again. Duplicates of this kind already happen in other orderings. Mainnet dev node, same machine, 13:15-06:40 UTC (17.4 h) on two consecutive days. The previous day ran without this change. The run with this change also includes #25542 (BLS key aggregation), which only affects the time before "head validated". | | without | with | |---|---|---| | EL "head validated" to head event, p50 / p90 | 88 / 109 ms | 0 / 1 ms | | Head event, normal slots, p50 (time after slot start) | 1.987 s | 1.914 s | | Head event, first slot of an epoch, p50 | 2.255 s | 2.176 s | | EL "head validated", normal slots, p50 | 1.890 s | 1.913 s | | Early path used | - | 5210 of 5216 | Per-slot comparison with other clients, from the public Xatu `beacon_api_eth_v1_events_head` data for 2026-10-29. On that day the node ran without this change from 00:00 to 13:06 UTC and with it from 13:15 to 24:00, so both halves share the same day. For each slot, the value is the client's median head time minus the Caplin node's head time; positive means Caplin is earlier. Normal slots, median over slots: | Client | without | with | |---|---|---| | Grandine | -0.177 s | -0.095 s | | Teku | -0.065 s | +0.020 s | | Lighthouse | -0.019 s | +0.052 s | | Lodestar | -0.027 s | +0.049 s | | Prysm | -0.021 s | +0.044 s | | Nimbus | +0.101 s | +0.167 s | The "head validated" time against Teku was almost the same in both halves (+0.026 s and +0.020 s), so the gain comes from this change and not from network conditions. On the first slot of an epoch, the node is also second after Grandine (-0.193 s before, -0.097 s after). - The head events are sent before the head-state copy finishes. The test blocks the copy and fails on `main`. - The early event and the post-copy event are identical, using states from a real fork graph: pre-Gloas, an advanced anchor across the Gloas fork (`full`), the first Gloas head slot, and a later Gloas slot. - Fallback to the post-copy path when the head is not the current state and when the saved state root is zero; exactly one pair of events per pass; no event when the head changes before the send. - The post-copy path reports the block's state root for a Gloas state changed by the envelope, and still hashes the state when no root is saved. - `ViewCurrentStateAtBlockRoot`: the callback is not called for a non-current root, it holds the read lock until the callback returns, and it waits for a writer.
…iew (#25646) ## Problem `payloadAttestationValidationContext` builds the per-block-root context used to validate PTC messages (slot, domain, PTC positions, PTC public keys). It got the state with `GetStateAtBlockRoot`, which copies the whole current beacon state under `f.mu.RLock`. At mainnet validator counts (1.14M) a fresh state copy is about 13-15 ms and allocates about 300 MB (local benchmark of CachingBeaconState.Copy on a synthetic state); it happens about once per Gloas block root, and the read lock blocks fork-choice writers (block import, PTC vote application) for the whole copy. ## Fix Build the context inside `ViewStateAtBlockRoot` and keep only owned data: the positions map, public keys copied into `Bytes48`, and the domain copied into a `Hash`. Under the read lock the work is now O(PTC size), not O(registry). - Non-current roots still replay outside the lock, as before. - Error classification is unchanged: view errors (replay failure, missing state) still wrap `ErrIgnore`, so gossip keeps IGNORE for them; build errors (out-of-range PTC member, PTC length, domain) are returned unchanged and stay REJECT. - The singleflight and cache logic is unchanged. ## Tests This is a behavior-preserving performance change. `TestPayloadAttestationValidationContextBuildsFromState` is the red test: on main its current-state row fails because the current-root path makes one state copy. Its replayed-state row and the other new tests characterize behavior that is unchanged and also pass on main: - a failed replay is `ErrIgnore` and wraps the cause; - a slot mismatch is `ErrIgnore` + `errPayloadAttestationSlotMismatch` on both the current and the replayed path (this guard had no test before); - duplicate PTC members keep every position and one public key; - an out-of-range PTC member is not `ErrIgnore`. Mutation checks: reverting to `GetStateAtBlockRoot`, returning the build error through the view, dropping the `ErrIgnore` wrap, and dropping the slot-mismatch guard each turn at least one test red.
…es in parallel (#25596) ParticipationBitList.HashSSZ rehashed the whole list on one goroutine each time a block changed participation flags. With ~2.1M validators that is ~65k chunks, and it is the slowest field of the per-block state root (VerifyTransition); beaconStateHasher already hashes dirty fields concurrently, so the slowest field sets the wall time. Hash subtrees of 4096 chunks concurrently, each in its own region of the list's hash buffer, then merkleize the subtree roots. Each call runs at most GOMAXPROCS subtrees at a time; when both participation lists are dirty, the two calls run side by side. On a 2.1M-entry list, subtree sizes of 1024-4096 chunks give the same wall time and 8192 or more are slower, so 4096 keeps the goroutine count lowest. An incremental tree does not fit this list: a block's ~25k attesters are spread across the registry and touch about a third of the chunks, and the incremental tree hashes pair by pair instead of in batches. This also fixes a latent stale-tail issue. The old code hashed the bytes from the logical length up to the next chunk boundary past it (a whole extra chunk when the length is a multiple of 32), so the root was non-canonical when those bytes were not zero: after UnmarshalJSON reused a larger buffer for a shorter list, or after Set wrote past the logical length into the slack of NewParticipationBitList. The new code reads only the logical length and zero-fills the tail. No production path is known to reach either case. The old code also allocated a padded copy of the whole list on every hash when the backing slice had no spare capacity (always for lists from DecodeSSZ, about 2.6 MB per hash at 2.1M entries). The new code reuses its hash buffer. Costs and scope: - The retained hash buffer grows from ~l/2 to ~l bytes (about +1 MB per hashed list at 2.1M validators), because each region is copied into the buffer before hashing. - Gloas states hash participation with HashSSZProgressive, which this change does not touch, so there is no gain after Gloas. Benchmarks, M3 Pro, old and new binaries run interleaved at load average 4-8, median: BenchmarkParticipationBitListHash-12 (2_100_001 entries, 25k updates outside the timer): 2.29 ms -> 0.39 ms BenchmarkParticipationBitListHash -cpu 1: 2.30 ms -> 2.31 ms BenchmarkHashSSZAfterBlock-12 (2.1M-validator Fulu state, one block's field updates outside the timer): current_epoch_attestations 2.34 ms -> 1.45 ms half_previous_epoch_attestations 2.39 ms -> 1.48 ms The state root is now bounded by the balances tree, whose root scans every node. On the mainnet node the gain is about 1 ms (see Node results). BenchmarkHashSSZAfterBlock builds a fresh state per case (~1 GB peak RSS) and skips in -short mode. TestParticipationBitListHashSSZMatchesSha256Oracle checks the root against a crypto/sha256-only oracle for lists built by NewParticipationBitList+Set, Set past the length, Append, DecodeSSZ, Copy, CopyTo, Clear+shorter Append, reused-buffer UnmarshalJSON and rehash after Set. Lengths are 1, 31 and 33 bytes, one full subtree, just past two subtrees (three-chunk last subtree), exactly three subtrees with a partial (31-byte) last chunk (`3*subtreeBytes-1`), exactly five subtrees, and 2.1M. The two stale-tail cases fail on the old code. The other cases, and TestParticipationBitListHashSSZSmallCapacity, pass before and after: they are safety-net tests for a change that keeps the root otherwise unchanged. ## Node results Mainnet node (Xeon E-2288G, Erigon with embedded Caplin, no validators). Two runs at the same hours (09:40-17:40 UTC, about 2390 slots each): before = main from 2026-10-30 plus the other Caplin performance PRs; after = the same plus #25594, #25595 and #25596, measured with fork-choice lock probes that are not part of this PR. | | before (10-30) | after (10-31) | |---|---|---| | verifyTransition p50 | 5 ms | 4 ms | | block process p50 | 16 ms | 15 ms | About 1 ms per block, close to the noise of these probes.
…-delay sidecar lazily (#25647) ## Problem `ValidatorSet` keeps a phase0-only sidecar (`phase0Data`: the min-inclusion-delay `PendingAttestation` pointers, two per validator). It was allocated on every construction, decode and copy, on every fork, and `CopyTo` allocated it with `cap = 2*len`. Only the phase0 state transition reads it. At 1.67M validators that is ~27 MB of pointers per decoded set and ~53 MB per fresh copy, which the GC also has to scan. Caplin keeps several full states alive and copies the head state at least once per slot. ## Fix - The sidecar is nil until the first phase0 setter writes it. Getters read a missing sidecar as nil, `Append` extends it only when present, `CopyTo` keeps nil versus present and leaves no stale target values, `DecodeSSZ` resets it, and `Clear()` drops it so a later `Append` cannot read a stale entry. - `UpgradeToAltair` drops the sidecar. - `reinitCaches` ran the phase0 validator initialization for `Version() >= Phase0Version`, which is true for every fork. It now runs only for phase0 states. On real Altair+ states the pending-attestation lists are empty, so this does not change behavior; it only removes a path that could process stale phase0 attestations on a reused state object. No consensus behavior changes: every reader of the sidecar and of the attester bits is gated to phase0 (attestation processing, participation record updates, justification, rewards). ## Measurements Throwaway benchmark at 1.67M validators: | Operation | Before | After | |---|---|---| | `DecodeSSZ` | 232.5 MB/op, 3 allocs | 205.8 MB/op, 2 allocs | | fresh `CopyTo` | 259.2 MB/op, ~7.6 ms | 205.8 MB/op, ~4.8 ms | ## Tests - New `ValidatorSet` tests: a post-Altair set (constructed and decoded) allocates no sidecar; `CopyTo` reuses the target backing array and copies nil versus present; `Append` keeps the sidecar aligned; `Clear`, `ClearPhase0Data` and `DecodeSSZ` drop it; out-of-bounds setters panic. All fail on main, except `TestValidatorSetPhase0DataAppend` and `TestValidatorSetPhase0DataSettersCheckBounds`, which pin existing invariants. - `TestUpgradeToAltairClearsPhase0ValidatorData` and `TestReinitCachesInitializesValidatorPhase0DataOnlyAtPhase0` fail on main. - Phase0 and Altair spectests (`cl/spectest`, `-run '/mainnet/(phase0|altair)/'`): 1714 pass, no new skips; the full `cl/spectest` result list is identical to main.
…#25619) `responseValidators` and `responseValidatorsBalances` check every validator in the registry against the `id` filter with `slices.Contains`. That's O(registry × ids), and it runs while the state is being read. On mainnet-sized registries, a request with a few thousand ids (the POST variants take an arbitrary list) spends seconds in that loop. Repro: `BenchmarkResponseValidatorsFiltered` (200k validators, 20k ids) takes about 1.5s/op on main on my machine and 2.9ms/op with this change. Fix: build a set from the parsed indices once and look each validator up in it. An empty filter still means "everything", same as before. Existing handler tests are unchanged and pass.
…5660) ## Problem `GET/POST /eth/v1/beacon/states/head/validators` builds the whole JSON response and writes it to the HTTP client inside `syncedData.ViewHeadStateWithIdentity`, that is, under the `synced.mu` read lock. For the full registry (about 2.4M validators; at 440–490 bytes per entry that is about 1.1 GB of JSON), the lock is held for the whole encoding (0.7–1.8 s of CPU) and for the network write. For a response this large the write blocks until the client has read almost all of it, so the lock hold time depends on the client. When the read lock is held, the head-state writer (`stages.postForkchoiceOperations` → `OnOwnedHeadStateWithBlockRoot`, and the other `OnHeadState*` publishers) waits on `synced.mu`. Go's `RWMutex` blocks new readers while a writer waits. So attestation, aggregate and block gossip validation, which also take `synced.mu` R, stall behind the slow HTTP client too. A single slow client can freeze head updates and gossip validation for the whole node. On a real Hoodi node we saw 55 s without a head update and wrong head votes. The same code is on main; the problem is not Gloas-specific. ## Fix In the head path of `writeValidatorsResponse`, copy what the response needs while the lock is held, then release the lock, and encode and write with no lock held. The copy always includes the state epoch and the optimistic flag of the head root. It also includes: - for a request filtered by `id`: only the requested validator records (121 bytes each, copied into one buffer) and their balances. The client chooses how many ids it sends, so the head path sorts and deduplicates them before it takes the lock; under the lock it only cuts them at the registry length with a binary search. This is the same selection as `sortedIndicesBelow` in the other paths; - for any other request: the whole balances list and validator registry. `DecodeSSZ` copies the validator records into a flat byte buffer without building Merkle trees; it also allocates the validator set's per-validator side arrays. `responseValidators` now encodes a sequence of `(index, validator, balance)` entries (`validatorEntries`), so the head copy and the historical and fork-choice paths share `appendValidator`. `appendValidator` takes the balance as a parameter, because the head copy holds only the requested balances. The response is unchanged. It has the same bytes, the same `execution_optimistic` and `finalized` flags, and status computed with the same epoch (`state.Epoch(headState)`). The copy is needed because the head state cannot be referenced after the lock is released: later publications rotate the head, previous and spare states, and `publishHeadStateLocked` overwrites a reusable spare in place with `CopyInto`. Prysm uses the same pattern: a short read lock for the copy, then encoding with no lock held. Why not `forkchoiceStore.GetValidatorSet`/`GetBalances`, as the non-head path for recent states does: each call materializes the whole beacon state on its own (a copy of the current state under `f.mu`, or a replay), so using both doubles that work. The published head state is already available under `synced.mu`. This PR builds on #25619, which is now in main (`sortedIndicesBelow` and the `appendValidator` closure in `responseValidators`). ### Cost (synthetic 2.4M-validator registry, measured locally) | | before | after | |---|---|---| | `synced.mu` R hold, full registry | encode 0.7–1.8 s + network write, which depends on the client (capped only by the Beacon API write timeout, one year by default) | copy 22–73 ms | | `synced.mu` R hold, `?id=` with 10 ids | ~10 ms (full-registry scan) | copy of 10 entries | | `synced.mu` R hold, 1000 ids on main (`slices.Contains` filter, O(n·k)) | filtering, encoding and the write all under the lock; end-to-end request latency p50 596 ms (load harness, main; not an isolated hold time) | copy of 1000 entries (0.47–0.71 ms in the load harness) | | extra transient memory, unfiltered head request | — | about 336 MiB (registry 279 MiB, balances 18 MiB, per-validator side arrays 39 MiB) | | extra transient memory, id-filtered head request | — | under 200 bytes per requested id | For a full-registry request the copy is small next to the response itself: the `strings.Builder` holds about 1.1 GB of JSON and `[]byte(b.String())` copies it again. A request filtered only by `status` still copies the whole registry; the status filter runs after the lock is released. An id-filtered request, the validator-client hot path, copies only its entries: on a 100k-validator registry (12.1 MB) a head request with 60,000 ids, of which only 3 distinct ids are in range, allocates 560 bytes under the lock, against 14.7 MB with the full copy. An id list that names most of the registry costs about twice the full copy under the lock, because the selected records are copied one by one into a single buffer (the review measured about 26 ms with one buffer against 12.6 ms for the full copy on a 2.4M-validator registry). In all cases the hold time no longer depends on the client. No concurrency bound is added. Each unfiltered head request already needed more than 2 GB for the two response buffers, and concurrent requests already shared the read lock; the copy adds about 15% to that. A general bound would need a policy decision (queue or reject) for the whole API, not just for this handler. ## Tests - `TestHeadValidatorsSlowClientDoesNotBlockHeadStatePublish`: serves a head validators request whose `ResponseWriter` blocks in `Write`. While it is blocked, it calls `SyncedDataManager.OnHeadState` and requires that call to return within 5 s. On main it fails: the writer is blocked by the handler's read lock. - `TestHeadValidatorsResponseMatchesHeadState`: requires the head response body to be byte-identical to an oracle that runs the old selection loop (full-registry `Range` with `slices.Contains`) on the live head state. It covers the full registry, an id + status filter, unsorted and duplicate ids with an out-of-range id, and a status filter that drops one of the requested ids. The head is optimistic, and some validators have a status that depends on the epoch. Mutations of the epoch (+1 and −1), of the optimistic flag, and dropping the sort or the deduplication of the ids make it fail. - `TestHeadValidatorsAreCopiedInsideTheView`: overwrites the head state in place with `CopyInto` right after the view callback returns (the existing `overwriteAfterViewSyncedData` test helper), and requires the response to match the state seen inside the view, for an unfiltered and an id-filtered request. Replacing the balances copy, the registry copy or the per-entry validator copy with a reference to the live head state makes it fail. - `TestHeadValidatorsFilteredByIdCopyOnlyTheRequestedEntries`: on a 100k-validator head state, a head request with 60,000 ids (0, 1 and 2 repeated, interleaved with 30,000 distinct out-of-range ids) must allocate less than 8 bytes per requested id (480 KB) inside the head-state view. It allocates 560 bytes. It fails with 1.45 MB when the ids go through `sortedIndicesBelow` under the lock, 1.2 MB when the entries are allocated for every distinct id, 5.0 MB without the deduplication, 803 KB when the whole balances list is copied, and 14.7 MB when the whole registry is copied. - `TestCopyValidatorEntriesByIdAllocatesOneBuffer`: copying every validator of the test state by id must take fewer than 10 mallocs. It fails with 258 mallocs for 256 ids when each record is cloned on its own. - A one-off check (not committed) compared the response bytes of eight head queries between the code before this change (with #25619 merged) and after it: unfiltered, an id + status filter, unsorted, duplicate and out-of-range ids, a known and an unknown pubkey id, a status-only filter, a status filter that drops a requested id, and a POST with duplicate and pubkey ids. All were identical. - The existing `head_validators_all` harness test still passes. It compares the decoded JSON, not the bytes. - Mutation check: with the fix reverted, the slow-client test fails again. ## Load harness (in-process, mainnet-size Gloas state, scenario b with all readers, slow `/states/head/validators` reader at 5 MiB/s for 30 s) Bundle is our local branch with the open Gloas performance PRs (it carries an older revision of #25619 with a map-based id filter). p15 adds the first commit of this PR (full copy for every head request); p15b adds both commits, in an earlier revision of the second commit that still sorted the ids under the lock. Bundle and p15 ran three rounds each, p15b two. The first five rows show samples / p50 / p99 / max (ms); the other rows state their own format. | row | bundle | p15 | p15b | |---|---|---|---| | envelope arrival → FULL head state published | 282 / 295 / 3201 / 26018 | 285 / 284 / 467 / 688 | 190 / 276 / 498 / 615 | | `synced.mu` W wait, `stages.postForkchoiceOperations` | 564 / 0 / 8.8 / 25642 | 570 / 0 / 8.8 / 9.6 | 380 / 0 / 8.9 / 11.3 | | `synced.mu` R wait, attestation gossip | 225012 / 0 / 15632 / 17098 | 226710 / 0 / 0 / 0.1 | 151140 / 0 / 0 / 0 | | `synced.mu` R wait, aggregate gossip | 225357 / 0 / 1829 / 25638 | 227946 / 0 / 0 / 8.9 | 151964 / 0 / 0 / 8.7 | | `synced.mu` R wait, block gossip | 285 / 0 / 7758 / 19132 | 285 / 0 / 0 / 0 | 190 / 0 / 0 / 0 | | FULL head never published / block never published as head | 3 / 3 | 0 / 0 | 0 / 0 | | `synced.mu` R hold of the slow full-registry request | 30003–30007 | 16–23 | 17.5–17.9 | | `synced.mu` R hold of a 1000-id request | 26–38 | 16–49, two outliers at 123 | 0.47–0.71 | | `validators_1000_ids` request latency, p50 / max | 36 / 42 | 52 / 143 | 2.5 / 3.9 | With only the first commit (p15), the 1000-id request paid for the full-registry copy. With both commits (p15b), it copies 1000 entries and is about 14 times faster end to end than bundle, whose map-based filter still scans the registry under the lock. Outside the slow-reader window, GC pause p99 went from 16 ms (bundle) to 10 ms. `getOwnedState` p99 went from 37 ms to 48 ms, which is within the spread seen across bundle and base rounds (44 ms in bundle round 3, 544 ms in base round 2). ## Follow-ups (not in this PR) - `GET/POST /eth/v1/beacon/states/head/validator_balances` and `GET /eth/v1/beacon/states/head/validator_identities` build their full-registry response objects (2.4M entries) under `synced.mu` R. The network write already happens after the lock is released, but the build cost grows with the registry size. - `responseValidators` buffers the whole response, and `w.Write([]byte(b.String()))` copies it once more (about 1.1 GB for mainnet). `io.WriteString(w, b.String())` would avoid that copy. Streaming the JSON in chunks would remove the buffer. - Never overwriting a published head state (immutable snapshots, as in Lighthouse and Teku) would remove the need for copies here, but it is a larger design change.
…ls (#24985) Before Gloas, every `get_head` recompute sums all validators' latest messages under `f.mu`. On a mainnet tip node that is 40–82 ms between execution and the FCU, about twice per slot. Gloas already keeps weights incrementally (`gloasWeightTree`), so this covers mainnet until Glamsterdam. ## Changes - `voteTally` keeps per-root vote weights across calls: a rebuild when the justified checkpoint state changes, otherwise only validators whose message or equivocation changed. - Proposer boost, `accountWeights` and the auxiliary-state path are unchanged. ## Numbers Mainnet tip node (`awskii/v3-commitment`), 3 interleaved 40-min arms per binary: | | base | this | |---|---|---| | execution done → FCU p50 | 107 ms | 41 ms (p 4e-5) | | FCU → head updated, after the reply | 5 ms | 8 ms | The +3 ms is Caplin's head-state copy, no longer queued behind the recount, overlapping the flush. --------- Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
…25595) ## Problem On a mainnet node without attached validators, all votes reach fork choice through block attestations: each block sets the latest message of about 25k validators. The pre-Gloas vote tally and the Gloas weight tree only marked these validators dirty, and the next get_head applied every dirty vote while holding f.mu. On the mainnet node, the first get_head after each block held the lock for about 9 ms (p50); with no new block it held it for about 0 ms. ## Change - setLatestMessage and setUnequivocating update the pre-Gloas vote tally and the Gloas weight tree right away, once a baseline exists. - The dirty maps, applyDirtyValidators and the "reset when more than half is dirty" rule are removed. - get_head still does a full rebuild when the justified checkpoint state changes, and for Gloas also after finalization pruning. Gloas votes for a block that is not in the tree yet are still tracked and counted when the block joins the tree. - Votes recorded right after the justified checkpoint moves are applied to the old baseline and dropped by the next rebuild. This happens about once per justification change and does not affect the weights. ## Numbers The per-vote work moved from get_head into vote recording. OnBlock applies the block's attestations under f.mu before forkchoiceUpdated, so the head-path gain is the total of both steps, not the get_head drop. New BenchmarkGetHeadPreGloas / BenchmarkGetHeadGloas: 2.2M registry, 815k active validators, 96-block tree, one block of votes (1/32 of active). Apple M5 Pro, ns/op, three interleaved runs; before = the merge base with main plus the same benchmark file: | | get_head after a block | record votes + get_head | |---|---|---| | pre-Gloas | -98.6..-98.8% | -19..-31% | | Gloas | -98.6..-98.7% | -29..-31% | Mainnet node (Xeon E-2288G, Erigon with embedded Caplin, no validators). Two runs at the same hours (09:40-17:40 UTC, about 2390 slots each): before = main from 2026-10-30 plus the other Caplin performance PRs; after = the same plus #25594, #25595 and #25596, measured with fork-choice lock probes that are not part of this PR. | | before (10-30) | after (10-31) | |---|---|---| | get_head f.mu hold p50 / p90 (H:lockStore) | 9 / 11 ms | 0 / 0 ms | | post-EL onBlock f.mu hold p50 / p90 | 43 / 57 ms | 42 / 53 ms | Gossip block receipt to SSE head event, measured on the same node for 2.4 h before and after the switch: p50 140 -> 137 ms, p10 99 -> 92 ms, p90 281 -> 273 ms. The get_head cost is gone; vote recording moved into OnBlock did not make the post-EL hold longer, but the end-to-end gain is a few ms. ## Tests - TestVoteTallyAppliesVotesWhenRecorded and TestGloasWeightTreeAppliesVotesWhenRecorded compare the incremental weights after each random vote or equivocation with a full recount (pre-Gloas) and with a full rebuild plus the full-scan weight store (Gloas). Both failed before the change. - TestGloasWeightTreeMatchesRebuildAcrossPrepares checks random votes against a full rebuild across finalization pruning, checkpoint-state switches and a voted block that joins the tree through OnBlock. - Tests that asserted the dirty maps now assert the applied weights.
domiwei
requested review from
AskAlexSharov,
sudeepdino008 and
yperbasis
as code owners
November 3, 2026 04:35
domiwei
requested review from
awskii and
lystopad
and removed request for
sudeepdino008
November 3, 2026 04:36
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unused vote roots accumulate during prolonged justification stalls, increasing memory usage and the cost of uncached head computations while the fork-choice mutex is held.
1 open issue
What changed in this PR
Backports #24985 and #25595 to the release/3.8 stack to reduce Caplin’s fork-choice head-computation cost and lock hold time.
Changes:
- Preserve pre-Gloas vote weights across
get_headcalls. - Apply votes and equivocations immediately in both the pre-Gloas tally and Gloas weight tree.
- Add upstream regression tests, benchmarks, and a v3.8.2 changelog entry.
| File | Description |
|---|---|
cl/phase1/forkchoice/weight_store_diff_bench_test.go |
Updates benchmarks for immediate vote application. |
cl/phase1/forkchoice/vote_tally.go |
Adds the persistent pre-Gloas vote tally. |
cl/phase1/forkchoice/vote_tally_test.go |
Tests tally updates, rebuilds, and fork-boundary reset. |
cl/phase1/forkchoice/vote_tally_bench_test.go |
Benchmarks full recounts and incremental updates. |
cl/phase1/forkchoice/on_attestation.go |
Applies recorded votes and equivocations immediately. |
cl/phase1/forkchoice/gloas_weight_tree.go |
Replaces deferred dirty-vote processing with immediate updates. |
cl/phase1/forkchoice/gloas_weight_tree_test.go |
Verifies immediate updates, pruning, and missing-root handling. |
cl/phase1/forkchoice/get_head.go |
Uses the cached pre-Gloas tally. |
cl/phase1/forkchoice/get_head_bench_test.go |
Adds realistic pre-Gloas and Gloas head benchmarks. |
cl/phase1/forkchoice/forkchoice.go |
Stores the vote tally in fork-choice state. |
ChangeLog.md |
Documents the performance backports under v3.8.2. |
Review effort: Max
Comment on lines
+83
to
+86
| if !ok { | ||
| id = uint32(len(t.weights)) | ||
| t.ids[message.Root] = id | ||
| t.weights = append(t.weights, 0) |
lystopad
approved these changes
Nov 3, 2026
lystopad
left a comment
lystopad
left a comment
Member
There was a problem hiding this comment.
Approving. Checked:
- Fidelity. Both cherry-picks match their upstream squash commits (#24985
b27fb9e0fe, #255951f662b3ab4) with zero differing change lines, and apply cleanly on top of #25704. - CI is green.
- Title, description and ChangeLog match the code. The v3.8.2 entry is accurate. "Stacked on #25704, retarget after it merges" is right, and #25704 is approved.
Copilot's point on unused vote roots: valid, low severity, already on main since #24985. Roots leave ids/weights only when the justified checkpoint or state is rebuilt, so the tally grows only during a long justification stall. Each entry is small, and the code it replaces recounted every validator's vote on each call, so this is still far cheaper. It is not a blocker for the backport. A follow-up on main that reclaims unused roots, independent of checkpoint changes, would close it, and could then be backported.
One note for the release. #25595 merged on main only on 11-02 and changes when fork-choice votes take effect, which is consensus-critical. I would like it to soak on main for a few days, or be run on a node, before 3.8.2 ships. Not a blocker for this PR.
awskii
approved these changes
Nov 3, 2026
Base automatically changed from
kewei/cherry-pick-r38-head-latency
to
release/3.8
November 3, 2026 12:39
…ick-r38-vote-tally # Conflicts: # ChangeLog.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #24985 and #25595 to release/3.8.
get_headvote weights across calls (addsvote_tally.go).Both cherry-picks applied without conflicts. A ChangeLog entry is added under Erigon v3.8.2.
Uses the existing upstream tests for this mechanical backport (TDD exemption).