Repository navigation
[r3.8] cl: stop fsyncing fork-graph dumps, validate boundary blocks without a state copy, and republish moved heads and queued envelopes - #25287
Merged
Conversation
DumpBeaconStateOnDisk and DumpEnvelopeOnDisk called File.Sync() on every dump. Both files live in path.Join(dirs.Tmp, "caplin-forkchoice"), which Caplin removes and recreates on every start (cmd/caplin/caplin1/run.go; the removal came with the dump and its Sync in #8546). It is the only production fork-graph filesystem, and no dump file is read after a restart, so fsync's durability guarantee (the content survives a power loss or kernel crash) protects nothing. Inside the running process the files stay usable without fsync: - reads see written data through the page cache, fsynced or not; - all content reads and writes of these files hold stateDumpLock (Prune removals and the hasBeaconState existence check run outside it, as before); - if a head-state dump cannot be read, replay walks back to an older dump when the block is still kept, or returns no state when only the header is kept (for example the anchor), the same as for any missing dump; - envelopes are still written to a .tmp file, closed with a checked Close, and renamed, so readers never see a partial file, and a write, flush, close or rename error keeps the existing final file. History: the head-state Sync came with the original feature (#8546, which gives no rationale; its error was always returned) and was moved in #8851. The envelope dump copied it in #18956, and #23548 moved envelopes to tmp+rename and kept the Sync. No change relies on the durability of these files. Cost, measured on a Hoodi node (1.14M validators): each head-state dump writes about 143 MB every 4th slot. dumpMs is p50 ~540 ms, p90 ~1.06 s, max 2.65 s, while encoding takes ~4 ms and snappy + write ~100 ms (measured locally). Every dump holds stateDumpLock (also needed by fork graph replay, req/resp envelope reads and, after Gloas, envelope import while it holds the fork-choice f.mu). The head dump after fork choice also runs under the synced-data read lock and inside the stage's indiciesDB RW transaction. A shorter dump shortens all of these. Trade-off: fsync is also the point where the kernel reliably reports a delayed writeback error (EIO). close(2) on a local filesystem usually does not report it. Such an error may now go unnoticed by the dump and show up later, for example when the data is read again after the pages were evicted. Before, the dump itself failed. A head-state read error makes replay fall back as described above. An envelope that fails to decode is marked invalid and can be fetched again. If the re-read itself returns EIO, the envelope stays marked as present, so normal ingestion skips it while the error lasts: until a later read succeeds, the anchor envelope is rewritten, or the block is pruned. This needs a failing disk, and it is accepted here for the cost above. The head-state dump closed its file with an unchecked defer. It now closes it once in a defer that joins the Close error into the result, like the envelope dump, so errors reported at close (for example on NFS) are not lost. The finalized-state save under dirs.CaplinLatest (saveFinalizedStateOnDiskIfNeeded / replaceDurableFile) is the restart anchor and keeps its fsyncs. Tests: with a file whose Sync fails, both dumps must succeed and read back equal; with a file whose Close fails, the head-state dump must return the error and close once. The "sync" stage of TestDumpEnvelopeFailurePreservesExistingFinal is removed because Sync is no longer called. (cherry picked from commit a859869)
…locks from the parent's proposer lookahead Gossip validation of the first block of an epoch copied the parent state and ran ProcessSlots across the epoch boundary to read the expected proposer, the latest block hash and the latest bid. From Fulu on, epoch processing only shifts proposer_lookahead and does not write the latest block hash or the latest bid, so the parent state already holds these values when no fork upgrade happens in between. Extract the lookahead read of GetBeaconProposerIndexForSlot into ProposerIndexFromLookahead, which reports whether the slot is inside the lookahead window. computeBlockValidationContext now also reads a parent from an earlier epoch in place when the parent is at most MinSeedLookahead epochs back, the scheduled state version is the same at both epochs and at least Fulu, the viewed state has that version, and the lookahead covers the slot. Fork activation epochs and parents further back still use the copy path. (cherry picked from commit bdaa78e)
…tart The proposer boost expires when a slot starts, so the fork choice head can move back to another branch without a new block. SleepForSlot ends at that moment, but it went to ChainTipSync without checking for an unpublished head, and ChainTipSync can wait a slot or longer for the next block. Until then the synced head state, the EL forkchoice and the Beacon API head stayed on the old head, and block production failed its head-change check because the head state never caught up with fork choice. SleepForSlot now routes an unpublished head to ForkChoice like the stages that import blocks. The stage args also tick fork choice to the start of the slot they target before they read the head, so the boost expiry is seen even when the fork choice clock ticker has not reached the slot yet. (cherry picked from commit fda0346)
…yload envelopes An envelope that arrives before its beacon block is ignored on gossip and queued until the block is known. Gossipsub does not forward ignored messages, so when the queued envelope later validated it was processed locally but never relayed. Peers that only hear the envelope through Caplin then miss it. Publish the envelope through the gossip manager once it is processed from the pending queue. Lighthouse, Prysm and Teku also propagate an envelope after deferred validation. go-libp2p-pubsub validates and forwards a local publish of a message that was seen but ignored. (cherry picked from commit 57fe39d)
domiwei
requested review from
AskAlexSharov,
sudeepdino008 and
yperbasis
as code owners
October 23, 2026 10:01
domiwei
requested review from
awskii and
lystopad
and
a balanced review from Copilot
October 23, 2026 10:01
domiwei
enabled auto-merge
October 23, 2026 10:01
lystopad
reviewed
Oct 23, 2026
lystopad
left a comment
lystopad
left a comment
Member
There was a problem hiding this comment.
Checked the picks: all four commits are patch-identical to the #25251/#25253/#25258/#25259 branches, and fork_graph, state, stages and the network-service tests pass with -race on this branch. I also checked #25251's premise on release/3.8: caplin-forkchoice under dirs.Tmp is removed on every start (run.go:365), and torn dumps are caught by the read path, so dropping the fsync is safe.
Since the four main PRs are still in the merge queue, I'll approve once they merge and the merge commits match these picks.
yperbasis
approved these changes
Oct 23, 2026
yperbasis
disabled auto-merge
October 23, 2026 11:32
This was referenced Oct 23, 2026
kriss39
pushed a commit
to kriss39/erigon
that referenced
this pull request
Oct 27, 2026
…tart (erigontech#25258) ## Problem The proposer boost expires when a slot starts, so the fork choice head can move back to another branch without a new block. When SleepForSlot ended at that moment, its transition could send a node that was behind to ChainTipSync even though `ArgsFunc` had marked the fork choice head as unpublished, and ChainTipSync can wait up to 25 seconds for the next block. Until the ForkChoice stage ran, the synced head state and the state-based Beacon API reads stayed on the old head, and the EL was not sent the new head. On Gloas, block production started from the old synced head state and saw a different fork choice head. It waits for the synced state to catch up before it retries, but nothing published the new head before the proposal cutoff, so no retry started and it returned the head-change error. We saw this in a local multi-client network after a network partition ended: a Caplin node that had to propose right after the boost expired did not produce its block. Before Gloas, block production does not check for a head change and builds on the synced head when the validator client asks, so the change helps there only when the publication comes first. On large states that is less likely. ## What changed - SleepForSlot's transition uses `catchUpPublishingHead`, which was `catchUpAfterImport`, like the stages that import blocks. When `MetaCatchingUp` would select ChainTipSync and the head is unpublished, it selects ForkChoice first. - The stage `ArgsFunc` ticks fork choice to the start of the current slot before it reads the head, so the boost expiry is processed even when the 2 ms fork choice ticker has not reached the slot yet. `OnTick` only moves the stored time forward, and after genesis the start of the current slot is never later than the wall clock. If `GetHead` fails there, the error is logged at Debug and the head counts as published. - At the tip, CleanupAndPruning goes back to ForkChoice when the head is still unpublished. Gossip imports blocks outside the stage loop, so the slot's block can arrive during the slot-start ForkChoice pass or during CleanupAndPruning. Before Gloas, SleepForSlot does not poll the head, so without this the block would be published at the next slot start, about ten seconds late. Before this PR, ChainTipSync was waiting at that moment and published the block on arrival. - The rerun is allowed once per `sleepForSlotRewakeInterval` (1 s). A publish that keeps failing therefore costs at most one extra pass per second, and CleanupAndPruning still runs between the passes. - The bound is time-only. A second head change within that second waits for the next slot start before Gloas. Before this PR such a head was not published before the next slot either. - The check is not in the ForkChoice transition: CleanupAndPruning runs right after it and also covers a block that arrives during CleanupAndPruning. - Dropping the Gloas gate on the SleepForSlot head poll would also close the gap. It is not used because on pre-Gloas networks attestations invalidate the head cache, so it would add five head computations a second under the fork choice lock and wake the stage on every attestation-driven head move. A race remains: the ticker advances the stored time with a lock-free compare-and-swap and takes the fork choice lock afterwards to clear the boost and the head cache. Readers that get the lock in between, for example the `GetHead` in `ArgsFunc`, can still see the cached, boosted head. The window is the time the ticker waits for the lock, not a single preemption point. Closing it needs the cached head to carry the slot it was computed at, which this PR does not do. This PR does not change the fork choice weight calculation or branch selection. The added tick only processes the boost expiry earlier, so the existing rules can select another branch at the slot start sooner than before. ## Tests - `TestCatchUpPublishesForkChoiceHeadBeforeChainTipSync`: - SleepForSlot selects ForkChoice when the node is behind, has peers and has an unpublished head. Before the change it selected ChainTipSync. - At the tip with an unpublished head, ForkChoice goes on to CleanupAndPruning, and CleanupAndPruning goes back to ForkChoice. - CleanupAndPruning goes to SleepForSlot when the head is published, when the node has no peers and is behind, and when a rerun happened less than a second ago. - `TestCleanupAndPruningRerunsForkChoiceOncePerInterval`: two transitions on the same `Cfg` give ForkChoice and then SleepForSlot. - `TestStageArgsTickForkChoiceToTheTargetSlot`: the fork choice store starts one slot behind the clock, and the test checks that `ArgsFunc` moves it to the target slot. Before the change the store stayed at the anchor slot. - Each change has a test that fails without it: - the SleepForSlot transition change; - the `OnTick` call; - the CleanupAndPruning rerun; - each of its three conditions; - the timestamp update. - No test checks that `OnTick` runs before `GetHead`: that needs a head that changes when the proposer boost expires, and this package has no such fixture. - `go test ./cl/phase1/stages/ ./cl/phase1/forkchoice/...` and `-race` on `./cl/phase1/stages/` pass. `make lint` reports 0 issues, and `make erigon integration` builds. ## Follow-ups These are not in this PR: - the tick and boost race above; - SleepForSlot with no peers still goes to ChainTipSync; - a slot boundary inside ForkChoice or CleanupAndPruning; - `payload_attributes` events with a past `proposal_slot` when the head goes back to an older slot. This change is on release/3.8 through erigontech#25287. The rerun commits need a separate backport.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #25251, #25253, #25258 and #25259 to release/3.8.
All four apply without conflicts and need no release-specific changes. They were picked from the PR branches while the PRs were still in the main merge queue.