Skip to content

refactor(telemetry): remove dead compaction crash-recovery marker - #495

Open
vishr wants to merge 1 commit into
mainfrom
chore/remove-dead-compaction-rollback
Open

vishr wants to merge 1 commit into
mainfrom
chore/remove-dead-compaction-rollback

Conversation

@vishr

@vishr vishr commented Nov 16, 2026

Copy link
Copy Markdown
Member

What was dead

Nothing in the current code creates <id>.retired-<output> directories — inputs stay as .batch and each compacted output's replaces.json commits the swap. So the move/rollback branch of RestoreRetiredInputs, protectedRetiredSuffix, and the .retired- handling in format.go were dead. And COMPACTION.json only ever rolled a stage→final rename forward, which boot's loadBatches already redoes from the output's replaces.json; yet a marker whose stage and final were both missing made Open refuse to start.

Closes #486.

The surviving recovery is equivalent

PublishReplacement writes and syncs the output's replaces.json into the staging directory before the single os.Rename(stage, final), so it travels atomically with the rename. On boot, loadBatches/retiredAtLoad reconstruct retirement from each present output's replaces.json. That makes both sides of the rename consistent with no durable marker:

  • Crash before the rename: the staging directory is discarded on boot (and in-process by defer), the selected inputs are still live .batch, and a later pass redoes the merge. No rows lost.
  • Crash after the rename: the output is present with its replaces.json, which loadBatches applies to retire the inputs exactly once. No rows lost, no double counting.

So the entire compaction boot-recovery protocol is now: discard compaction/ staging and let replaces.json decide. Open no longer refuses to start on a marker.

Removed

compactionMarker, recoverCompaction, completeCompaction, RecoverParquet, validateCompactionMarker, the pre-pass marker check, the COMPACTION.json write, RestoreRetiredInputs, protectedRetiredSuffix, logUnresolvedCompaction + compactionRunbookURL + the refuse-to-start path, and the .retired-<output> handling (bare .retired retention handling stays). compactParquet now completes by calling PublishReplacement directly.

The interrupted-merge attempt-file backoff (COMPACTION-ATTEMPT.json, writeCompactionAttempt/readCompactionAttempt/clearCompactionAttempt, mergeCompaction halving, errCompactionPreempted) is a separate mechanism and is untouched, as are replaces.json/retiredAtLoad and retention's bare .retired delete.

Repair-quarantine guard

The marker-based "refuse to quarantine a live compaction member" guard (ensureBatchOutsideLiveCompaction) is removed. It is superseded by the data-directory flock (#485 / #490): repair cannot run against a live data directory, so there is no in-process compaction to collide with. This PR should therefore merge after #490. The merge queue is serial and #490 is already ahead, so ordering holds.

Tests — crash-matrix (the whole justification)

New internal/telemetry/store/compaction_recovery_test.go drives a real ParquetStore/Repository on a real filesystem, abandons it in each crash window, reopens the same directory, and asserts exact row counts and catalog composition:

  • TestInterruptedCompactionBeforeRenameKeepsInputsLive — W1: all inputs live, output absent, staging discarded.
  • TestInterruptedCompactionAfterRenameRetiresInputsFromReplaces — W2: output live, inputs masked once (not double-counted).
  • TestInterruptedCompactionMidInputDeletionRetiresRemainder — W3: re-expresses the removed interrupted-swap recovery test.
  • TestFailedCompactionPublishKeepsInputsLiveAndRetryPreservesRows — a failed publish leaves inputs live; a retry preserves rows.

These were confirmed to fail if loadBatches stops masking inputs by replaces.json (W2 and W3 double-counted), proving they actually exercise the recovery. The removed marker tests' crash-safety coverage is preserved by these.

Verification

just test "./internal/telemetry/..." and just test "./internal/engine/...", just vet, and just check-static (gofmt, deadcode — nothing newly orphaned — behavioral test names) all green. Net diff −888/+382 across 15 files; troubleshooting/data-layout docs updated to drop the removed marker.

@vishr
vishr enabled auto-merge November 16, 2026 00:45
Compaction never produced the state its recovery machinery was built to
undo. Nothing creates `<id>.retired-<output>` input directories: inputs
stay `.batch` and a published output's `replaces.json` retires them. So the
`COMPACTION.json` marker, `RestoreRetiredInputs` and its move branch, the
`.retired-<output>` handling, the marker-protected cleanup set, and the
`Open` path that read the marker and refused to start were all dead or
redundant with recovery that already runs on every boot.

A compaction is now purely in-process: it merges into `compaction/<output>`
staging and commits by renaming that into the batch namespace, where the
output's `replaces.json` retires its inputs. The whole recovery protocol is
to discard the `compaction/` staging on boot and let `replaces.json` decide.
That is equivalent and crash-safe on both sides of the single rename: a
crash before it leaves the inputs live as `.batch` and the staging is
dropped; a crash after it leaves the output present with its `replaces.json`,
which `loadBatches` applies to retire the inputs. Neither window loses rows
or orphans inputs.

Deleted: `compactionMarker`, `recoverCompaction`, `completeCompaction`,
`RecoverParquet`, `validateCompactionMarker`, `RestoreRetiredInputs`,
`protectedRetiredSuffix`, `logUnresolvedCompaction`, `compactionRunbookURL`,
the `COMPACTION.json` write, and the unused `pathExists`/`parquetPublishFunc`
helpers. `compactParquet` now calls `PublishReplacement` directly after
`PrepareReplacement`. The interrupted-merge attempt-file backoff
(`COMPACTION-ATTEMPT.json`) and `replaces.json` itself are untouched.

The repair quarantine guard that consulted the marker to refuse a batch
mid-compaction is removed; it is superseded by the data-directory flock
(#485), so repair cannot run against a live data dir and has no in-process
compaction to collide with.

A crash-matrix test (`TestInterruptedCompaction*`,
`TestFailedCompactionPublishKeepsInputsLiveAndRetryPreservesRows`) drives
each window on a real filesystem and asserts exact row counts and catalog
composition, so it fails if `replaces.json` masking regresses, if boot
completes a merge it should discard, or if any row is lost. The removed
marker tests' crash-safety coverage is preserved by these.

Closes #486
@vishr
vishr force-pushed the chore/remove-dead-compaction-rollback branch from f39dc83 to 432f582 Compare November 16, 2026 02:22
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: remove dead compaction rollback machinery that can refuse boot

1 participant