Skip to content

fix(engine): self-heal a missed rollup-cache checkpoint instead of degrading readiness for an hour - #494

Merged
vishr merged 1 commit into
mainfrom
fix/checkpoint-readiness
Nov 16, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/checkpoint-readiness

Conversation

@vishr

@vishr vishr commented Nov 16, 2026

Copy link
Copy Markdown
Member

Problem

The DuckDB CHECKPOINT that folds the rollup cache's write-ahead log back into its file ran only inside the hourly runRepositoryMaintenance, with a 250ms retry budget shorter than any read that began before the last write commit and is still running. So a busy instance reliably missed the fold. The miss folded into the whole maintenance error, which made /readyz report degraded until the next maintenance pass — one hour later by default — even though ingest and queries keep working and a restart would not help. Starting maintenance at boot also put the first checkpoint in the window with the most startup reads.

Closes #482.

Fix

Separate checkpoint health from maintenance health. A checkpoint-only failure no longer touches lastMaintenanceErr / maintenanceFailures. Retention DELETEs (extracted to pruneRollupCache), prune, compaction, and recovery failures still degrade maintenance readiness immediately, because those let storage grow unchecked. The checkpoint gets its own health fields and a CheckpointHealth() accessor mirroring MaintenanceHealth().

Retry a missed fold on the 10s compaction cycle. A checkpointPending flag is honored by both the hourly pass and runParquetCompactionCycle. When nothing is pending the cycle takes no write gate, so an idle WAL adds no contention with the 60s rollup tick; when a fold is pending it holds both write gates and the background slot exactly as the old maintenance checkpoint did. A transient refusal therefore self-heals in ~10s instead of ~1h, and the boot checkpoint no longer has to succeed in the busy-read startup window.

Budget raised to a few seconds. checkpointRetryBudget is now 3s (from 250ms); both gates are idle-cheap relative to the 60s rollup tick, so giving an older reader a few seconds to drain lets the common transient conflict clear in place. The exact-message retry gate (checkpointRetryable / checkpointBusyMessage) is unchanged, so checkpoint_retry_test.go's real-conflict reproduction still pins the diagnostic.

Readiness surfaces the checkpoint separately and self-healingly. A new checkpoint readiness check reports degraded only after checkpointDegradeAfterFailures (6, ~1 minute at the 10s cadence) consecutive failures, and never unhealthy. A single transient miss stays ok, so one conflict does not flap the instance out of rotation; a fold that is genuinely stuck still surfaces. A wedged engine is still caught independently by the telemetry SELECT 1 probe.

Tests

Reconciled the previously-passing contracts that asserted a checkpoint failure increments maintenance failures, moving those assertions to checkpoint health, and kept the exact-message and caller-budget-retention tests intact. New behavioral tests (red before the fix):

  • TestCheckpointMissDoesNotDegradeMaintenanceHealth — a miss leaves maintenance health clean and is recorded only in checkpoint health.
  • TestMissedCheckpointRetriesOnCompactionCycle — driving runParquetCompactionCycle directly, the first cycle records the miss and the second retries and clears it (self-heal proven without a real clock).
  • TestRetentionFailureDegradesMaintenance — a real retention/prune failure still degrades maintenance (no readiness regression for genuine maintenance failures).
  • TestReadinessToleratesTransientCheckpointMiss — checkpointResult: never-attempted / clean / a few recent misses are ok; past the threshold is degraded; never unhealthy.

TestMaintenanceCheckpointRetryExhaustionIsExplicitAndBounded now asserts maintenance stays clean, the miss is recorded in checkpoint health with the exact budget-exhausted and transaction-active text, and the pass is bounded by the 3s budget.

Verification

just test "./internal/engine/..." and just test "./internal/api/..." (full suites), just vet, and just check-static (gofmt, deadcode, behavioral test names) all green. No config, schema, or migration changes.

…grading readiness for an hour

The rollup-cache checkpoint folds the write-ahead log back into its file.
Its non-FORCE form refuses while a read that began before the last write
commit is still open, and the old 250ms retry budget was shorter than almost
any such read, so a busy instance reliably missed the fold. The miss folded
into the whole maintenance error, which flipped /readyz to degraded until the
next maintenance pass -- an hour away by default. A missed fold harms nothing
a restart or a page would fix: the WAL stays correct and queryable, just
larger until the next successful fold.

Separate checkpoint health from maintenance health. A checkpoint-only failure
is recorded in new CheckpointHealth state (lastCheckpointOK/At/Err,
checkpointFailures, a pending flag) and never touches maintenance health.
Retention DELETEs, prune, compaction, and recovery still fail maintenance as
before -- those let storage grow unchecked, so they must degrade readiness
immediately. The retention DELETEs move into their own gated helper
(pruneRollupCache) so the fold can be driven separately.

Drive the checkpoint off a pending flag that both the hourly pass and the 10s
compaction cycle honor, rather than checkpointing every 10s. When nothing is
pending the cycle takes no write gate, so an idle WAL adds no contention with
the 60s rollup tick; when a fold is pending it holds both write gates and the
background slot exactly as the hourly pass did. The hourly pass marks the fold
pending (an interval of rollup commits plus the retention DELETEs always leave
work) and attempts it; a miss stays pending and the next ~10s cycle retries,
so a transient conflict self-heals in seconds instead of an hour. Only a
context cancellation propagates, so the maintenance loop still stops promptly.

Raise the retry budget from 250ms to 3s. Both write gates are idle-cheap
relative to the 60s rollup tick, so giving an older reader a few seconds to
drain lets the common transient conflict clear in place. The exact-message
retry gate (checkpointRetryable/checkpointBusyMessage) is unchanged.

Report checkpoint separately in readiness via a new checkpoint check. Because
the fold retries every 10s, a single recent miss is not degraded; only a fold
that keeps failing past several cycles (checkpointDegradeAfterFailures) is. It
never reports unhealthy -- the instance serves while the WAL grows -- and a
genuinely wedged engine is still caught by the independent telemetry SELECT-1
probe.

Closes #482
@vishr
vishr enabled auto-merge November 16, 2026 00:14
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 57f6d99 Nov 16, 2026
31 checks passed
@vishr
vishr deleted the fix/checkpoint-readiness branch November 16, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ops: a missed checkpoint degrades readiness for an hour

1 participant