Skip to content

CON-601: aws_dynamodb_cdc: retry a failed shard completion write on refresh - #5015

Merged
squiidz merged 1 commit into
jira/CON-599from
jira/CON-601
Oct 27, 2026
Merged

squiidz merged 1 commit into
jira/CON-599from
jira/CON-601

Conversation

@squiidz

@squiidz squiidz commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

What

aws_dynamodb_cdc retries a closed shard's failed completion write on every shard refresh, so the shard's tracker entry no longer lingers against max_tracked_shards until the shard ages out of the stream.

Why

finalizeShard left finalizing=true when MarkShardCompleted failed, so the completion was never retried and the entry stayed tracked for up to 24h (found while working on CON-599, #5001).

How

  • finalizeShard clears the finalizing claim when the completion write fails, so a later caller can take it again.
  • New RecordBatcher.RetryFinalize claims and finalizes every closed shard with no inflight batches.
  • refreshShards and refreshTableShards call it once per refresh cycle via retryShardFinalization, right after forgetShards; failures are logged at Warn and retried next cycle.
  • Stacked on CON-599: aws_dynamodb_cdc: stop re-reading exhausted closed shards #5001 (jira/CON-599): the finalization code this fixes only exists there.

Testing

  • Unit tests: 5 regression tests fail without the claim release and pass with it
  • go test -race ./internal/impl/aws/dynamodb/... green
  • golangci-lint 0 issues; task fmt clean; task build:redpanda-connect succeeded
  • Integration: dynamodb package with -tags integration against Docker, passed in 145s
  • Skipped: dynamodb-cdc end-to-end scenario (enterprise connector, no license file)

Risks

Behavior change only after a failed completion write: one extra DynamoDB write per affected shard per refresh cycle plus a Warn log each cycle while the store keeps failing. No config or spec changes.

Jira: https://redpandadata.atlassian.net/browse/CON-601

@squiidz
squiidz marked this pull request as ready for review October 21, 2026 22:53
@squiidz
squiidz merged commit 21cdd6f into jira/CON-599 Oct 27, 2026
16 checks passed
@squiidz
squiidz deleted the jira/CON-601 branch October 27, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants