Repository navigation
skills: Redpanda Connect skill updates for v4.115.1 (2026-10-30) - #118
Merged
Merged
Conversation
Synced the Connect skills against Connect v4.115.1 (benthos v4.84.0, unchanged). The release itself carries no structural change: its two kafka entries are franz-writer fixes and its oracledb_cdc entry is a schema-unit change, handled below. The rest of this change corrects migrator drift and records two durable CDC constraints the skills were missing. connect/references/migration.md - The migrator input's lag gauge is `redpanda_lag`, not `input_redpanda_migrator_lag` (which no longer exists in the tree), and it is emitted only when the input has a `consumer_group` (internal/impl/kafka/lag.go, internal/impl/redpanda/migrator/migrator.go). - A single migrator pair needs no labels: the shared Migrator is keyed by label with a "default" fallback. Labels matter only with several pairs in one config, where a mismatch makes the pair fail to connect rather than silently mis-coordinate. Dropped the labels from the workflow YAML and replaced the incorrect label-syntax claim with the engine's own rule (benthos internal/docs/config.go). - Documented exact consumer-group offset translation for groups in the Empty state via the output's offset header, which the "timestamp-based and approximate" guarantee had flattened (internal/impl/redpanda/migrator/migrator_groups.go). - Recorded that an existing destination topic with fewer partitions than its source has partitions added to match. connect-cdc-postgres - postgres_cdc requires the primary instance; a read replica or physical standby is not supported. Product requirement stated in the connector's reference documentation; provenance in the skill's SOURCES.md. - The snapshot phase reads through long-lived REPEATABLE READ transactions on an exported snapshot, holding the database's xmin horizon for its whole duration; large tables should use an incremental snapshot (internal/impl/postgresql/pglogicalstream/snapshotter.go, logical_stream.go). connect-cdc-oracle - TIMESTAMP columns declare microseconds in the emitted schema (DATE keeps milliseconds), so TIMESTAMP(6) is no longer truncated and snapshot and streaming now agree (internal/impl/oracledb/schema.go, TYPES.md). Each skill's SOURCES.md carries the full sync log, including what was checked and deliberately not documented, and the open question about duplicate-label linting versus the documented multi-pair scheme. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s best-effort Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scheduled Connect sync against Connect
v4.115.1(current stable tag; changelog section dated 2026-10-23, nothing newer tagged). The benthos dependency is unchanged atv4.84.0ingo.modat the tag, so no engine, CLI or Bloblang surface moved with the release.The release itself
Nothing in the
v4.115.1changelog is a structural skill surface:kafkaentries are fixes inside the franz writer — a data race between the produce-error and record-decoration slices, and the dispatch signal now firing when a record is handed to the producer, restoring the hand-off the ordered per-partition input relies on. No flag, field, component name or documented behaviour changes, and no skill asserted the broken behaviour.oracledb_cdcentry is a schema-unit change, applied below.The remaining changes are drift corrections and two durable CDC constraints the skills were missing. Per-field additions elsewhere in the window stay deferred to the generated reference and
rpk connect create.Changes
skills/connect/references/migration.md(Migration row ofreferences/SOURCES.md)redpanda_lag, notinput_redpanda_migrator_lag(absent from the tree), and it is emitted only when the input has aconsumer_groupredpanda-data/connectinternal/impl/kafka/lag.go(lagMetricNameRedpanda),internal/impl/redpanda/migrator/migrator.go(both specs) andinternal/impl/redpanda/migrator/README.md, atv4.115.1internal/impl/redpanda/migrator/migrator.go(newMigratorFrom'smigratorKey{label, stream}with its"default"fallback, the input spec's pairing paragraph,validateInitialized)[a-z0-9_-], no leading_/-); the previous "3–128 chars,A-Za-z0-9-_" claim was wrong on both countsredpanda-data/benthosinternal/docs/config.go(labelExpression,ValidateLabel) atv4.84.0Emptystate via the output's offset header — the skill presented translation as timestamp-based and approximate, full stopinternal/impl/redpanda/migrator/migrator_groups.go(tryFindExactOffset, gated onState == "Empty"and a non-empty header) and theoffset_headerfield spec inmigrator.gointernal/impl/redpanda/migrator/migrator.goskills/connect-cdc-postgres/(SKILL.md,references/config-reference.md)postgres_cdcrequires the primary instance — a read replica or physical standby is not supported. A product requirement stated in the connector's reference documentation; provenance is recorded in the skill'sreferences/SOURCES.md.xminhorizon for as long as it runs, for every table, so a scan measured in hours defers cleanup of row versions that died after it started; large tables should use an incremental snapshot. Verified ininternal/impl/postgresql/pglogicalstream/snapshotter.go(openTxn: read-onlyREPEATABLE READon a background context, thenSET TRANSACTION SNAPSHOT; rolled back only inreleaseSnapshot) andpglogicalstream/logical_stream.go(SnapshotAction: "EXPORT_SNAPSHOT").No
postgres_cdcsource changed betweenv4.115.0and the tag.skills/connect-cdc-oracle/references/pipeline-and-output.mdschemametadata field carries (DATEkeeps milliseconds), soTIMESTAMP(6)is no longer truncated and the snapshot and streaming schemas agree; a consumer holding the pre-4.115.1 schema sees the unit change. Verified ininternal/impl/oracledb/schema.go(columnToCommon,oracleTimestampLogical) andinternal/impl/oracledb/TYPES.mdat the tag. The JSON wire format of a timestamp is unchanged, so the type table still reads ISO-8601.Checked and deliberately not documented
oracledb_cdc/microsoft_sql_server_cdcbatcher refactor onto a shared flush-ticket package, and the second "flush queue is sealed" variant of their rebuild-publisher warning: internal, and no skill quotes those log lines.offset_headeris named only where its semantics change a documented guarantee.TODO for a human
Duplicate-label linting versus the documented multi-pair scheme. benthos
v4.84.0registers every component label in one per-run lint context and returns a duplicate-label lint for a label used twice in a config, which appears to collide with the migrator's documented scheme of giving an input and its paired output the samelabel.migration.mdnow tells readers to lint a multi-pair config and confirm the behaviour on their version rather than asserting either way. Worth a Connect-team answer on whether a deliberately matched pair lints clean.🤖 Generated with Claude Code