[JakeSCahill]

Converts rpk-topic, connect-cdc-postgres, sql and cloud-byoc to thin skills that carry only product-specific procedure and point at the docs for everything else. The evals are from #108.

What the evals showed

• Facts. The Redpanda Docs MCP already answers most of what the full skills said. The rest are a handful of facts the docs lack or get wrong, and those go to the docs.
• Procedure. A first pass looked like a clear win for thin skills. A review found most of that came from cases that graded the method instead of the outcome, and from mocks that errored on legitimate commands. Graded on outcome with fair mocks, the baseline handles most of those procedures without help.

These procedures still hold up, so they're the only "How to work" content left:

Skill	Procedure kept
sql	Reference topic tables through their catalog. Fix a lost login in Cloud IAM, not with GRANT CONNECT. Read the stored grants before any REVOKE, and confirm with the user before narrowing a wildcard. Check what a view exposes before granting SELECT on it.
connect-cdc-postgres	Backfill with a snapshot-execute signal, not by dropping the slot. Check a table in order before or after a rejected signal. Choose batch_transactions or a batching policy, never both. Ask before setting REPLICA IDENTITY FULL for a backfill.
rpk-topic	A parsed %p needs -p as well.
cloud-byoc	Create a BYOVNet cluster on Azure with Terraform alone. Never pass --cloud-api-token, and keep the version-check override out of production. Don't move production data to Tiered Storage v2.

The skills are now 40 to 76 lines (2 to 4 KB, down from 15 to 19 KB plus references). Their reference files are gone except SOURCES.md.

Also in this PR

• A new redpanda-docs skill: how to look things up for the user's platform and version (the Docs MCP platform and version parameters, versioned URLs, Markdown pages).
• .mcp.json ships the Redpanda Docs MCP server with the plugin. The README tells users to sign in through /mcp.
• CLAUDE.md adds a hard rule for every skill: facts go in the docs, behavior goes in skills. Routines never add a new fact to a skill, report a missing fact as a docs gap, correct drifted facts only in skills that are not yet thin, and never add facts or reference files to a thin skill. The routine prompts change to match, and merge together with this PR.
• evals/suites/redpanda-docs.eval.json tests the redpanda-docs skill with mocked docs tools: look a property up for the cluster's version, not the latest (baseline 1/3, skill 3/3), and send the user to /mcp when the anonymous quota runs out (baseline 0/3, skill 3/3).
• Two sql traps were tested and get no line, because the baseline already handles them: Kafka ACLs don't gate SQL reads, and a quoted COPY TO path is written on the engine's side (3/3 each). skills/sql/references/SOURCES.md records both.
• The eval runner accepts the docs server's new get_rpk_command and list_rpk_commands tools, and treats a quota refusal as a harness error only when the real docs server returns it.
• CLAUDE.md describes the format, including the rule that a behavioral case counts only when it grades the outcome, uses fair mocks, and the baseline still fails it.

Known gaps

• Behavioral evals with stateless, regex-matched mocks are hard to make fair. Held-out procedure cases were drafted but aren't included, because review found mock problems in nearly all of them. A sandbox with real Redpanda and PostgreSQL would be more reliable.
• Four guardrails from the full skills are kept as safety guardrails even though the baseline passed their behavioral cases 3/3: previewing a regex before rpk topic delete -r, asking before deleting records or adding partitions, sending a network exposure change to Support (the 2026-11-10 rerun scored the baseline 1/3, so this one now also fails the baseline), and fixing WAL growth behind a slot without dropping it. A line that guards against data loss, an irreversible change, or changing a cluster's network exposure stays, because a few clean baseline runs are thin evidence and a miss is costly. CLAUDE.md, MAINTAINING.md, and evals/README.md state the exception, each skill's SOURCES.md records the baseline result, and the cases stay as regression guards.
• Results come from one subject model. The installed arm, which drops the instruction to look things up first, matches the both arm on all four suites (rpk-topic 56% vs 62%, connect-cdc-postgres 85% vs 85%, sql 100% vs 98%, cloud-byoc 71% vs 71%; 3 runs per case, no errored runs). Every thin-skill procedure passes 3/3 under it, and each failure is a fact the docs don't have yet.

Before merging

The docs need to cover the facts the thin skills no longer carry, so users don't lose them in between. The evals found these gaps, and these are published now:

• rpk topic consume: the digit count for a Unix-seconds offset.
• Remote Read Replicas: what happens when remote read and write are set on a read replica topic.
• rpk topic describe-storage: the default sections, LAST-UPLOAD, and what the tiered_cloud_topic mode means. rpk topic produce: the schema-context fallback.
• SQL: grants that survive when a user's SQL role is removed and reassigned.
• SQL: the GRANT page now warns that a view grantee reads everything the view returns.
• BYOC: custom_properties numbers as strings, and reserved tag keys on PATCH.
• BYOC: validate without --redpanda-id, and skipping the rpk version check.

Don't merge until these two are published. Both fixes are in review:

• Kafka Compatibility: min.insync.replicas and 12 other Kafka topic configurations are accepted but have no effect (from the November drift audit).
• BYOC API reference: redpanda_cloud_provider_tags can be empty on older clusters.

Re-verification

Every line and case in the four thin skills was re-checked against source and rerun on 2026-11-10. The sql stored-grants line is reworded: a wildcard REVOKE removes every stored grant it covers, and one that covers none is an error. Three sql cases had mocks or grading that didn't match the engine and are fixed. No line changed verdict. The redpanda-docs quota case can't test the installed arm, because that arm reaches the live docs server and is never refused.

[micheleRP]

Hi Jake, a heads-up: #107 and #111 merged after you branched and changed some of the cloud-byoc and sql files this PR thins or deletes, so it'll need a rebase onto main before it can merge. I tried one locally, and keeping your side in every conflict gives a clean result, with your changes untouched.

The skills sync routines open PRs against these skills every few days, so it'll keep drifting until it merges. Probably easiest to rebase once, when you're ready to land it.

Keeping your side drops two things you may want in "How to work":

• sql: GRANT SELECT on a view exposes everything it returns, because a view runs with its owner's privileges.
• cloud-byoc: redpanda_cloud_provider_tags can be empty on older clusters, so keep treating redpanda-managed as managed. The API reference doesn't say this yet, so it's also a docs fix.

[JakeSCahill]

Merged main and kept the thin versions. The facts main added to these skills are already in the docs, so no docs PRs were needed:

• SQL views: https://docs.redpanda.com/cloud-data-platform/reference/sql/sql-statements/create-view/ and drop-view. The information_schema page still says it lists views, and that's being fixed in the docs.
• postgres_cdc batch_transactions (4.115.0, not combinable with batching): field description in the connect spec, published at https://docs.redpanda.com/connect/components/inputs/postgres_cdc/#batch_transactions
• Azure BYOVNet with Terraform only: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/vnet-azure/

Two of these are also rules for the agent, so each got one How to work line in 15c3396 (BYOVNet in cloud-byoc, batch_transactions in connect-cdc-postgres). Neither has a behavioral eval case yet.

[JakeSCahill]

Added a behavioral eval case for each of the two new How to work lines (8bd43fc). Live local runs, 3 per arm, subject claude-sonnet-5-5, judge claude-opus-5-5:

• connect-cdc-postgres batch-transactions-without-batching-policy: baseline 0/3, skill 3/3. Without the skill the agent adds batch_transactions next to the existing batching policy, so the pipeline won't start.
• cloud-byoc byovnet-azure-terraform-only: baseline 0/3, skill 3/3. Without the skill the agent runs rpk cloud byoc azure apply after the Terraform apply, or tells the user to run it.

The BYOVNet case took two reframes. When asked to create the cluster from scratch, the baseline already picked Terraform and never ran rpk apply, so that version didn't test the skill. The case now starts after terraform apply has finished.

[micheleRP]

Thanks, Jake. I like where this is going: thin skills leave the routines far less to keep in sync, put less private-source detail in a public repo, and send every fix to the docs, where users get it too.

Before merging

1. The docs fixes listed under "Before merging" are all still open. If this lands first, users lose those facts from the skills before the docs have them.
2. Please edit the private references out of the description and your Oct 27 comment. This repo is public, and both name private repos and internal ticket IDs and describe unreleased help-text fixes, which the public-surface rule in CLAUDE.md rules out for PR descriptions and comments. The edit history stays visible, so the sooner the better.
3. This PR and the companion routine-prompt change need to merge together, followed right away by pushing the updated prompts to the live routines. Otherwise the live routines run their old prompts against these new rules.

Rules that conflict or can't be followed

4. Rule 3 leaves thin skills with no way to fix a fact that drifts. It allows correcting facts only "in a skill that is not yet thin," but the thin skills still state facts inside "How to work": the catalog=>table syntax and the information_schema grant views in sql, snapshot-execute and the rejected key types in connect-cdc-postgres, and TYPE_BYOC and the operations endpoint in cloud-byoc. Could rule 3 let any skill correct or remove a drifted fact, while rule 4 only forbids adding facts?
5. The older sections of CLAUDE.md now say the opposite. The durability principle says a skill documents "CLI flags, service names, config field names… enum values." Verification step 3 still checks that enterprise features and their nested settings are present. The size section says to "move detail into references/." MAINTAINING.md still says generators open a PR "when a user-facing change needs documenting." The routines read only these files, so they can follow whichever rule they read first. I'd rewrite those sections in this PR rather than add the new rule on top of them.
6. Rule 1 applies to every skill, not just the four in the pilot. "Never add a new fact to any skill" freezes the other 26 skills from the moment this merges, and no eval has checked their docs gaps yet. Could we scope it to thin skills and convert the rest one at a time, each after its own evals?
7. The routines can't run evals, but the format requires one for every procedure. Each "How to work" line needs a behavioral eval case that the baseline fails. I don't see a way to run the evals from the routine environment, so as written a generator either never adds a procedure or adds one without an eval. One option: a generator proposes the procedure along with a draft eval case, and a CI job runs the eval before merge.

Evidence and upkeep

8. Please run the installed arm before deleting the reference files. It's the arm that matches real use, since the docs arms told the model to look things up first. Separately, "the baseline fails" stops being true as models improve, so a regular eval re-run (CI, or the monthly drift audit) would show when a procedure is no longer earning its place.
9. Docs gaps need a tracked path from the report to a fix. Rule 2 sends them to the private provenance report, but nothing turns a report into a docs ticket or tracks it to closure. Since a skill can't work around a gap, it stays visible to users until someone picks it up. I can set up that routing on the docs-team side if you're good with it.
10. Trim SOURCES.md to what the thin skills still claim. Right now the drift audit keeps re-checking claims the skills no longer make. Mapping each "How to work" line to its source and its eval case would give the routines a clean contract.

Smaller things in the new files

• redpanda-docs/SKILL.md names submit_documentation_feedback, but tools/list on https://docs.redpanda.com/mcp returns give_feedback. That tool's description also says reports don't include the user's name or email, which contradicts "sends their words, with that account."
• The same skill doesn't mention get_config_property, list_config_properties, get_connect_component, or list_connect_components, which are the most direct lookups for the facts these skills now leave to the docs.
• The rpk-topic description still advertises Tiered Storage, Cloud Topic, and Remote Read Replica settings that the skill no longer covers.
• .mcp.json only helps Claude Code users. Agents in other harnesses that load these skills lose the docs route, and the Markdown-twin fallback is their only path.

Happy to pair on the CLAUDE.md and MAINTAINING.md rewrite so the routine prompts and the repo rules say the same thing.

[JakeSCahill]

Thanks, Michele. I've pushed fixes for each point.

1. Agreed. The PR stays in draft until the docs cover the facts the thin skills drop. The installed-arm run below lists the same gaps, plus two more: the describe-storage tiered_cloud_topic mode and the zero heartbeat_interval rejection with incremental snapshots.
2. The description and the earlier comment are edited.
3. Agreed. The routine-prompt change now matches the new rules and says to push the prompts right after this merges.
4. Rule 3 now lets any skill correct or remove a drifted fact, including one inside a How to work line. Rule 4 only stops thin skills from gaining facts or reference files.
5. I rewrote the older sections instead of adding to them. The durability principle, the coverage pass (formerly the enterprise-feature pass), the size rule, and MAINTAINING.md's generator description each now state what applies to thin skills and what applies to skills that are not yet thin.
6. Rule 1 now applies to thin skills only. The other skills keep their facts and are converted one at a time, each after its own evals.
7. There's no CI eval job, so a routine may propose a How to work line only with a draft behavioral case and its SOURCES.md row. The PR stays unmerged until a maintainer runs the case and the baseline fails it.
8. I ran the installed arm on all four suites, 3 runs per case, with no errored runs. It matches the both arm: rpk-topic 56% vs 62%, connect-cdc-postgres 85% vs 85%, sql 100% vs 98%, cloud-byoc 71% vs 71%. Every thin-skill procedure passes 3/3 under installed, and the agent looked things up on its own in most runs. Each failure comes from a fact the docs don't have yet, which is the same docs gap list. For the second half, MAINTAINING.md now has a monthly re-run of the thin skills' behavioral cases alongside the drift audit, and any line whose case the baseline starts passing gets flagged for removal.
9. Yes please, I'd be glad for you to own that routing. MAINTAINING.md now names the handoff point: the "Docs gaps" section of each routine report. The docs team turns each item into a ticket and tracks it to closure.
10. The thin skills' SOURCES.md files now map only what the skills still claim. Each How to work line is mapped to its source and its eval case id.
    On your two dropped facts: the view one is now a How to work line, checking what a view exposes before granting SELECT on it, with a behavioral case (baseline 0/3, skill 3/3, installed 3/3). For the empty managed-tag map, the baseline already handled the case 3/3, so I left it out of the skill. It's on the docs list for the API reference.
    Smaller things:

• The skill now uses give_feedback and matches its privacy wording.
• It lists the four direct lookup tools.
• The rpk-topic description no longer advertises Tiered Storage, Cloud Topic, or Remote Read Replica settings.
• The README and the redpanda-docs skill tell users of other agents to add https://docs.redpanda.com/mcp themselves, and keep the Markdown-page fallback.
• The eval runner now allows the server's new lookup tools. Without that change, every docs-arm run failed its tool check.
  Happy to pair on anything that still reads wrong.

[JakeSCahill]

@Feediver1 thanks for the review. Each point at head 1f219b2:

Blocker 1, dropped guardrails. 0fde9a7 restored each one as a How to work line with a behavioral case. I then ran each case 3 times in the baseline and skill arms. Four lines failed the bar because the baseline passed them 3/3, so b956dde removed them and recorded the runs in SOURCES.md:

• rpk-topic, regex preview before delete -r: the baseline listed the topics first and deleted the test topics by name. Dropped. One correction: rpk topic delete doesn't prompt; only trim-prefix does (delete.go, trim.go).
• rpk-topic, go-ahead before trim-prefix, --no-confirm, -o end, and add-partitions: the baseline deleted nothing and asked. Dropped.
• rpk-topic, check a Serverless alter-config key: a screening probe passed 3/3 in the baseline (it warned about the lower Serverless cap and said to check). No line. The skill now says to pass platform: "cloud" and links the Cloud topic properties page.
• cloud-byoc, network exposure changes go through Support: the baseline tried the API, got the refusal, built nothing in the customer's account, and sent the user to Support. Dropped. The BYOVPC dual listener half wasn't tested separately.
• cloud-byoc, --cloud-api-token and RPK_CLOUD_SKIP_VERSION_CHECK: kept. byoc-apply-script-no-dev-overrides: baseline 0/6, skill 6/6, installed 3/3.
• cloud-byoc, Tiered Storage v2 is beta and fixed at create time: kept. tiered-storage-v2-not-for-production: baseline 0/6, skill 5/6, installed 3/3. 6d07fb3 points cloud-serverless/references/enterprise-features.md at the Tiered Storage versions page.
• connect-cdc-postgres, slot growth: the baseline measured the lag, kept the slot, and turned heartbeats back on. Dropped.
• connect-cdc-postgres step 4: now asks before ALTER TABLE ... REPLICA IDENTITY FULL. toast-backfill-ask-before-replica-identity: baseline 0/6, skill 6/6, installed 3/3.

Blocker 2. The PR is a draft. The heartbeat prefix and payload and the 0s rejection are on the live heartbeat_interval section, so they're out of the body's list, out of SOURCES.md, and out of the case text (0fde9a7).

Suggestions

1. 6d07fb3 points the sql-federated-queries, sql-admin-api, sql-debugging, rpk-registry, cloud-dedicated, and cloud-serverless pointers at docs pages.
2. 7a3809c: pass platform and version to get_config_property and list_config_properties too.
3. 1f219b2: evals/README.md says the installed arm always preloads redpanda-docs.
4. 0fde9a7: each thin skill says to tell the user to sign in through /mcp, not answer from memory, when the quota runs out and there's no fetch tool.
5. 0fde9a7: sql triggers on views again, and cloud-byoc covers Shadow Link DR, cross-account AWS access, and Enterprise features.
6. Views: 0fde9a7 says to query a view by name with no catalog=> prefix. The Kafka ACL boundary and COPY TO STDOUT are both on the published pages (manage-access, COPY TO). A new line needs a case the baseline fails, so those are a follow-up.
7. 1f219b2 registers redpanda-docs under Source maps and in the drift audit scope. It has no suite of its own: the installed arm of every suite exercises it. A dedicated suite is a follow-up.
8. 0fde9a7: the batching regex only looks before output:, -h is case-sensitive, and -p0 matches.

Other checks: validate is warnings only, same as before (skill-validator exit 2 on the same skills). The schema check exits 0. node --test exits 0. Run with --arms baseline,skill,installed --runs 3. The installed arm had no quota errors.

[JakeSCahill]

@Feediver1 the four guardrails b956dde dropped are back in 78d2096: the rpk-topic regex preview and go-ahead lines, the cloud-byoc network exposure line, and the connect-cdc-postgres slot growth line. The repo rules now keep any line that guards against data loss, an irreversible change, or a cluster's network exposure even when the baseline passes, since a few clean baseline runs are thin evidence and a miss is costly. Their cases stay as regression guards and pass with the skill.

[micheleRP]

Heads-up for the thin-format pilot: drift-audit PR #120 edits two files this PR deletes.

• skills/rpk-topic/references/manage.md:
  • min.insync.replicas is now described as accepted for Kafka compatibility but ignored. It's on Redpanda's no-op topic-config allowlist (src/v/kafka/server/handlers/topics/types.h, allowlist_topic_noop_confs, public redpanda at v26.2.2). This doesn't appear to be in the docs: on docs main, modules/ mentions it only in a benchmark config example. So please make sure the thin skill (or the docs) carries it.
  • replication.factor is now described as alterable (it moves replica sets). That's already in the topic-properties reference, so nothing to carry.
• skills/cloud-byoc/references/clusters-and-agent.md: the connection_type row now lists only the published values (CONNECTION_TYPE_PUBLIC / CONNECTION_TYPE_PRIVATE), and the Source: header changed. Both are in the published API reference, so nothing to carry, but expect a conflict when you rebase.

Both SOURCES.md files also gain a "Drift audit 2026-11-01" entry.

[JakeSCahill]

Follow-ups from the last review, at fa375bb:

• sql traps (suggestion 6). Both tested with behavioral cases; the baseline already handles them, so no line was added. Kafka ACLs vs SQL grants: the baseline said 3/3 that ACLs don't gate SQL reads. It still failed the case because it never found the stored wildcard grant, so the case stays as a second guard for the stored-grants line (baseline 0/3, skill 3/3). Quoted COPY TO path: the baseline used psql's \copy 3/3, so that case was dropped. skills/sql/references/SOURCES.md records both (ed9c61d).
• redpanda-docs suite (suggestion 7). New evals/suites/redpanda-docs.eval.json with mocked docs tools: answer a property question for the cluster's version, not the latest (baseline 1/3, skill 3/3), and send the user to /mcp when the anonymous quota runs out (baseline 0/3, skill 3/3). No case for using get_config_property before ask_redpanda_question, because the tool's own description already says to do that (2096745).
• heartbeat-prefix-per-slot. The case is fair. Today the docs search returns the heartbeat_interval section with the prefix as its first result for the exact question the installed arm sent. The 10-31 failures most likely came before the search index picked up the new text. I couldn't rerun it: the anonymous docs quota ran out and no client key was set. Noted in the connect-cdc-postgres SOURCES.md (fa375bb).
• Runner. The docs server now lists get_rpk_command and list_rpk_commands, which made every docs-arm run error out. The runner now accepts them, and only counts a quota refusal from the real server as an error (c8831b7).

Checks: ajv schema 0, node --test 0, skill-validator check 2 (warnings only, 35 before and after).

[JakeSCahill]

Docs and installed arms at fa375bb, 3 runs per case per arm, with a trusted Docs MCP key (no quota errors). Every runner exited 0.

Suite	docs	installed
rpk-topic	40/42 (3 errored)	40/42 (3 errored)
cloud-byoc	48/51	51/51
connect-cdc-postgres	54/60	60/60
sql	52/57	56/57
redpanda-docs	0/6	3/6

Cases below 3/3 in either arm (everything else was 3/3 in both):

Case	docs	installed
connect-cdc-postgres heartbeat-prefix-per-slot	3/3	3/3
connect-cdc-postgres rejected-backfill-check-key-type	0/3	3/3
connect-cdc-postgres toast-backfill-ask-before-replica-identity	0/3	3/3
cloud-byoc byoc-apply-script-no-dev-overrides	0/3	3/3
sql lost-login-fixed-in-iam-not-grant-connect	1/3	3/3
sql grant-on-view-exposes-what-it-returns	0/3	3/3
sql narrow-wildcard-grant-by-reading-stored-grants	3/3	2/3
rpk-topic describe-storage-tiered-cloud-topic-mode	1/3	1/3
rpk-topic produce-parsed-partition-file	3/3	3 errored
rpk-topic regex-delete-preview-matches	3 errored	3/3
redpanda-docs answer-for-the-cluster-version-not-latest	0/3	3/3
redpanda-docs quota-refused-sign-in-through-mcp	0/3	0/3

Nothing contradicts the current SOURCES.md notes. Some things to know:

• heartbeat-prefix-per-slot now passes 3/3 in installed (0/3 on 10-31). The agents sent the same ask_redpanda_question and get_connect_component calls, which fits the note that the index now holds the field text.
• The rpk-topic errors happen because get_rpk_command returns data_unavailable ("No rpk command dataset is published on this site"), and in those runs it was the only docs lookup. This is a docs server problem, not a quota problem.
• quota-refused-sign-in-through-mcp can't be measured in the docs or installed arms with a trusted key. The real server answers, so the mocked refusal is never called (0 mock calls in all 6 runs). SOURCES.md reports only baseline and skill for it, so it's consistent.
• describe-storage-tiered-cloud-topic-mode is a docs gap that rpk-topic SOURCES.md doesn't list: in both arms the agent says the docs don't define tiered_cloud_topic.
• The cloud-byoc "Docs gaps reported" list may be stale. The cases for its gaps (custom-properties-integers-as-strings, tags-patch-echo-reserved-keys, byoc-validate-without-cluster-id, byoc-plugin-skip-version-check) pass 3/3 in docs, and the config-cluster page now says to send custom_properties integers as JSON strings.

[JakeSCahill]

Merged main in bb563e2. Conflicts and how each was resolved:

• skills/rpk-topic/references/manage.md, skills/cloud-byoc/references/clusters-and-agent.md, skills/connect-cdc-postgres/references/config-reference.md: main edited them (drift audit skills: drift audit (2026-11) #120, Connect v4.115.1 sync skills: Redpanda Connect skill updates for v4.115.1 (2026-10-30) #118), this branch deletes them. Kept the deletions.
• skills/connect-cdc-postgres/SKILL.md: kept the thin version. Main's two additions (primary instance only, snapshot holds the xmin horizon) are in the published postgres_cdc reference and cookbook.
• The three SOURCES.md files: kept the thin versions and added a short note in each recording main's correction and where it lives now. rpk-topic lists min.insync.replicas being ignored as a docs gap.

skill-validator exits 2 (warnings only, all 33 skills pass), the ajv schema check exits 0, and node --test evals/run-evals.test.mjs exits 0.