Repository navigation
skills: convert the last five skills to thin behavioral guidance - #132
Open
JakeSCahill wants to merge 48 commits into
Open
JakeSCahill wants to merge 48 commits into
JakeSCahill wants to merge 48 commits into
Conversation
Add a runner that asks each case in four arms (no skill or docs, skill only, Docs MCP only, both) and grades every answer the same way, so a skill's value over the published docs can be measured. Subjects run isolated from plugins, CLAUDE.md, memory, and the skill files on disk, and each run checks the tools its arm actually had. Every case must fail an empty answer before any subject runs. Docs quota refusals count as errors, not results. Pilot suites for rpk-topic, connect-cdc-postgres, sql, and cloud-byoc.
Check the skills, slash commands, and plugins each subject reports at start, not just its tools, so a leaked plugin errors the run instead of being scored. Record input, cache, and output tokens per run.
A case can declare a mocked tool surface, served to every arm by a small stdio MCP server, and assert on the calls the subject made: first call, expected, forbidden, and ordered calls. Calls are graded from the mock's log, not the transcript.
The installed arm gives the subject the skill, the shared redpanda-docs skill and the docs server with no instruction to look things up, which is what a plugin user gets. The other docs arms keep the instruction, so they measure what the docs contain. Each row now records the subject and judge models, the runner commit and a hash of the skill text, and report mode refuses to pool rows from different skill text or models. A docs lookup that comes back as an upstream error or timeout is a harness error, not a docs answer. Suites with unknown case keys stop the run, and --no-judge no longer dies on invariant-only cases.
rpk-topic, connect-cdc-postgres, sql, and cloud-byoc now carry only the product-specific procedure an agent wouldn't follow on its own, each backed by a behavioral eval case, and point at the docs for everything else. Their reference files are gone except SOURCES.md. Add a redpanda-docs skill for using the docs at the user's platform and version, and ship the Redpanda Docs MCP server with the plugin. Describe the format in CLAUDE.md so the sync routines follow it.
A review found that several behavioral cases graded the method rather than the outcome, and that their mocks errored on legitimate commands. Graded on outcome with fair mocks, the baseline passes most of them. This drops the thin-skill steps those cases had backed and fixes the cases: outcome grading, fuller mocks, looser literals, and why text that matches the recorded runs. What remains: rpk-topic's parsed-partition rule, sql's catalog, login, and stored-grants procedures, and connect-cdc-postgres's two backfill procedures. cloud-byoc keeps no procedure. The sql wildcard step now asks the user to confirm before narrowing access.
Keep the thin versions of sql, connect-cdc-postgres, and cloud-byoc. The facts main added to them (SQL views, batch_transactions, Azure BYOVNet via Terraform) are in the Redpanda docs. Keep both README sections.
Add one How to work line each for rules main added as reference text: create an Azure BYOVNet cluster with Terraform alone, and never set postgres_cdc batch_transactions together with a batching policy.
batch-transactions-without-batching-policy (connect-cdc-postgres) and byovnet-azure-terraform-only (cloud-byoc). Live runs, 3 each: baseline 0/3 and skill 3/3 on both.
Keep the thin sql skill. The schema, ALTER ... OWNER TO and EXPLAIN facts that redpanda-data#115 added are on the published pages (create-schema, drop-schema, alter-owner, explain), so the skill reaches them through the docs MCP.
Make it a hard rule for every skill: never add a new fact, report a missing fact as a docs gap, correct drifted facts in skills that are not yet thin, and never add facts or reference files to a thin skill.
The docs server now lists get_config_property, list_config_properties, get_connect_component, list_connect_components and give_feedback, so every docs-arm run failed the tool check. Allow the four lookups and deny give_feedback, which replaced submit_documentation_feedback.
A view runs with its owner's privileges, so SELECT on it lets the grantee read everything it returns. Adds the How to work line and a behavioral case: baseline 0/3, skill 3/3, installed 3/3.
Names the docs server's real feedback tool and matches its privacy wording, lists the property and Connect component lookups, and tells users of other agents how to add the server. Drops the Tiered Storage, Cloud Topic and Remote Read Replica claims from the rpk-topic description, which the thin skill no longer covers.
Each How to work line now maps to its source and its eval case. Claims the thin skills no longer make are gone, so the drift audit stops re-checking them. Docs gaps the evals found are listed separately, as items for the docs, not claims to re-verify.
Rule 1 now applies only to thin skills; the others keep their facts and are converted one at a time, after their own evals. Any skill may correct or remove a drifted fact. The durability principle, the coverage pass, the size rule and the generator description now each state the thin and not-yet-thin cases. Adds the process for a new How to work line (a draft behavioral case, run by a maintainer before merge), a monthly re-run of the behavioral cases with the drift audit, and the handoff of docs gaps from the provenance report to the docs team. Documents the installed arm in the evals README.
The CREATE VIEW page already warns that a view grantee reads everything the view returns; only the GRANT page lacks the warning.
The docs section listed only three Docs MCP tools, two of them for API references. Follow redpanda-docs like the other thin skills, and name get_connect_component for postgres_cdc and its fields.
Each line got a behavioral case: rpk-topic's get-the-cluster-version step and cloud-byoc's confirm-before-change step. The baseline passed both 3/3, so under the thin format neither line earns its place. SOURCES.md records the removal. Also name the direct property lookups in both docs sections: get_config_property and list_config_properties for topic properties, and get_config_property with platform cloud for BYOC cluster properties.
Non-thin skills sent readers to thin skills for content those skills no longer carry. Link the docs pages directly.
Restore the rpk-topic, cloud-byoc, and connect-cdc-postgres guardrails as How to work lines with behavioral cases, ask before REPLICA IDENTITY FULL, add a quota fallback, widen the sql and cloud-byoc triggers, drop the two published postgres_cdc gaps, and loosen three eval regexes.
The baseline passed the cases for the rpk-topic regex preview and destructive-change lines, the cloud-byoc network exposure line, and the connect-cdc-postgres WAL growth line 3/3. Record the results in SOURCES.md and drop the lines and their cases.
Register the redpanda-docs SOURCES.md in the source maps and drift audit scope, and say in the evals README that the installed arm always preloads redpanda-docs.
Restore the rpk-topic regex preview and go-ahead lines, the cloud-byoc network exposure line, and the connect-cdc-postgres slot growth line, with their cases as regression guards. A line that guards against data loss, an irreversible change, or changing a cluster's network exposure now stays even when the baseline passes, because a few clean runs are thin evidence and a miss is costly. CLAUDE.md, MAINTAINING.md, and the evals README state the exception; SOURCES.md records each line as a safety guardrail with its baseline result.
The live Docs MCP server now lists get_rpk_command and list_rpk_commands, so every docs arm errored on unexpected tools. Count a quota refusal as a harness error only when the real docs server returns it, so a behavioral case can mock one.
The baseline already knows Kafka ACLs do not limit SQL reads and uses psql's \copy for a local file (3/3 each), so neither gets a line. SOURCES.md records both. The ACL case stays as a second case for the stored-grants line: the baseline fails it 3/3, the skill passes 3/3.
Two behavioral cases with mocked docs tools: look up a property for the cluster's version rather than the latest (baseline 1/3, skill 3/3), and send the user to /mcp when the anonymous quota runs out (baseline 0/3, skill 3/3).
The Docs MCP search returns the heartbeat_interval section with the prefix as its first result for the question the installed arm sent, so heartbeat-prefix-per-slot is fair as written.
Keep the thin versions of rpk-topic, connect-cdc-postgres, and cloud-byoc, and record main's corrections to their pre-thin references in each SOURCES.md.
Conflicts: - skills/cloud-byoc/references/clusters-and-agent.md: main added the tier-only scaling rule (redpanda-data#124); this branch deletes the file. Kept the deletion. The published scale-cluster page states the rule, and SOURCES.md records where it lives now. - skills/cloud-byoc/SKILL.md: kept the thin description and added main's pointer to /redpanda:cloud-byoc-sizing (redpanda-data#91). - skills/cloud-byoc/references/SOURCES.md: kept the thin version. - CLAUDE.md, MAINTAINING.md: kept both sides (redpanda-docs entry and the thin-skill rules, plus main's cloud-byoc-sizing entries).
JakeSCahill
changed the title
The REST gateway derives update_mask from the request body when it is left out, so the API reference omits it by design. Remove the docs-gap entry, record the correction, and let the case pass with or without the mask.
JakeSCahill
force-pushed
the
skills/thin-last-batch
branch
from
November 10, 2026 20:16
2fe0687 to
7bd93aa
Compare
The re-verification rerun scored the baseline 0/3 on stop-moves-balancer-off-then-cancel, so the line meets the keep rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #110. Review the last 11 commits; the rest are #110's and drop out once it merges.
Converts the last five skills to thin skills with the method from #110:
cloud-dedicated,cloud-serverless,streaming,streaming-admin-apiandstreaming-debugging. With #127 to #130, every remaining skill except the hand-maintainedcloud-byoc-sizingcalculator is now thin. The Connect skills are removed in #133 in favor of theredpanda-connectplugin, soconnectandconnect-debuggingare no longer in this PR. Each skill here is 55 to 70 lines, and every reference file exceptSOURCES.mdis gone.How to work lines kept
Rates are baseline / skill / installed.
cleanup.policy=compactdeletes records)iceberg_default_catalog_namespacewhen clusters share a catalogpartition_autobalancing_modeoff first, thenmove-cancelredpanda.remote.delete=falsebefore deleting a Tiered Storage topic whose archive must stayFailOverwithoutshadow_topic_name(or with a misspelled field) fails over the whole linkupdate_masknaming a repeated field appends; to remove an entry, send the parent wholepartition_autobalancing_modeoff first, thenmove-cancelFor the two
streaminglines the facts are in the docs; agents don't look them up mid-task, so the lines stay.Dropped because the baseline already follows them, recorded in each
SOURCES.md: tag PATCH replaces the map, shadow link updates from the full config, self-test at peak, and asking before seeking a group forward.Docs and source fixes
Each
SOURCES.mdlists the page and the source.${secrets.<id>}password, but the migration page's Cloud API example sends an inline password; only the BYOC page says network-exposure changes go through Support.INVALID_ARGUMENT; the published Data Plane API spec lacksinclude_draftsandSTATE_DRAFT. To confirm: the Data Plane API says user quotas aren't supported, while rpk acceptsuserentities.force_replicason the controller partition needsevil_mode=true; theUpdateShadowLinkupdate_maskappends to repeated fields.force_recover_from_nodesreference says the body is an array of node IDs, but the handler needsdead_nodesandpartitions_to_force_recover; the CPU profilewait_mshas no description.Checks
Every skill has a
SOURCES.md,skill-validatorhas no failures (warnings only),node --test evals/run-evals.test.mjspasses, and the five suites validate againstevals/schema.json. The evals need Node 26, as in #110. Total eval cost about $210 for these five skills.One correction after the eval run: the cloud-dedicated agent first recorded a missing
update_maskonUpdateClusteras a docs gap. It isn't one: the REST gateway derives the mask from the request body when it's left out, so the reference omits it by design. The last commit removes that entry and rewrites the case to accept either form.Re-verification
Every line and case was re-checked against product source and rerun on 2026-11-10. Five cases with mocks or grading that didn't match the real API are fixed (
serverless-privatelink-setup,add-partitions-to-reach-total,pipeline-slots-hidden-drafts,iceberg-second-cluster-shared-catalog,shadow-link-update-full-config). The balancer-off line is restored tostreaming-debugging. No other line changed verdict.