Skip to content

skills: convert the three remaining SQL skills to thin behavioral guidance - #129

Merged
JakeSCahill merged 10 commits into
redpanda-data:mainfrom
JakeSCahill:skills/thin-sql-family
Nov 12, 2026
Merged

JakeSCahill merged 10 commits into
redpanda-data:mainfrom
JakeSCahill:skills/thin-sql-family

Conversation

@JakeSCahill

@JakeSCahill JakeSCahill commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #110. Review the last 7 commits; the rest are #110's and drop out once it merges.

Converts the three remaining Redpanda SQL skills to thin skills with the method from #110: sql-admin-api, sql-debugging and sql-federated-queries. Each gets its own eval suite (fact cases plus behavioral cases on mocked tools), run in all five arms, 3 runs each. Each skill is now 52 to 67 lines, and every reference file except SOURCES.md is gone. The Docs MCP covers the SQL docs under https://docs.redpanda.com/cloud-data-platform/ with platform: "cloud".

How to work lines kept

Rates are baseline / skill / installed.

Skill Line Rates Why
sql-admin-api Save the SQL state (catalogs, tables, grants) before disabling 0/3, 3/3, 3/3 Baseline fails; also a safety guardrail, because disabling deletes the engine metadata for good
sql-admin-api Don't disable SQL to reach another goal (such as moving its zone); say what is lost and leave the call to the user or Support 1/3, 3/3, 3/3 Safety guardrail. The baseline never disabled SQL, but hedged on whether disabling loses the state
sql-admin-api On AWS BYOVPC, detach the Glue policy from the SQL engine's IAM role before disabling 0/3, 3/3, 3/3 Baseline fails
sql-debugging A table returns fewer rows than its topic, or decode errors: read its mapping in system.kafka_sources first (DROP_RECORD skips bad records silently) 0/3, 3/3, 3/3 Baseline fails
sql-debugging Before deleting an Iceberg-enabled topic, set redpanda.iceberg.delete=false if the history must stay 3/3, 3/3, 3/3 Safety guardrail (data loss)
sql-federated-queries none The one behavioral case (fix a rotated REST catalog secret on the cluster side) passed 3/3 in the baseline

sql-federated-queries keeps no procedure, and the docs arm answered all 13 of its fact cases. By evals/README.md it is a candidate for deletion rather than a thin skill; this PR keeps it as a routing stub for reviewers to decide.

Dropped because the baseline already follows them, recorded in each SOURCES.md: finding a stuck query through system.queries and system.execs, and the rotated catalog secret fix.

Docs gaps

Facts the published docs lack or get wrong. Each SOURCES.md lists the page and the source.

  • The DESCRIBE page is wrong: DESCRIBE REDPANDA CATALOG <name> returns the catalog's connection row (name, options, iceberg_catalog), not a list of tables and topic mappings. The old skill said the same wrong thing.
  • Degraded SQL node: the troubleshooting page says the node needs a reboot, but not that the user can't do it and should contact Support with the degradation_error text. In the failing docs run, the agent told the user to delete the SQL pod with kubectl.
  • SQL debug bundle: the rpk sql debug bundle reference doesn't tell a Cloud user, who can't reach the pods, to open a Support ticket with the cluster ID, the time and the error.
  • SQL metrics, needs a decision: the Cloud monitoring page says the Prometheus endpoint exports broker and connector metrics, while the SQL out-of-memory page tells users to track an engine gauge without saying where to scrape it. One of the two is incomplete; the SQL team should say which.
  • SQL zone pages: the code refuses a zone change while SQL is enabled and allows one after disable and re-enable, as the BYOVPC AWS page says. The pages that say the zone can never change are wrong, and so is the Terraform page's claim that an omitted zone is chosen automatically (the cluster's first zone is used). The BYOVPC enable pages also overstate the grouped update_mask case: a fully grouped mask applies nothing, and a mask with rpsql.enabled as a leaf path is rejected.

Checks

Every skill has a SOURCES.md, skill-validator has no failures (warnings only), node --test evals/run-evals.test.mjs passes, and the three suites validate against evals/schema.json. The evals need Node 26, as in #110. Total eval cost about $66.

Re-verification

Every line and case was re-checked against product source and rerun on 2026-11-10. The BYOVPC leaf-path line is dropped: its case wrongly failed a PATCH with no update_mask (the REST gateway derives one), and once fixed the baseline passed 3/3. The rotated-catalog-secret-fixed-on-cluster case mocks mishandled quoted arguments and are fixed. No other line changed verdict.

@micheleRP micheleRP left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the seven commits on top of #110. Each skill has only "How to work" and "Use the docs", and every line maps in SOURCES.md to a source and an existing case. The Iceberg-delete and zone lines are recorded as safety guardrails with their baseline results. I checked the facts behind the lines against the published pages: disable clears catalog, mapping, and grant state (deploy-sql-cluster.adoc:241); detach the Glue policy before disabling (enable-sql-byovpc-aws.adoc:380); both grant views exist (information-schema.adoc:43); error_handling_policy, DROP_RECORD, and FILL_NULL are documented (system-virtual-tables.adoc, create-table.adoc:48); and redpanda.iceberg.delete defaults to true. The "oxla" strings are user-visible literals the docs also show. It merges cleanly onto #110 and main.

Docs gaps: the DESCRIBE and degraded-node items are fixed by cloud-docs#844 (merged today). The debug-bundle item (rpk help), the metrics decision (SQL team), and the zone pages have no PR yet.

Non-blocking:

  1. sql-federated-queries keeps no procedure, so per evals/README.md it's a deletion candidate. I'm fine keeping it as a routing stub for now.
  2. The sql-admin-api description says "SQL Manage or SQL Access roles", but the docs name them the SQL: Manage and SQL: Access permissions.

Drop the BYOVPC leaf-path line: once its case accepts an omitted update_mask, the baseline passes. Fix the case to accept that request, reject a wrapped body, and match escaped quotes. Record rerun rates, two docs gaps, and the resolved zone item.
Update each line's rates to the rerun against the thin skill.
Let the rotated-secret case accept quoted rpk arguments and record the rerun rates.
@JakeSCahill
JakeSCahill force-pushed the skills/thin-sql-family branch from 0800591 to 8146f7e Compare November 12, 2026 20:26
@JakeSCahill
JakeSCahill merged commit 55d74e3 into redpanda-data:main Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants