Repository navigation
skills: convert the three remaining SQL skills to thin behavioral guidance - #129
Conversation
micheleRP
left a comment
There was a problem hiding this comment.
Approving the seven commits on top of #110. Each skill has only "How to work" and "Use the docs", and every line maps in SOURCES.md to a source and an existing case. The Iceberg-delete and zone lines are recorded as safety guardrails with their baseline results. I checked the facts behind the lines against the published pages: disable clears catalog, mapping, and grant state (deploy-sql-cluster.adoc:241); detach the Glue policy before disabling (enable-sql-byovpc-aws.adoc:380); both grant views exist (information-schema.adoc:43); error_handling_policy, DROP_RECORD, and FILL_NULL are documented (system-virtual-tables.adoc, create-table.adoc:48); and redpanda.iceberg.delete defaults to true. The "oxla" strings are user-visible literals the docs also show. It merges cleanly onto #110 and main.
Docs gaps: the DESCRIBE and degraded-node items are fixed by cloud-docs#844 (merged today). The debug-bundle item (rpk help), the metrics decision (SQL team), and the zone pages have no PR yet.
Non-blocking:
sql-federated-querieskeeps no procedure, so perevals/README.mdit's a deletion candidate. I'm fine keeping it as a routing stub for now.- The
sql-admin-apidescription says "SQL Manage or SQL Access roles", but the docs name them the SQL: Manage and SQL: Access permissions.
Drop the BYOVPC leaf-path line: once its case accepts an omitted update_mask, the baseline passes. Fix the case to accept that request, reject a wrapped body, and match escaped quotes. Record rerun rates, two docs gaps, and the resolved zone item.
Update each line's rates to the rerun against the thin skill.
Let the rotated-secret case accept quoted rpk arguments and record the rerun rates.
0800591 to
8146f7e
Compare
Stacked on #110. Review the last 7 commits; the rest are #110's and drop out once it merges.
Converts the three remaining Redpanda SQL skills to thin skills with the method from #110:
sql-admin-api,sql-debuggingandsql-federated-queries. Each gets its own eval suite (fact cases plus behavioral cases on mocked tools), run in all five arms, 3 runs each. Each skill is now 52 to 67 lines, and every reference file exceptSOURCES.mdis gone. The Docs MCP covers the SQL docs under https://docs.redpanda.com/cloud-data-platform/ withplatform: "cloud".How to work lines kept
Rates are baseline / skill / installed.
system.kafka_sourcesfirst (DROP_RECORDskips bad records silently)redpanda.iceberg.delete=falseif the history must staysql-federated-querieskeeps no procedure, and the docs arm answered all 13 of its fact cases. Byevals/README.mdit is a candidate for deletion rather than a thin skill; this PR keeps it as a routing stub for reviewers to decide.Dropped because the baseline already follows them, recorded in each
SOURCES.md: finding a stuck query throughsystem.queriesandsystem.execs, and the rotated catalog secret fix.Docs gaps
Facts the published docs lack or get wrong. Each
SOURCES.mdlists the page and the source.DESCRIBE REDPANDA CATALOG <name>returns the catalog's connection row (name,options,iceberg_catalog), not a list of tables and topic mappings. The old skill said the same wrong thing.degradation_errortext. In the failing docs run, the agent told the user to delete the SQL pod with kubectl.rpk sql debug bundlereference doesn't tell a Cloud user, who can't reach the pods, to open a Support ticket with the cluster ID, the time and the error.update_maskcase: a fully grouped mask applies nothing, and a mask withrpsql.enabledas a leaf path is rejected.Checks
Every skill has a
SOURCES.md,skill-validatorhas no failures (warnings only),node --test evals/run-evals.test.mjspasses, and the three suites validate againstevals/schema.json. The evals need Node 26, as in #110. Total eval cost about $66.Re-verification
Every line and case was re-checked against product source and rerun on 2026-11-10. The BYOVPC leaf-path line is dropped: its case wrongly failed a PATCH with no
update_mask(the REST gateway derives one), and once fixed the baseline passed 3/3. Therotated-catalog-secret-fixed-on-clustercase mocks mishandled quoted arguments and are fixed. No other line changed verdict.