Repository navigation
ci: run the affected evals on pull requests and every eval weekly - #119
Draft
JakeSCahill wants to merge 1 commit into
Draft
JakeSCahill wants to merge 1 commit into
JakeSCahill wants to merge 1 commit into
Conversation
Same-repository pull requests run the cases their change can reach in the baseline, skill, and installed arms, 3 runs each. A How to work case named in a skill's SOURCES.md fails the check when the baseline passes it, when a skill arm fails it, or when too many runs errored to tell. Other cases are reported only. Harness and workflow changes run a small smoke set chosen by rule. Fork pull requests get a notice. A weekly sweep runs every suite in every arm and lists any How to work case whose baseline now passes. Results go to the job summary and to artifacts only. Model calls go through the AI Gateway with a per-job token minted from OIDC and Secrets Manager, with no API key fallback. The runner now redacts credentials from a run's error reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.github/workflows/evals.yml, which runs the skill evals in CI.redpanda-docsruns every suite's How to work cases. A change to the runner, its scripts, or the workflow runs a two-case smoke set. A pull request runs at most 20 cases; the cap drops only advisory cases.SOURCES.md"How to work" table) is passed by the baseline in more than a third of its runs, is passed by the skill or installed arm in fewer than two thirds, or has too many errored runs to tell. Other cases are reported only.Model calls go through the Redpanda AI Gateway with a token minted per job from GitHub OIDC; there is no API key fallback. All actions are pinned by commit SHA, checkouts do not persist credentials, and only the job that runs the evals can request an OIDC token.
evals/scripts/ci.mjsholds the selection and the gating rule, with tests inevals/scripts/ci.test.mjs(node --test evals/scripts/ci.test.mjs).Needs, before the evals can run: this repository's AWS role and its two secrets, and the gateway agent. Until then the eval jobs fail at authentication.
Draft until the credentials exist.