Repository navigation
New and improved testing dashboard - #55
Merged
Merged
Conversation
The harness now writes TRAPI 2.0 queries by default, built from
translator_tom (TOM) models so each query is validated against the spec
before it is sent. --trapi_version 1.6.x still works for services that
have not moved: those queries are built from TOM's v1_6 models, and the
SmartAPI registry is filtered to the same major.minor as before.
Unsupported versions are refused at the CLI and by QueryRunner.
Queries
- MVP2 qualifiers go in constraints.qualifiers as one
{qualifier_type_id: qualifier_value} mapping instead of 1.x's
qualifier_constraints, which 2.0 servers such as Gandalf reject.
- Unset qualifiers are left out rather than sent as "" (2.0 forbids an
empty mapping); with none set, no constraints are sent.
- Pathfinder queries are a plain QueryGraph with paths and no edges.
- An absent Pathfinder category is omitted rather than sent as [null].
Responses
2.0 binds each QNode / QPath to one object with an ids list; 1.x to a
list of objects each with an id. test_harness/trapi.py's binding_ids
reads both, since the ARS can return either. Before this, every 2.0
response raised in the acceptance analysis and the agent was FAILED.
The expected-answer lookup now checks membership in a binding's ids
rather than only its last entry.
Dependencies
translator_tom needs pydantic>=2.12 and Python 3.10, so the pydantic pin
is lifted and python_requires rises to 3.10 (CI and images are on 3.11).
reasoner-validator was unused and conflicted with the new pydantic, so
it is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LyWodasv7FcfGMu7dGgvuR
First phase of replacing the Zebrunner-based Information Radiator. The harness now dual-writes: Zebrunner is untouched, and a new client posts structured results to the replacement radiator in batches. - radiator_schema: the API contract shared by harness and radiator. Writes are upserts on client-chosen ids, so retries and replays are safe; results key on Tests-repo ids, which are stable across runs. - Per agent, the harness now records n_results, HTTP status, and response time (direct queries only; ARS fan-out is only seen through 10s polling, so timing it would mostly measure the polling). - An unreachable radiator never fails a run. Runs that aren't uploaded are saved to --output_dir and can be replayed with the new `test-harness-radiator push`. - docs/information-radiator.md records the agreed design and phases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
The replacement for the Zebrunner-based Information Radiator, as its own package and (later) image in this repo. - FastAPI + Postgres (SQLAlchemy, Alembic). Ingest endpoints take the radiator_schema contract with a bearer token; writes are upserts on the contract's ids, so retries and replays never duplicate. - One shared consortium login (signed session cookie, throttled), and every API route requires the token or a login. - Pages: runs list; a run's results grid (asset x agent, with rank) and per-agent summary; what changed since the previous run in the same series; an asset's status/rank/result-count history; agent pass-rate trends by expected output; performance trends. Charts are server-side SVG with a small hover layer, each paired with a table. - `python -m radiator.demo` seeds made-up history for development. - Tests run against real Postgres, including the harness's own client posting to the app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
`python -m radiator.zebrunner_import` reads Zebrunner's own read API (launches, their tests with labels, and per-test logs) and posts each run to the radiator's ingest API. Statuses come from the per-agent labels the harness set; found, rank, score and pks from the TestReport the harness logged (--no-logs skips those for speed). Zebrunner never had result counts, HTTP status, response times or the environment, so those stay unknown; --env sets the environment, otherwise runs with prod-only agents are marked prod. Run ids derive from the launch id, so a re-run skips (or with --force updates) what's already there. Performance tests are skipped: they predate HelmsDeep and have no summary. All pk links now open the ARAX UI on ci, as the harness always did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
- radiator/Dockerfile (same base image as the harness, non-root). - deploy/: the radiator (Deployment with a migration init container, Service, Ingress), an optional in-namespace Postgres, and a one-off Job for the Zebrunner import. - CI runs the radiator's tests against a Postgres service and checks the migrations match the models; releases push a second image, ghcr.io/translatorsri/testharness-radiator. - httpx is a runtime dependency (the import uses it). - Pages no longer overflow at phone width, and the wide trend chart has a narrow twin so its labels stay legible there. - docs/information-radiator.md covers deploying, importing, and developing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
…e env A next=/\host link passed the on-site check, and browsers treat it like //host. An asset link with env= empty mixed environments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Harness: - Zebrunner or Slack failing no longer costs the radiator a run: a failed Zebrunner run open falls back to the local reporter, a failed create_test still collects the asset, and the radiator run is closed and saved before Zebrunner and Slack are finished with. A run that crashes is saved but left open. - A non-numeric status (the ARS's, after a failed poll) no longer invalidates an asset's radiator record. - A performance run that raises is recorded in the radiator as failed. Radiator: - The login throttle keys on the X-Forwarded-For entry the ingress adds, so forging the header can't dodge it, and its memory is bounded. - The login redirect keeps the whole URL, query string included. - A crashed or still-uploading run is never a diff baseline or a trend point. - The asset page defaults to the environment with the most history, and runs with no recorded environment have their own history. - The import re-imports runs that stopped partway, and checks for existing runs without loading their results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
When the radiator has the run, the harness's final Slack message gains a
headline against the previous run of the same series (pass rate and its
change, regressions, fixes, a link to what changed), and a chart of
each agent's pass rate over the series' last 30 runs is posted under it.
The radiator draws the chart as a PNG (GET /api/runs/{id}/history.png,
matplotlib in the dashboard's palette) and serves the numbers at
/api/runs/{id}/summary; the harness fetches both with its token, since
Slack can't render SVG or reach pages behind the login. Best effort:
without the radiator the report is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Performance runs now get the same treatment as acceptance runs in the
Slack report: a headline per service (max sustainable concurrency and
its change since the service's previous run, and the checkpoint
verdict) and a chart of that concurrency over the service's last 30
runs, with missed checkpoints marked (GET /api/runs/{id}/performance.png).
A service's series is one host under one HelmsDeep run type and
profile, in one environment; the dashboard's performance page now
groups the same way instead of mixing profiles.
Trend charts, in Slack and on the dashboard, are step lines: a run's
value holds until the next run, so rises and drops read plainly. A
single-series chart drops its legend and gives the value in the
subtitle. The demo data now sweeps one service per run, like
test-harness-sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
`docker compose up --build` now starts Postgres, migrates it, starts the radiator on localhost:8000, and runs a small suite (local_acceptance, or HARNESS_ARGS) that reports to it, and only to it: the harness gets no Zebrunner settings. Slack is opt-in via .env. The README has a section on trying it, reporting to it, and deploying it. - RADIATOR_PUBLIC_URL: where people open the radiator, for links in logs and Slack, when the harness uploads elsewhere (the compose service name, or an in-cluster Service). - The Slack report only links to Zebrunner when the run went there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
# Conflicts: # setup.py
…aude/radiator-phase1 # Conflicts: # test_harness/main.py
Results were held until 50 had built up or the run ended. Each test case waits minutes on its queries, so a real run can be fewer results than one batch and still take an hour, and the radiator showed nothing for it until the very end. The collector now flushes after every test case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
--query_parameters (or QUERY_PARAMETERS) takes a JSON object, or @file, of parameters to send with every query of a run: eg '{"timeout": 300, "bypass_cache": true}'. TRAPI 2.0 queries carry them in Query.parameters, where services may also define their own; TRAPI 1.6 queries take log_level and bypass_cache as top-level fields, and anything else is refused. They're checked against the TRAPI version before the run starts, so a typo stops the run instead of failing every query. The run's parameters go in its Slack message and are recorded in the Information Radiator (a new runs.query_parameters column), shown on the run page and as a badge in the runs list. Docker Compose passes QUERY_PARAMETERS through from .env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
A run of 600+ assets made the run page one very long table. The grid now shows 50 assets a page by default (100, 250, or all on request), with "Showing 51-100 of 612" and first/previous/next/last links above and below it. Pages keep the current filters; changing a filter starts again at page 1 at the same page size, and a page past the end shows the last one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
The runs list already paged, 30 at a time with only newer/older links. It now uses the results grid's pager: "Showing 26-50 of 75 runs", page N of M with first/previous/next/last links above and below the table, and 25, 50, or 100 runs a page. Only that page's runs are read from the database. Changing the suite or environment keeps the page size and starts again at page 1. Both lists share one paging helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
A new section under "By agent" splits each agent's counts by the assets' expected output (TopAnswer, Acceptable, BadButForgivable, NeverShow): a results bar, pass rate, and passed/failed/no results/ error/skipped counts per row, grouped by agent. Each count opens the results grid filtered to that agent, status, and expected output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Every full acceptance run of a suite, one lane per environment, colored by whether its pass rate went up, down, or held against the previous run in that environment. Excluded environments (RADIATOR_EXCLUDED_ENVS, default dev) are greyed out. Runs get radiator numbers (migration 0003), numbered by start time. The dashboard shows it under Run grid; the radiator draws it as grid.png and the harness posts it with each full acceptance run's Slack report, with the run marked NEW. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
test-harness-cycle runs acceptance, then pathfinder, then performance (the ARS) in one environment, one after another, and marks the runs with a shared cycle id. When it finishes it posts the radiator's board for the cycle: each test against its previous run in that environment, a verdict, the regressions and fixes by agent, and flags for oddities a pass rate hides (results gone, new errors, slower responses, big rank drops while passing, concurrency drops). test-harness-radiator digest posts the same for every environment, weekly. The radiator gains runs.cycle_id (migration 0004), the cycle and digest API and PNGs, and Weekly and run cycle pages. The per-run Slack report no longer posts the run grid; the grid stays on the dashboard. Example CronJobs for a Monday-to-Thursday rotation and the Monday digest are in deploy/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
deploy/helm/radiator deploys the Information Radiator (migrations in an init container, Service, optional Ingress, optional Postgres, optional one-off Zebrunner import). deploy/helm/test-harness deploys a weekly run cycle CronJob per environment, the Monday digest, and an optional performance sweep. Secrets come from an existing Secret or are created from values. Each chart ships a values.example.yaml to copy; the copies (values.<name>.yaml) are git-ignored. CI lints and renders both charts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
A running run's page now fills in by itself as the harness uploads each test
case's results, with a progress bar of assets tested out of the run's total
(the harness now sends expected_assets; migration 0005). The runs list and
the Weekly page pick up runs starting, uploading and finishing.
Pages poll a small JSON check (/api/runs/{id}/progress, /api/live) and, when
it changes, refetch the page and swap in only the marked regions, so filters
and pagination stay put. Polling pauses while the tab is hidden and backs off
on errors. Uploads now touch the run's updated_at; an open run with no upload
for 2 hours is shown as "didn't finish" instead of running. While a run is
going, its numbers say "so far" and What changed doesn't count untested
assets as dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
…ebrunner With RADIATOR_SLACK_WEBHOOK_URL and RADIATOR_SLACK_CHANNEL set (the token defaults to SLACK_TOKEN), each run posts twice: SLACK_CHANNEL gets the report it got before the new radiator (the Zebrunner link, summary and result files), and the new channel gets the radiator's report (its link, headline, history charts and result files), plus the run cycle boards and the weekly digest. A failure posting to the new channel is logged and never costs the old one its report. Without these settings nothing changes. The Helm chart, compose, .env.example and the plain manifests pass the new settings through; the README explains the setup and the cutover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Without RADIATOR_SLACK_*, a run reporting to Zebrunner still posted the new radiator's link, headline and charts to SLACK_CHANNEL. Now, while ZE_* is set, SLACK_CHANNEL gets only the report it always did, and the new radiator's report, cycle board and digest go only to its own channel (RADIATOR_SLACK_*), or aren't posted without one. Without Zebrunner, SLACK_CHANNEL is the radiator's, as before. This keys off Zebrunner being configured, not reachable, so an outage never switches the old channel over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
ZE_BASE_URL and ZE_REFRESH_TOKEN in .env now reach the harness, so a compose run can report to Zebrunner alongside the new radiator (and SLACK_CHANNEL then gets Zebrunner's report). Unset, it reports to the new radiator only, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Harness: - Says at the start where the run's results go (or why they're only saved), and at the end how many of the suite's assets the radiator has, naming any without a result so they can be searched for in the log. - A failed upload logs the radiator's answer (eg which field a 422 rejected) or the connection error, with the URL, run and what was in the batch. - Every place an asset can be left out names its test case and asset and says why: no assets or objective, an untested expected output, a query that couldn't be built, no answer or an error status from a service. - "still Running" polling moves to debug. Radiator: - Logs each run opened, each batch received and each run finished, and the fields of a rejected upload. RADIATOR_LOG_LEVEL (chart: config.logLevel). - Alembic's logging setup no longer disables the app's loggers when migrations run in the same process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Zebrunner's /api/reporting/v1/launches has no list endpoint (only
/v1/launches/{id} and below), so the import's first request was a 404. List
launches through reporting-service's LegacyLaunchController,
/api/reporting/api/tests/runs, which takes the same page and pageSize.
Zebrunner errors now say what Zebrunner answered (and suggest --project-id
on a 400 without one), and the CLI reports a failed login or listing in one
line instead of a traceback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
The previous fix was wrong: /api/reporting/api/tests/runs looks up one run by its CI run id. Reading the controllers' mappings out of the published images (reporting-service 1.54.1, test-execution-logs-service 1.6.1): - GET /api/reporting/v1/launches is the list after all (LaunchController's bare @GetMapping); the 404 was Zebrunner wanting a projectId, so --project-id is now required. - A test's logs, paged, are .../tests/{testId}/logs-and-screenshots (pageToken, maxPageSize; _meta.nextPageToken). .../logs is a file download. The Helm chart's import Job takes zebrunnerImport.projectId, and the docs and the example Job pass --project-id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Zebrunner 1.28 (the consortium's) has no /v1 read API. Its UI lists runs with GET /api/reporting/api/tests/runs/search and a run's tests with POST /api/reporting/api/tests/search, both paged in `results`. The import now speaks both: --api v1|legacy|auto, auto by default (try v1, fall back on a 404). Legacy runs are mapped to the fields the import reads (the suite name from testSuite, epoch-milli times, the end from elapsed), and the fields of the first legacy run and test are logged so a dry run shows any mismatch. A Zebrunner that doesn't serve test logs where the import looks now gets one warning and an import without ranks/scores/pks, instead of every launch failing. --project-id is optional again (only the v1 API needs it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Zebrunner 1.28 serves a test's logs at
GET /api/reporting/v1/test-runs/{id}/tests/{testId}/logs?page=&pageSize=,
paged by number in `results`, and the harness's TestReport is one entry's
message, as on newer versions. Legacy imports now read them there, so ranks,
scores and pks come across. The entries have no ids, so they page by
totalPages rather than through the id-based paging the runs use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Zebrunner never recorded the environment. A pk only exists on the ARS that made it, and the harness asked the ARS to retain every response, so without --env the import now asks each environment's ARS (from the SmartAPI registry, as the harness finds them; --ars ENV=URL overrides) for a few of a run's pks, parent pks first, and records the run under the one that has one. Runs no ARS knows fall back as before (prod-only agents mean prod, else no environment). Each run's log line says how its environment was decided, the summary counts environments, and an unreachable ARS gets one warning. --no-env-lookup turns it off. The Helm import Job no longer forces an environment by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
--trapi_version auto (now the CLI default) uses the version the run's environment's ARS (or the --target service) is registered with in SmartAPI, the newest the harness speaks if there are several, and says what it chose; with nothing registered or SmartAPI unreachable it falls back to 2.0 with a warning. Explicit versions work as before. The query-parameter check now runs after the version is known. Each run's TRAPI version is recorded in the radiator (migration 0006), shown on the run, in the runs list and in the Slack report's start, and a change of version since the previous run is called out on the run page, What changed, the cycle board's flags (so the cycle message and digest) and the Slack headline, so an upgrade's effect isn't read as a regression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
- Remove the unused run-grid PNG (route, renderer, NEW badge, window fitting) and other dead helpers (run_exists, VERDICT_ORDER, RADIATOR_SLACK_VARS, pk_url's env). - test-harness-cycle posts its board via the radiator the steps reported to, and a Slack failure there is only a warning. - test-harness-radiator takes --radiator_url/--radiator_token after the subcommand; TRAPI auto falls back when SmartAPI answers non-JSON. - Add .dockerignore so .env and values copies never reach an image. - Docs: TRAPI version in the radiator, module map, Zebrunner versions, Slack routing wording, Helm as the maintained deploy path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WR6TD7jzBVdYPod4pKWcg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.