Claude/helmsdeep performance runner y0q51y

README.md

@@ -41,6 +41,35 @@ saved to `--output_dir` automatically. Likewise, if the Information Radiator
 isn't configured (no `ZE_BASE_URL` / `ZE_REFRESH_TOKEN`), the harness falls
 back to a local reporter.
 
+### Running local test files
+`test-harness download <suite>` fetches the suites from
+[NCATSTranslator/Tests](https://github.com/NCATSTranslator/Tests) every run,
+which is no good when you want to change a CURIE and see what happens. A few
+small, editable suites ship in `test_harness/test_suites/`; run one with `load`
+instead of `download`:
+- `test-harness --local load local_acceptance`
+
+| Suite | What it is |
+| --- | --- |
+| `local_acceptance` | Two acceptance cases — one MVP1 (drug treats disease), one MVP2 (chemical affects gene) — so `--query_type` has something to filter. |
+| `local_performance` | One `QuantitativeTest` case, run by HelmsDeep. |
+| `local_pathfinder` | One Pathfinder case: two pinned endpoints, expecting a named node on a path between them. |
+
+The suite name is the file's name without `.json`, the same way the downloaded
+suites are named, so the same name works with either subcommand.
+
+The harness reads `test_suites/` in the working directory when that exists —
+so a suite you're editing in your checkout is found without a flag — and
+otherwise the copy shipped with the package. `--tests_dir` overrides both:
+- `test-harness --local load my_suite --tests_dir ~/scratch/suites`
+
+These are ordinary Translator Testing Model `TestSuite` documents, identical in
+shape to the downloaded ones. Copy one, change the CURIEs, run it. See
+[`test_harness/test_suites/README.md`](test_harness/test_suites/README.md) for
+which fields actually drive behavior — in particular, a performance case's
+`test_run_time`/`spawn_rate` and its asset's query are inert, because HelmsDeep
+owns the ramp and the corpus (see [Performance tests](#performance-tests)).
+
 ### Running one type of query
 A suite normally mixes the MVP query types. To run just one of them — handy
 for a local evaluation run of a change that only affects one — pass
@@ -79,6 +108,133 @@ then contributes four columns:
 These are the same numbers as the `actual_output` in the per-test report JSON
 uploaded to the Information Radiator.
 
+### Performance tests
+Performance tests (`QuantitativeTest` cases) are run by
+[**HelmsDeep**](https://github.com/TranslatorSRI/HelmsDeep), installed from
+`requirements-runners.txt`. The harness does not measure anything itself: it
+picks the HelmsDeep run type for the test case, runs its CLI, and reports what
+came back.
+
+HelmsDeep owns two things the harness used to own, and this is deliberate:
+
+- **The ramp.** HelmsDeep's `LoadTestShape` drives users, spawn rate, and
+  duration from its own per-layer `stages` table, so a test case's
+  `test_run_time` and `spawn_rate` are ignored. A run takes as long as the run
+  type's ramp says (tens of minutes to over an hour). HelmsDeep can compress a
+  run to a wall-clock budget, but its own docs are explicit that a compressed
+  run is indicative rather than a measurement, so the harness never asks for it.
+- **The queries.** HelmsDeep sends its own corpus and varies the pinned entity
+  per request, so the numbers cover a real cost surface instead of warming one
+  cache. The test asset's TRAPI query is not what goes under load; the
+  Information Radiator log for the test records the *run plan* (layer, ramp,
+  SLO, checkpoints) instead, which is what you need to read the result.
+
+#### Which run type a test gets
+The layer follows the component: `ars` uses HelmsDeep's async submit/poll/merge
+targets, and every other component is queried as an ARA (a blocking
+`POST /query`). The stack cascades ARS → ARAs → KPs, so exactly one layer is
+loaded per run.
+
+The query profile within that layer defaults to `default` — the layer's own
+single-class corpus, which answers the open question *how far can we go?* and
+reports a **knee** (max sustainable concurrency). Two heavier profiles are
+available:
+
+- `mixed` — the 2:1 inferred/Pathfinder acceptance profile. This is the only
+  profile that carries **pass/fail checkpoints** (does 30 concurrent hold? 45?
+  does 60 avoid substantial failures?).
+- `pathfinder` — the two-pinned-endpoint path queries, the heaviest class.
+
+Select one with `--performance_profile`, or per test case with a `mixed` /
+`pathfinder` entry in its `test_runner_settings`; the flag wins over the test
+case.
+- `test-harness --local --performance_profile mixed download <suite>`
+
+#### What gets reported
+Two artifacts per run are uploaded to Slack (or saved to `--output_dir`):
+- `<host>_<run_type>_summary.json` — HelmsDeep's `summary.json` verbatim. This
+  is the authoritative result: the knee, every stage's numbers, the checkpoint
+  verdicts, measurement-quality warnings, and (for ARS runs) health and red
+  flags.
+- `<host>_<run_type>_report.html` — locust's own HTML report: charts and
+  request/failure tables.
+
+Both are uploaded with a comment carrying the run's checkpoint verdict, and the
+Slack summary message leads with the overall pass/fail. A run with no
+checkpoints reports its knee and says so explicitly rather than claiming a pass
+it never tested for. A missed checkpoint, or a HelmsDeep run that never produced
+a summary at all, is reported as `FAILED` in the Information Radiator too, so
+the dashboard agrees with Slack.
+
+The rest of HelmsDeep's output (`_stages.csv`, `_by_qtype.csv`,
+`_checkpoints.csv`, and the ARS debug logs) is left on disk under
+`--output_dir` for digging in.
+
+### Sweeping several services in one scheduled job
+HelmsDeep loads exactly one layer per run, and the harness resolves one target
+per invocation, so measuring the ARS and each ARA means several runs. They must
+not overlap: the stack cascades ARS → ARAs → KPs, so loading two layers at once
+double-loads whatever sits underneath and corrupts both measurements.
+
+`test-harness-sweep` runs them sequentially in one process tree, which is what
+lets a single Kubernetes Job (or `docker run`, or a local shell) do the whole
+sweep:
+
+```bash
+test-harness-sweep --suite performance_tests --download \
+    ars=https://ars.ci.transltr.io \
+    aragorn=https://aragorn.ci.transltr.io \
+    arax=https://arax.ci.transltr.io \
+    bte=https://bte.ci.transltr.io \
+    -- --performance_profile mixed
+```
+
+Each `NAME=URL` pair becomes one `test-harness --target_url URL --target NAME`
+invocation, in the order given, and the sweep waits for each to finish before
+starting the next. Anything after `--` is passed through to every invocation.
+Targets can come from the `PERFORMANCE_TARGETS` environment variable instead, as
+a comma-separated list of the same pairs — usually the easier knob in a CronJob,
+since changing the sweep is then a manifest edit rather than an image rebuild.
+
+Two behaviors worth knowing:
+- **Every target runs even if an earlier one fails.** A broken ARS shouldn't
+  cost you the ARA numbers. The sweep exits non-zero if any target failed, and
+  prints a per-target summary with exit codes and durations.
+- **Each target gets its own `--output_dir` subdirectory.** HelmsDeep names its
+  raw files after the run type and test case id, neither of which mentions the
+  host, so every ARA in a sweep writes the same `helmsdeep_aras_case_<id>_*`
+  names and would otherwise overwrite the previous target's files.
+
+#### Exit status
+The harness's exit status answers **"did the run happen?"**, never "did the
+tests pass?":
+
+- **Exit 0** — the run happened, whatever it found. Failed acceptance tests, a
+  missed performance checkpoint, a service that was down and produced no
+  summary: those are *results*. They go to Slack and the Information Radiator,
+  and a scheduled job stays green, because a slow ARA is news for the channel
+  rather than a broken cron.
+- **Exit 1** — the harness couldn't carry out the run: `--target_url` without
+  `--target`, a suite that isn't there, a suite that `--query_type` filtered
+  down to nothing, or a crash. That job produced no results at all, which is
+  exactly the failure that otherwise sits green and silent for months.
+
+The sweep inherits this: it exits non-zero only if a harness invocation couldn't
+carry out its run, so a red Job always means something operational.
+
+[`deploy/cronjob.example.yaml`](deploy/cronjob.example.yaml) is a worked CronJob
+for this, with the fields a multi-hour job actually needs — `concurrencyPolicy:
+Forbid` so the next schedule can't start a second overlapping sweep,
+`backoffLimit: 0` so a failure doesn't re-run hours of load, an
+`activeDeadlineSeconds` ceiling, `PYTHONUNBUFFERED` so `kubectl logs` shows
+progress, and CPU requests that keep the load generator from measuring its own
+scheduling delay.
+
+Budget the time before you schedule it: with `--performance_profile mixed` a
+four-service sweep is roughly 70 min for the ARS plus 61 min per ARA (~4.2
+hours); the single-class default profile is ~58 + ~35 × 3 (~2.7 hours). Nothing
+overlaps, by design.
+
 ### Overriding the target service
 Tests specify which component to run against (`ars`, `ara`, ...), and the
 harness normally resolves those components to deployed services through the

deploy/cronjob.example.yaml

@@ -0,0 +1,124 @@
+# Example CronJob for a sequential performance sweep.
+#
+# One Job, four harness runs, strictly in order: the ARS first, then each ARA.
+# They must not overlap -- the Translator stack cascades ARS -> ARAs -> KPs, so
+# loading two layers at once double-loads whatever sits underneath and corrupts
+# both measurements. `test-harness-sweep` runs them one at a time and waits.
+#
+# This is an example to adapt, not a manifest to apply as-is: the image tag,
+# namespace, secret names, and hosts are all placeholders.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: test-harness-performance-sweep
+spec:
+  # Weekly, Sunday 02:00. A full sweep is hours long (see activeDeadlineSeconds
+  # below), so this is not a thing to run hourly.
+  schedule: "0 2 * * 0"
+  timeZone: "America/New_York"        # requires Kubernetes >= 1.27
+
+  # The single most important field here. A sweep runs for hours; without
+  # Forbid, the next schedule starts a second sweep while the first is still
+  # loading the same services, which is exactly the concurrent load the
+  # layering rule forbids. Both runs' numbers would be garbage.
+  concurrencyPolicy: Forbid
+
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+
+  jobTemplate:
+    spec:
+      # Never retry. A retry re-runs the whole multi-hour sweep and re-posts
+      # every Slack message; if a service is genuinely down, the next scheduled
+      # run is soon enough.
+      #
+      # Note this Job only ever fails for operational reasons. Test results --
+      # a missed checkpoint, a service that never answered -- exit 0 and are
+      # reported in Slack, so a red Job here means the sweep itself could not
+      # run (bad config, a missing suite, a crash, an OOM kill).
+      backoffLimit: 0
+
+      # Hard ceiling, comfortably above the expected duration. With the `mixed`
+      # profile a sweep is roughly 70 min (ARS) + 61 min per ARA ~= 4.2 h; the
+      # single-class default profile is ~2.7 h. 6 h leaves room without letting
+      # a wedged run hold the schedule indefinitely.
+      activeDeadlineSeconds: 21600
+
+      template:
+        spec:
+          restartPolicy: Never
+
+          # HelmsDeep drains in-flight queries at the end of a run (the ARS
+          # completion sidecar keeps polling past the failure threshold). Give
+          # it room to finish rather than killing it mid-drain.
+          terminationGracePeriodSeconds: 120
+
+          containers:
+            - name: test-harness
+              image: ghcr.io/translatorsri/testharness:latest
+              imagePullPolicy: IfNotPresent
+
+              command: ["test-harness-sweep"]
+              args:
+                - "--suite"
+                - "performance_tests"
+                - "--download"
+                # Everything after `--` is passed to each harness invocation.
+                - "--"
+                - "--performance_profile"
+                - "mixed"
+                - "--log_level"
+                - "INFO"
+
+              env:
+                # The sweep, in order. Keeping the list here means changing
+                # which services get swept is a manifest edit, not an image
+                # rebuild. ARS first: it loads the ARAs as a side effect, so
+                # measuring it before them keeps each ARA's own run clean.
+                - name: PERFORMANCE_TARGETS
+                  value: >-
+                    ars=https://ars.ci.transltr.io,
+                    aragorn=https://aragorn.ci.transltr.io,
+                    arax=https://arax.ci.transltr.io,
+                    bte=https://bte.ci.transltr.io
+
+                # Without this, Python buffers stdout when it isn't a terminal
+                # and `kubectl logs` shows nothing for hours.
+                - name: PYTHONUNBUFFERED
+                  value: "1"
+
+                - name: SLACK_WEBHOOK_URL
+                  valueFrom:
+                    secretKeyRef: {name: test-harness-secrets, key: slack-webhook-url}
+                - name: SLACK_TOKEN
+                  valueFrom:
+                    secretKeyRef: {name: test-harness-secrets, key: slack-token}
+                - name: SLACK_CHANNEL
+                  valueFrom:
+                    secretKeyRef: {name: test-harness-secrets, key: slack-channel}
+                - name: ZE_BASE_URL
+                  valueFrom:
+                    secretKeyRef: {name: test-harness-secrets, key: ze-base-url}
+                - name: ZE_REFRESH_TOKEN
+                  valueFrom:
+                    secretKeyRef: {name: test-harness-secrets, key: ze-refresh-token}
+
+              resources:
+                # The load generator is the workload here: locust drives up to
+                # 60 concurrent users in the mixed profile, all in one process,
+                # and a CPU-starved generator measures its own scheduling delay
+                # instead of the service's latency. Give it real CPU, and make
+                # requests == limits so throttling can't silently skew a run.
+                requests: {cpu: "2", memory: "2Gi"}
+                limits: {cpu: "2", memory: "4Gi"}
+
+              volumeMounts:
+                - name: results
+                  mountPath: /app/test_results
+
+          volumes:
+            # Results land here per target; the reportable artifacts are
+            # uploaded to Slack by each run, so this only needs to outlive the
+            # pod if you want the raw CSVs. Swap for a PVC if you do.
+            - name: results
+              emptyDir: {}

requirements-runners.txt

@@ -2,4 +2,10 @@
 # benchmarks-runner==0.1.3
 # ui-test-runner==0.0.2
 # graph-validation-test-runners==0.1.5
+# HelmsDeep is the performance runner: it owns the step-load ramps, the TRAPI
+# corpuses, the ARS submit/poll/merge protocol, and the knee/checkpoint
+# arithmetic. test_harness/performance_test_runner.py only drives its CLI.
+helmsdeep @ git+https://github.com/TranslatorSRI/HelmsDeep@main
+# HelmsDeep brings locust in; pinned here so the harness's image is
+# reproducible rather than tracking whatever HelmsDeep's floor resolves to.
 locust==2.38.1

requirements.txt

@@ -1,5 +1,4 @@
 httpx==0.27.0
-matplotlib==3.9.2
 pydantic==2.7.1
 # reasoner_pydantic==4.1.6
 setproctitle==1.3.3

setup.py

@@ -16,12 +16,19 @@
     long_description=readme,
     packages=["test_harness"],
     include_package_data=True,
+    # The editable local test suites ship with the package so `test-harness
+    # load <suite>` works from a plain (non-editable) install too, not just
+    # from a checkout.
+    package_data={"test_harness": ["test_suites/*.json", "test_suites/README.md"]},
     zip_safe=False,
     license="MIT",
     python_requires=">=3.9",
     entry_points={
         "console_scripts": [
             "test-harness = test_harness.main:cli",
+            # Runs one suite against several services in turn, so a single
+            # scheduled Job can sweep the ARS and each ARA sequentially.
+            "test-harness-sweep = test_harness.sweep:cli",
         ],
     },
 )