Skip to content

fix: draw each drift threshold step at its own slice's threshold - #14

Open
JingHao-Leon wants to merge 2 commits into
rlaope:mainfrom
JingHao-Leon:fix/drift-threshold-staircase
Open

JingHao-Leon wants to merge 2 commits into
rlaope:mainfrom
JingHao-Leon:fix/drift-threshold-staircase

Conversation

@JingHao-Leon

@JingHao-Leon JingHao-Leon commented Oct 19, 2026 •

Copy link
Copy Markdown

What broke

jeval.report.charts.drift.render_threshold_steps — the "Recommended threshold over time" staircase — drew each horizontal step at the next slice's threshold, so the line did not read the threshold of the slice it was drawn over and the segments never met.

For slices with thresholds 0.50 / 0.60 / 0.70 the chart emitted:

segment from to
horizontal (x0, 0.60) (x1, 0.60)
connector (x1, 0.50) (x1, 0.60)
horizontal (x1, 0.70) (x2, 0.70)
connector (x2, 0.60) (x2, 0.70)

The step over the first slice sat at 0.60 although that slice was measured at 0.50, and the connector at the first boundary ended at 0.60 while the step leaving it was at 0.70 — two horizontal lines with two connectors hanging below them, instead of one staircase.

Root cause

jeval/report/charts/drift.py:222-245. The connector at x(index + 1) was drawn from left_value to right_value — a step-after join, the value changes at the right edge of a slice — but the horizontal segment over the same span was drawn at right_value, the step-before height. The two halves of one step disagreed about which side of the boundary the value belonged to, so the horizontal segment was one slice ahead of the slice it covers.

The change makes the horizontal segment sit at left_value, the slice's own threshold, which is the convention the connector already used. The docstring now states that invariant.

How it was verified

The regression test reads the emitted SVG back — the y-axis labels carry the scale, and two of them are enough to turn a pixel height back into a threshold — and asserts that the step over a slice is at that slice's own threshold and that each connector spans the two thresholds it joins.

Before the change, on the pristine source:

$ uv run pytest tests/test_qa_hardening.py::test_the_drift_threshold_staircase_sits_at_each_slice_own_threshold
E       AssertionError: the step over a slice must sit at that slice's own threshold, got [0.6, 0.7]
E       assert [0.6, 0.7] == approx([0.5 ±... 0.6 ± 0.001]
E
E         comparison failed. Mismatched elements: 2 / 2:
E         Max absolute difference: 0.09999999999999998
tests/test_qa_hardening.py:688: AssertionError
1 failed

After the change:

$ uv run pytest tests/test_qa_hardening.py::test_the_drift_threshold_staircase_sits_at_each_slice_own_threshold
1 passed

Full suite, the gate CONTRIBUTING.md asks for:

$ uv run pytest
659 passed in 48.65s     # before this change
660 passed in 41.95s     # after this change (one new regression test)

$ uv run ruff format --check .
140 files already formatted

$ uv run ruff check .
All checks passed!

$ uv run mypy jeval
Success: no issues found in 32 source files

$ uv run jeval demo --out-dir /tmp/jeval-demo-after
report.html written:   391671 bytes

CI status

  • DCO — ran and passed (gh pr checks reports one check, DCO, SUCCESS).
  • CI — blocked and has not run. The workflow run for this head SHA is
    status=completed, conclusion=action_required with 0 jobs: the repository's
    "Approve and run workflows" gate is holding it, so the lint-and-type-check job and the
    pytest matrix on Python 3.10 / 3.11 / 3.12 have not executed. A maintainer has to approve
    the workflow for them to run.

Because of that gate, the numbers above are all from a local run of the same gate
CONTRIBUTING.md asks for, not from CI.

Notes

  • The HTML drift section renders this chart from the compared slices, so it appears whenever those slices carry a recommended threshold. jeval report currently builds its drift view without cost-derived thresholds, so the chart is not emitted by that command today; the defect is in the chart builder itself.
  • The two commits are the reproducer and the fix, so the failing test and the change that makes it pass are separate.

Fixes #13

This change was prepared with the assistance of an AI coding agent.

The horizontal step over a slice was drawn at the next slice's threshold,
so the line read a threshold the slice underneath it never had and the
segments did not join up.

Signed-off-by: JingHao-Leon <102573344+JingHao-Leon@users.noreply.github.com>
The horizontal step over a slice was drawn at the next slice's threshold
while the dashed connector at the same x joined the two, so the staircase
never met: the line read a threshold the slice underneath it never had, and
the first slice's own threshold was drawn nowhere.

Signed-off-by: JingHao-Leon <102573344+JingHao-Leon@users.noreply.github.com>
@JingHao-Leon
JingHao-Leon force-pushed the fix/drift-threshold-staircase branch from b359145 to 217d3e6 Compare October 19, 2026 21:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The drift threshold staircase is drawn one slice ahead: each step sits at the next slice's threshold

1 participant