Repository navigation
Conversation
The tests/test_examples.py harness (5s timeout + XFAIL for slow examples, gated behind --run-examples) already exists, but the CI pytest steps ran without --run-examples, so example files were never imported in CI. Add --run-examples to the Linux and Windows steps so the example suite runs. Fixes PriorLabs#51
|
|
There was a problem hiding this comment.
Code Review
This pull request increases the timeout for slow example tests from 1 to 5 seconds and moves the imports of threading and conftest to the top level of tests/test_examples.py. The feedback points out that importing conftest at the top level is a pytest anti-pattern that can lead to import errors, and suggests reverting to importing HAS_TABPFN and TABPFN_SOURCE locally within the test function.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| import pytest | ||
|
|
||
| import conftest |
There was a problem hiding this comment.
Importing conftest at the top level of a test module is a pytest anti-pattern. conftest.py is loaded automatically by pytest, and importing it directly can lead to ModuleNotFoundError during test collection (since the tests/ directory is typically not in sys.path at import time) or double-import issues. It is safer to import the required variables locally within the test function where conftest is already guaranteed to be loaded.
| """ | ||
| from conftest import HAS_TABPFN, TABPFN_SOURCE | ||
|
|
||
| file_name = example_file["name"] |
There was a problem hiding this comment.
| if not conftest.HAS_TABPFN: | ||
| pytest.skip( | ||
| f"Example {file_name} requires TabPFN package, but it's not installed", | ||
| ) | ||
| elif TABPFN_SOURCE == "tabpfn_client": | ||
| elif conftest.TABPFN_SOURCE == "tabpfn_client": |
There was a problem hiding this comment.
Revert the usage of conftest.HAS_TABPFN and conftest.TABPFN_SOURCE back to the locally imported HAS_TABPFN and TABPFN_SOURCE variables.
| if not conftest.HAS_TABPFN: | |
| pytest.skip( | |
| f"Example {file_name} requires TabPFN package, but it's not installed", | |
| ) | |
| elif TABPFN_SOURCE == "tabpfn_client": | |
| elif conftest.TABPFN_SOURCE == "tabpfn_client": | |
| if not HAS_TABPFN: | |
| pytest.skip( | |
| f"Example {file_name} requires TabPFN package, but it's not installed", | |
| ) | |
| elif TABPFN_SOURCE == "tabpfn_client": |
|
Thanks for kicking this off, @mvanhorn 🙏 — closing in favour of #327, which carries it forward. #327 keeps the same goal (run the example files in CI) but swaps the bounded-timeout/XFAIL harness for one that fails on real errors, and fixes the example breakages it surfaced instead of XFAILing past them. Your work here is credited as co-author on #327. |
…327) test: run example files as CI tests (PRI-328) (#327) Body: Replace the old 5s-timeout/XFAIL example harness with a real one: each example runs as a subprocess with a fixed smoke timeout — an example that errors fails, one that times out passes (only asserts it starts and runs without crashing). Added as an always-run example_smoke job in the PR workflow. A regularly scheduled full run (run to completion) is tracked in PRI-330. Running the examples for real surfaced breakages, fixed here: - shapiq examples: the interpretability shapiq floor was too low for lowest-direct CI (needs TabPFNExplainer/class_index). - shap_example: shap added to a new "examples" dependency group, floored to avoid an unbuildable numba. - generate_data_following_dag: import the unsupervised.experiments submodule explicitly. - tabebm example: xfailed and documented; broken with current TabPFN (#225). Closes #51. Supersedes #321 (builds on @mvanhorn's work there). Co-Authored-By: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
The
tests/test_examples.pyharness (5s timeout + XFAIL for slow examples, gated behind--run-examples) is implemented, but the CI pytest steps ranpytest tests/without--run-examples, so example files were never imported in CI — which is exactly what #51 asks for.Changes
Add
--run-examplesto the Linux and Windows pytest steps in.github/workflows/pull_request.yml, plus a changelog entry.Fixes #51
AI was used for assistance.