Skip to content

test: run example-file tests in pull request CI - #321

Closed
mvanhorn wants to merge 2 commits into
PriorLabs:mainfrom
mvanhorn:test/reenable-example-testing
Closed

mvanhorn wants to merge 2 commits into
PriorLabs:mainfrom
mvanhorn:test/reenable-example-testing

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

Summary

The tests/test_examples.py harness (5s timeout + XFAIL for slow examples, gated behind --run-examples) is implemented, but the CI pytest steps ran pytest tests/ without --run-examples, so example files were never imported in CI — which is exactly what #51 asks for.

Changes

Add --run-examples to the Linux and Windows pytest steps in .github/workflows/pull_request.yml, plus a changelog entry.

Fixes #51

AI was used for assistance.

The tests/test_examples.py harness (5s timeout + XFAIL for slow examples,
gated behind --run-examples) already exists, but the CI pytest steps ran
without --run-examples, so example files were never imported in CI. Add
--run-examples to the Linux and Windows steps so the example suite runs.

Fixes PriorLabs#51
@mvanhorn
mvanhorn requested a review from a team as a code owner June 23, 2026 01:07
@mvanhorn
mvanhorn requested review from alanprior and removed request for a team June 23, 2026 01:07
@CLAassistant

CLAassistant commented Jun 23, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ adrian-prior
❌ mvanhorn
You have signed the CLA already but the status is still pending? Let us recheck it.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the timeout for slow example tests from 1 to 5 seconds and moves the imports of threading and conftest to the top level of tests/test_examples.py. The feedback points out that importing conftest at the top level is a pytest anti-pattern that can lead to import errors, and suggests reverting to importing HAS_TABPFN and TABPFN_SOURCE locally within the test function.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread tests/test_examples.py

import pytest

import conftest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Importing conftest at the top level of a test module is a pytest anti-pattern. conftest.py is loaded automatically by pytest, and importing it directly can lead to ModuleNotFoundError during test collection (since the tests/ directory is typically not in sys.path at import time) or double-import issues. It is safer to import the required variables locally within the test function where conftest is already guaranteed to be loaded.

Comment thread tests/test_examples.py
"""
from conftest import HAS_TABPFN, TABPFN_SOURCE

file_name = example_file["name"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Re-introduce the local import of HAS_TABPFN and TABPFN_SOURCE from conftest inside the test function to avoid importing conftest at the top level.

Suggested change
file_name = example_file["name"]
from conftest import HAS_TABPFN, TABPFN_SOURCE
file_name = example_file["name"]

Comment thread tests/test_examples.py
Comment on lines +147 to +151
if not conftest.HAS_TABPFN:
pytest.skip(
f"Example {file_name} requires TabPFN package, but it's not installed",
)
elif TABPFN_SOURCE == "tabpfn_client":
elif conftest.TABPFN_SOURCE == "tabpfn_client":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Revert the usage of conftest.HAS_TABPFN and conftest.TABPFN_SOURCE back to the locally imported HAS_TABPFN and TABPFN_SOURCE variables.

Suggested change
if not conftest.HAS_TABPFN:
pytest.skip(
f"Example {file_name} requires TabPFN package, but it's not installed",
)
elif TABPFN_SOURCE == "tabpfn_client":
elif conftest.TABPFN_SOURCE == "tabpfn_client":
if not HAS_TABPFN:
pytest.skip(
f"Example {file_name} requires TabPFN package, but it's not installed",
)
elif TABPFN_SOURCE == "tabpfn_client":

@adrian-prior

Copy link
Copy Markdown
Collaborator

Thanks for kicking this off, @mvanhorn 🙏 — closing in favour of #327, which carries it forward.

#327 keeps the same goal (run the example files in CI) but swaps the bounded-timeout/XFAIL harness for one that fails on real errors, and fixes the example breakages it surfaced instead of XFAILing past them. Your work here is credited as co-author on #327.

adrian-prior added a commit that referenced this pull request Jul 20, 2026
…327)

test: run example files as CI tests (PRI-328) (#327)

Body:
Replace the old 5s-timeout/XFAIL example harness with a real one: each
example runs as a subprocess with a fixed smoke timeout — an example that
errors fails, one that times out passes (only asserts it starts and runs
without crashing). Added as an always-run example_smoke job in the PR
workflow. A regularly scheduled full run (run to completion) is tracked in
PRI-330.

Running the examples for real surfaced breakages, fixed here:
- shapiq examples: the interpretability shapiq floor was too low for
  lowest-direct CI (needs TabPFNExplainer/class_index).
- shap_example: shap added to a new "examples" dependency group,
  floored to avoid an unbuildable numba.
- generate_data_following_dag: import the unsupervised.experiments
  submodule explicitly.
- tabebm example: xfailed and documented; broken with current TabPFN (#225).

Closes #51. Supersedes #321 (builds on @mvanhorn's work there).

Co-Authored-By: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve testing: reenable testing example files

3 participants