Code and results for the paper by Klemens Flöge, Simon Bing, Leo Grinsztajn, Eliott Kalfon, Lennart Purucker, Noah Hollmann and Frank Hutter (AI4Mat workshop, NeurIPS 2026).
We compare tabular foundation models (TabPFN-3, TabICL) with Gaussian processes, random forests and CatBoost as surrogates for pool-based Bayesian optimization on real experimental campaigns. Each round, the optimizer picks the next candidate to measure from a finite pool of settings that were measured in the source study.
a, design space. b, the conditions the source study measured form the pool. c, features x are visible; outcomes y stay hidden until queried. d, the surrogate, fit on the queried rows, scores the rest, and the acquisition rule picks the next candidate, whose outcome is then revealed.
| path | contents |
|---|---|
src/matbo/ |
data loaders, surrogates, acquisition rules, the BO loop, analysis and figures |
reference_results/ |
the traces behind every number in the paper: which candidate each arm picked at each step, for every campaign and seed |
scripts/ |
run arms locally or on Slurm, merge runs, export traces, check coverage and pool provenance |
data/ |
the Liang et al. datasets (licenses and sources) |
docs/PROVENANCE.md |
how each candidate pool is built from its source |
Benchmarks. liang: 5 materials campaigns from
Liang et al. (2021),
50 seeds, scored on Top-5% found. olympus: 10 chemistry and colorimetry
campaigns from Häse et al. (2021),
100 seeds, scored on best value found. Every run starts from 2 random points,
picks one candidate per round, and continues until the pool is exhausted.
Arms. 11 surrogates × 5 acquisition rules, plus Random: 56 arms.
- Surrogates:
TabPFN,TabICL,CatBoost,RF, and seven GP kernels (GP-ARD,GP-M32,GP-M12,GP-RBF,GP-iso,GP-MLP,GP-ARD-Warp). - Rules:
greedy,EI,PI,UCB1,UCB2. - An arm is named
<surrogate>-<rule>, for exampleTabPFN-UCB1. The registry issrc/matbo/methods/__init__.py.
uv sync --locked --extra dev
uv pip install olymp --no-deps # the Olympus datasets; see data/README.mduv.lock pins the library versions the paper's runs used. The TabPFN arms
run TabPFN-3 (tabpfn 8.2.0, regressor checkpoint
tabpfn-v3-regressor-v3_default.ckpt); the code selects this model explicitly,
so a newer default checkpoint cannot change the results. The weights are gated: accept the license once when TabPFN first asks, or set TABPFN_TOKEN.
All figures are computed from reference_results/. No model is fitted.
uv run python -m matbo.visualisation # Figures 1, 2, D.1-D.9 -> figures/paper/
uv run python -m matbo.visualisation 1 D.3 # selected figures only
uv run python -m matbo.analysis.posthoc # Table D.1 and the table behind D.9 -> figures/paper/posthoc/Normalized improvement (NI) rescales each campaign and budget so that the worst of the 56 arms scores 0 and the best scores 1 (App. C).
The full grid is expensive. For example, one GP arm on olympus runs 100 seeds
through pools of up to 1,386 candidates. To re-run some or all arms and
compare them with the record:
# one arm on one campaign, 2 processes
FAMILIES=olympus ONLY=snar METHODS=RF-EI,Random WORKERS=2 TAG=check \
uv run python scripts/run_benchmark.py
# every arm on a benchmark, one arm at a time
FAMILIES=liang TAG=liang ./scripts/run_arms.sh
# write a finished run into reference_results/, then check that no cell is missing
BENCHMARK=liang TAG=liang uv run python scripts/export_reference.py
uv run python scripts/coverage.pyRuns write to results/<TAG>/. scripts/merge_runs.py joins runs that were
split by arm or by seed (SEED_SLICE=start:stop).
On a Slurm cluster, scripts/slurm/ runs the grid as a job array with one task
per arm and seed range:
FAMILIES=liang TAG=liang ./scripts/slurm/submit.sh # print the plan
FAMILIES=liang TAG=liang CONFIRM=1 ./scripts/slurm/submit.sh # submit it
TAG=liang ./scripts/slurm/collect.sh # merge the finished tasksSet the partition, account and time limit with sbatch's own variables
(SBATCH_PARTITION, SBATCH_ACCOUNT, SBATCH_TIMELIMIT) or SBATCH_ARGS.
Run on CPU (DEVICE=cpu; the Slurm scripts set it). The recorded traces were
computed on CPU, except on hplc, photo_pce10 and photo_wf3: there, the
five TabICL arms and TabPFN-PI, TabPFN-UCB1 and TabPFN-UCB2 ran on a GPU
(run tags ending in _rtx). On a GPU, small floating-point differences change which
candidate an arm picks, and the run follows a different path.
A re-run picks the same candidates as the recorded trace when the numerics match. Most cells we re-ran on a different machine matched exactly. Some didn't: GP fits that fail and fall back to the prior, and some random-forest ties, depend on the platform's floating-point behaviour.
reference_results/README.md describes the trace format and one documented
substituted seed (GP-M12-UCB1 on photo_pce10, seed 5).
uv run pytest tests -qThe tests check, among other things, that every committed trace decodes
against the current pool (pool_fingerprint) and that the figures use exactly
the 56 arms.
Apache 2.0 (see LICENSE). The datasets keep their own licenses
(see data/README.md).
