Skip to content

Fix feature selection limits, CV permutation importance and shared input budgets - #202

Open
SergeevVladislav wants to merge 1 commit into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/issue-62-feature-selection
Open

SergeevVladislav wants to merge 1 commit into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/issue-62-feature-selection

Conversation

@SergeevVladislav

@SergeevVladislav SergeevVladislav commented Nov 18, 2026 •

Copy link
Copy Markdown

What was done:

  • Enforce the iterative selector's limit on original inputs, including forced features. Evaluate the forced subset first, truncate groups to the available slots, validate invalid limits, and keep selected and dropped lists consistent. Support already-numeric data and permutation scoring with a pre-fitted model and its OOF predictions.
  • Add an opt-in shared max_input_features budget for tabular ensembles, stacked levels, time-utilized configurations and the two branches of a T-learner. A seeded shallow Extra Trees screen runs on training inputs. In uplift, each arm contributes equally to the shared ranking. The usual per-pipeline selection still applies inside that budget. The FAQ includes training and serving examples.
  • Calculate CV permutation importance on each fold's held-out rows with its fitted model. Preserve weights, partial coverage and overlapping folds. Accurate importance reports use a local random seed and the same serving path for baseline and permutations, handle overlapping target/feature names, and load labeled CSVs with their metric weights.
  • Respect explicit fold IDs and assign chronological folds back to their original rows. Accept a separate labeled CSV validation set in the tabular preset.

Validation on Python 3.12:

  • python -m pytest -q tests/unit/test_pipelines/test_feature_selection.py tests/unit/test_automl/test_presets/test_input_feature_budget.py with the project warning policy: 29 passed. Covers strict budgets, weighted OOF against a manual reference, weighted DataFrame/CSV reports, explicit/time folds, multiple levels/configurations, uplift and serialization.
  • pre-commit run --all-files: passed. sphinx-build -b html docs <build-dir>: succeeded with 14 warnings about existing inventories, links and documentation.
  • RTX 5090 / CatBoost 1.2.10: verified three native GPU-trained folds, a five-input ensemble and identical full/shortlist predictions after deserialization in a fresh CUDA-hidden process. Loaded 60 additional binary artifacts trained on unmodified development and reproduced their stored predictions exactly on both input schemas.
  • Re-ran the audit cases on synthetic noise: OOF AUC 0.464, permutation importances between -0.021 and +0.005. The former comparison to ensemble training predictions gave importances below -0.45. Preserved all 1,600 supplied fold IDs.
  • Focused validation on the final code: the 29 new cases plus reader/iterator unit tests and four selector/stacking integration demos gave 37 passed. During development the broader CPU-only run (tests/unit and the four demos) gave 265 passed, 7 skipped, 3 failed, 4 subtests passed. Reproduced all three failures separately on unmodified development: the binary XGBoost holdout AUC is 0.6861005434782608 (threshold >0.69), and the two multiclass XGBoost paths fail with CUDA hidden. Other integration modules and the full multi-version suite were not run locally.
  • Matched quality check for the new shared budget on the repository's sampled applications dataset (10,000 rows, three outer stratified 75/25 splits, one three-fold CatBoost, up to 80 trees, per-pipeline selection disabled). The budget reduced 110 inputs to 30. Mean external ROC-AUC with and without the budget was 0.7038 vs 0.6881 on CPU and 0.7096 vs 0.7008 on GPU. Results are specific to this dataset and configuration.
  • GitHub pre-commit, all three dependency compatibility jobs and both optional feature-generation jobs passed. The general Python 3.10 CI job was still running at the time of this update.
  • Based on development 212e38e with a single independent commit. Compared changed lines and simulated merging all seven other open PRs. No overlapping changed lines or additional conflict blocks. Add extended evaluation metrics and Task integration #195 and Fix boosting training and recover from failed Optuna trials #198–Fix uplift training, categorical treatment and model selection #201 merge cleanly with this change. [LAML-4] Add SAINT #131 and Parallel runs for Optuna #166 retain their existing conflicts against development.

Why / Goal:
Users need a predictable limit on the raw columns required to serve a model, including when an ensemble or two uplift branches use different subsets. Validators need permutation scores that use the same held-out rows, labels and weights for both terms. Selection quality should be evaluated on a separate holdout or with the whole selection procedure repeated within each outer fold.

Related issues:

Closes #62. Related to #49 and #138.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant