Repository navigation
Fix feature selection limits, CV permutation importance and shared input budgets - #202
Open
SergeevVladislav wants to merge 1 commit into
Open
SergeevVladislav wants to merge 1 commit into
SergeevVladislav wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was done:
max_input_featuresbudget for tabular ensembles, stacked levels, time-utilized configurations and the two branches of a T-learner. A seeded shallow Extra Trees screen runs on training inputs. In uplift, each arm contributes equally to the shared ranking. The usual per-pipeline selection still applies inside that budget. The FAQ includes training and serving examples.Validation on Python 3.12:
python -m pytest -q tests/unit/test_pipelines/test_feature_selection.py tests/unit/test_automl/test_presets/test_input_feature_budget.pywith the project warning policy: 29 passed. Covers strict budgets, weighted OOF against a manual reference, weighted DataFrame/CSV reports, explicit/time folds, multiple levels/configurations, uplift and serialization.pre-commit run --all-files: passed.sphinx-build -b html docs <build-dir>: succeeded with 14 warnings about existing inventories, links and documentation.tests/unitand the four demos) gave 265 passed, 7 skipped, 3 failed, 4 subtests passed. Reproduced all three failures separately on unmodified development: the binary XGBoost holdout AUC is 0.6861005434782608 (threshold >0.69), and the two multiclass XGBoost paths fail with CUDA hidden. Other integration modules and the full multi-version suite were not run locally.212e38ewith a single independent commit. Compared changed lines and simulated merging all seven other open PRs. No overlapping changed lines or additional conflict blocks. Add extended evaluation metrics and Task integration #195 and Fix boosting training and recover from failed Optuna trials #198–Fix uplift training, categorical treatment and model selection #201 merge cleanly with this change. [LAML-4] Add SAINT #131 and Parallel runs for Optuna #166 retain their existing conflicts against development.Why / Goal:
Users need a predictable limit on the raw columns required to serve a model, including when an ensemble or two uplift branches use different subsets. Validators need permutation scores that use the same held-out rows, labels and weights for both terms. Selection quality should be evaluated on a separate holdout or with the whole selection procedure repeated within each outer fold.
Related issues:
Closes #62. Related to #49 and #138.