Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 60 additions & 9 deletions docs/pages/FAQ.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,17 +132,68 @@ An unspecified string column in ``TabularAutoML`` is normally treated as a
categorical feature. Assign a text role and use the appropriate NLP preset or
build a custom pipeline when TF-IDF or transformer features are required.

Does feature selection affect every model?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
How can I limit the number of columns needed at inference?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Set a shared input budget when constructing a tabular preset:

.. code-block:: python

automl = TabularAutoML(Task("binary"), selection_params={"max_input_features": 30})
oof = automl.fit_predict(train, roles={"target": "outcome"})
inputs = list(automl.reader.used_features)
predictions = automl.predict(test[inputs])

The budget counts distinct original predictor columns across every pipeline and
stacking level of this fitted preset. Targets, weights, folds and intermediate
stacking predictions do not consume slots. Mandatory predictors
(``force_input=True``) occupy slots first. If they exceed the limit, fitting
raises ``ValueError``. With the default ``None``, the shared selection step is
disabled. For a custom tabular ``AutoML``, pass ``max_input_features`` directly
to its constructor. For ``TabularUtilizedAutoML``, pass the shared budget in
``selection_params`` to restrict the input once across configurations and seeds.

If the training input exceeds the budget, a shallow, seeded Extra Trees model
ranks columns by impurity importance, using training weights when present.
Categorical values are factorized for this ranking. Downstream pipelines keep
their own categorical processing and may retain fewer inputs. Validate the
budget and quality on held-out data. For an independent cross-validation
estimate of supervised selection, repeat the entire fit inside each outer
training fold. A supplied ``folds`` role takes precedence over generated folds;
use consecutive integer fold IDs starting at zero.

For an uplift T-learner, set the budget on the meta-learner:

Not necessarily. ``selection_params["select_algos"]`` controls which
algorithm families receive selected features. In the default tabular
configuration it contains ``"gbm"``; linear and random-forest pipelines do
not automatically receive the same selected subset.
.. code-block:: python

uplift = TLearner(treatment_model, control_model, max_input_features=30)
uplift.fit(train, roles={"target": "outcome", "treatment": "arm"})
inputs = sorted(
set(uplift.treatment_learner.reader.used_features)
| set(uplift.control_learner.reader.used_features)
)
effects, treated, control = uplift.predict(test[inputs])

Each arm contributes equally to the shared importance score. Both learners
train on the resulting covariates. Treatment assignment is a training role.

How does the per-pipeline selector use its limit?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``automl.collect_used_feats()`` returns the features retained by the fitted
AutoML object. Fast feature importance should not be interpreted as proof
that every lower-ranked feature is safe to remove.
``selection_params["select_algos"]`` controls which families receive additional
selection. The default is ``["gbm"]``. In mode 2, ``max_features_cnt_in_result``
bounds one iterative selector. It starts with forced inputs and evaluates
groups of up to ``feature_group_size`` additional inputs, truncating the last
group to the available space. Mode 1 applies an importance cutoff. A shared
``max_input_features`` also applies when this per-pipeline selection is disabled.

After fitting or loading a model, ``automl.reader.used_features`` lists the
input columns for prediction. The full input DataFrame can also be passed.
``automl.collect_used_feats()`` includes inputs to internal stacking levels.
``get_feature_scores("fast")`` reports selector importances.
``get_feature_scores("accurate", data, random_state=42)`` measures the effect
of a reproducible row permutation on the fitted model using a labeled DataFrame
or file. Include the weights column to evaluate the weighted metric.


Configuration and resources
Expand Down
9 changes: 9 additions & 0 deletions lightautoml/addons/uplift/metalearners.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
from lightautoml.tasks import Task
from lightautoml.utils.timer import Timer
from lightautoml.validation.np_iterators import UpliftIterator
from lightautoml.pipelines.selection.input_budget import restrict_training_inputs
from lightautoml.reader.base import PandasToPandasReader

from .utils import _get_target_role
from .utils import _get_treatment_role
Expand Down Expand Up @@ -229,6 +231,7 @@ class TLearner(MetaLearner):
treatment_learner: AutoML model, if `None` then will be used model by default
control_learner: AutoML model, if `None` then will be used model by default
base_task: task
max_input_features: Shared upper bound on original predictors for both arms.
timeout: Timeout
cpu_limit: CPU limit that that are passed to each automl.
gpu_ids: GPU IDs that are passed to each automl.
Expand All @@ -243,6 +246,7 @@ def __init__(
timeout: Optional[int] = None,
cpu_limit: int = 4,
gpu_ids: Optional[str] = "all",
max_input_features: Optional[int] = None,
):
assert any(
x is not None
Expand All @@ -261,6 +265,7 @@ def __init__(

super().__init__(base_task, timeout, cpu_limit, gpu_ids)

self.max_input_features = max_input_features
self.treatment_learner = (
treatment_learner if treatment_learner is not None else self._get_default_learner(self.base_task)
)
Expand All @@ -287,6 +292,10 @@ def _fit(
treatment_col,
) = _get_treatment_role(roles)

if self.max_input_features is not None:
reader = PandasToPandasReader(self.base_task, cv=None, advanced_roles=False, n_jobs=1)
train_data, roles = restrict_training_inputs(train_data, roles, reader, self.max_input_features)

new_roles = copy.deepcopy(roles)
new_roles.pop(treatment_role)

Expand Down
12 changes: 12 additions & 0 deletions lightautoml/addons/utilization/utilization.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
from ...dataset.utils import concatenate
from ...ml_algo.base import MLAlgo
from ...pipelines.ml.base import MLPipeline
from ...pipelines.selection.input_budget import restrict_training_inputs
from ...reader.base import PandasToPandasReader
from ...reader.tabular_batch_generator import read_data
from ...tasks import Task
from ...utils.logging import set_stdout_level
from ...utils.logging import verbosity_to_loglevel
Expand Down Expand Up @@ -284,6 +287,15 @@ def fit_predict(
logger.info("\x1b[1mIf one preset completes earlier, next preset configuration will be started\x1b[0m\n")

timer = PipelineTimer(self.timeout, **self.timing_params).start()
limit = (self.kwargs.get("selection_params") or {}).get("max_input_features")
if limit is not None:
frame, file_roles = read_data(
train_data, train_features, self.cpu_limit, self.kwargs.get("read_csv_params")
)
reader_params = {**(self.kwargs.get("reader_params") or {}), "cv": None, "advanced_roles": False}
reader = PandasToPandasReader(self.task, **reader_params)
train_data, roles = restrict_training_inputs(frame, {**(roles or {}), **(file_roles or {})}, reader, limit)
train_features = None
history = []

amls = [[] for _ in range(len(self.configs_list))]
Expand Down
60 changes: 33 additions & 27 deletions lightautoml/automl/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from ..dataset.base import LAMLDataset
from ..dataset.utils import concatenate
from ..pipelines.ml.base import MLPipeline
from ..pipelines.selection.input_budget import select_input_features
from lightautoml.dataset.roles import TargetRole
from ..reader.base import Reader
from ..utils.logging import set_stdout_level
Expand Down Expand Up @@ -64,6 +65,7 @@ class AutoML:
>>> automl.fit_predict(data, roles={'target': 'TARGET'})

Args:
max_input_features: Optional shared upper bound on original input columns.
reader: Instance of Reader class object that
creates :class:`~lightautoml.dataset.base.LAMLDataset`
from input data.
Expand Down Expand Up @@ -96,8 +98,9 @@ def __init__(
skip_conn: bool = False,
return_all_predictions: bool = False,
debug: bool = False,
max_input_features: Optional[int] = None,
):
self._initialize(reader, levels, timer, blender, skip_conn, return_all_predictions, debug)
self._initialize(reader, levels, timer, blender, skip_conn, return_all_predictions, debug, max_input_features)

def _initialize(
self,
Expand All @@ -108,12 +111,14 @@ def _initialize(
skip_conn: bool = False,
return_all_predictions: bool = False,
debug: bool = False,
max_input_features: Optional[int] = None,
):
"""Same as __init__. Exists for delayed initialization in presets.

Args:
reader: Instance of Reader class object that
creates :class:`~lightautoml.dataset.base.LAMLDataset` from input data.
max_input_features: Optional shared upper bound on original input columns.
levels: List of list of :class:`~lightautoml.pipelines.ml..base.MLPipelines`.
timer: Timer instance of :class:`~lightautoml.utils.timer.PipelineTimer`.
Default - unlimited timer.
Expand All @@ -139,6 +144,7 @@ def _initialize(
if timer is None:
self.timer = PipelineTimer()
self.reader = reader
self.max_input_features = max_input_features
self._levels = levels

# default blender is - select best model and prune other pipes
Expand Down Expand Up @@ -210,6 +216,11 @@ def fit_predict(
self.timer.start()
train_dataset = self.reader.fit_read(train_data, train_features, roles)

if self.max_input_features is not None:
inputs = select_input_features(train_dataset, self.max_input_features)
self.reader.upd_used_features(remove=set(train_dataset.features).difference(inputs))
train_dataset = train_dataset[:, inputs]

# Saving class mapping
target_col_name = roles["target"] if "target" in roles else roles[TargetRole()]
if self.reader.task.name == "binary":
Expand Down Expand Up @@ -347,33 +358,28 @@ def predict(
"""
dataset = self.reader.read(data, features_names=features_names, add_array_attrs=False)

for n, level in enumerate(self.levels, 1):
# check if last level

level_predictions = []
for _n, ml_pipe in enumerate(level):
level_predictions.append(ml_pipe.predict(dataset))

if n != len(self.levels):

level_predictions = concatenate(level_predictions)

if self.skip_conn:

try:
# convert to initial dataset type
level_predictions = dataset.from_dataset(level_predictions)
except TypeError:
raise TypeError(
"Can not convert prediction dataset type to input features. Set skip_conn=False"
)
dataset = concatenate([level_predictions, dataset])
else:
dataset = level_predictions
else:
return self._predict_from_level(dataset, 0, return_all_predictions)

def _next_level_input(self, dataset, predictions):
predictions = concatenate(predictions)
if self.skip_conn:
try:
predictions = dataset.from_dataset(predictions)
except TypeError as exc:
raise TypeError(
"Can not convert prediction dataset type to input features. Set skip_conn=False"
) from exc
return concatenate([predictions, dataset])
return predictions

def _predict_from_level(self, dataset, start, return_all_predictions=False):
for n in range(start, len(self.levels)):
predictions = [pipe.predict(dataset) for pipe in self.levels[n]]
if n == len(self.levels) - 1:
if (return_all_predictions is None and self.return_all_predictions) or return_all_predictions:
return concatenate(level_predictions)
return self.blender.predict(level_predictions)
return concatenate(predictions)
return self.blender.predict(predictions)
dataset = self._next_level_input(dataset, predictions)

def collect_used_feats(self) -> List[str]:
"""Get the union of features used by fitted inference pipelines.
Expand Down
2 changes: 2 additions & 0 deletions lightautoml/automl/presets/tabular_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ selection_params:
feature_group_size: 1
# max features count (mode2)
max_features_cnt_in_result:
# Shared budget of original input columns across all pipelines and levels (all modes).
max_input_features:
# list of algos to apply selector. Possible values - 'gbm', 'rf', 'linear_l2'. gbm stands for both catboost and lgb
select_algos: [ 'gbm' ]

Expand Down
15 changes: 10 additions & 5 deletions lightautoml/automl/presets/tabular_presets.py
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,7 @@ def create_automl(self, **fit_args):
levels,
skip_conn=self.general_params["skip_conn"],
blender=blender,
max_input_features=None if self.is_time_series else self.selection_params.get("max_input_features"),
return_all_predictions=self.general_params["return_all_predictions"],
timer=self.timer,
debug=self.debug,
Expand Down Expand Up @@ -784,7 +785,7 @@ def fit_predict(
if upd_roles:
roles = {**roles, **upd_roles}
if valid_data is not None:
data, _ = read_data(valid_data, valid_features, self.cpu_limit, self.read_csv_params)
valid_data, _ = read_data(valid_data, valid_features, self.cpu_limit, self.read_csv_params)

if self.is_time_series:
train = {"seq": {"seq0": train}}
Expand Down Expand Up @@ -888,6 +889,7 @@ def get_feature_scores(
data: Optional[ReadableToDf] = None,
features_names: Optional[Sequence[str]] = None,
silent: bool = True,
random_state: int = 42,
):
"""Calculate feature importances for the fitted AutoML.

Expand All @@ -902,6 +904,7 @@ def get_feature_scores(
data: Dataset used by the accurate method.
features_names: Optional feature names when they cannot be
inferred from ``data``.
random_state: Seed for the accurate method's row permutation.
silent: If ``False``, log why importances cannot be returned.

Returns:
Expand Down Expand Up @@ -940,8 +943,7 @@ def get_feature_scores(
logger.info2("Data parameter is not setup for accurate calculation method. Aborting...")
return None

read_csv_params = self._get_read_csv_params()
data, _ = read_data(data, features_names, self.cpu_limit, read_csv_params)
data, _ = read_data(data, features_names, self.cpu_limit, self.read_csv_params)
used_feats = self.collect_used_feats()
fi = calc_feats_permutation_imps(
self,
Expand All @@ -950,6 +952,7 @@ def get_feature_scores(
self.reader.target,
self.task.get_dataset_metric(),
silent=silent,
random_state=random_state,
)
return fi

Expand Down Expand Up @@ -1124,6 +1127,7 @@ def get_feature_scores(
data: Optional[ReadableToDf] = None,
features_names: Optional[Sequence[str]] = None,
silent: bool = True,
random_state: int = 42,
):
"""Calculate aggregated feature importances for the utilized preset.

Expand All @@ -1138,6 +1142,7 @@ def get_feature_scores(
data: Dataset used by the accurate method.
features_names: Optional feature names when they cannot be
inferred from ``data``.
random_state: Seed for the accurate method's row permutation.
silent: If ``False``, log why importances cannot be returned.

Returns:
Expand Down Expand Up @@ -1181,8 +1186,7 @@ def get_feature_scores(
return None

automl = self.outer_pipes[0].ml_algos[0].models[0][0]
read_csv_params = automl._get_read_csv_params()
data, _ = read_data(data, features_names, self.cpu_limit, read_csv_params)
data, _ = read_data(data, features_names, self.cpu_limit, automl.read_csv_params)

used_feats = set()
for pipe in self.outer_pipes:
Expand All @@ -1195,6 +1199,7 @@ def get_feature_scores(
automl.reader.target,
automl.task.get_dataset_metric(),
silent=silent,
random_state=random_state,
)
return fi

Expand Down
Loading
Loading