Repository navigation
Reduce peak CUDA memory in AutoInt and FT-Transformer - #203
Open
SergeevVladislav wants to merge 2 commits into
Open
SergeevVladislav wants to merge 2 commits into
SergeevVladislav wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was done:
AutoInt and FT-Transformer split large attention calculations along the query dimension. Each query still normalizes over every key. On PyTorch versions with non-reentrant checkpointing, backward recomputes one block at a time. AutoInt preserves its existing post-dropout weighting. FT-Transformer uses PyTorch's scaled dot-product attention when available during normal training. Direct calls to its attention layer still return the full attention map by default.
The table shows peak GPU memory allocated by PyTorch during one
TabularAutoML.fit_predictfold on an RTX 5090 (32 GB). Each run used 2,048 training rows, 512 validation rows, one epoch, numeric features and batch size 512.Across three runs at 300 features, median fold time rose from 0.607 to 0.639 seconds for AutoInt and from 0.515 to 0.546 seconds for FT-Transformer. Both also trained under a 3 GiB limit with the PyTorch math attention backend. A V100 was unavailable for direct measurement.
Validation
pytest -q tests/unit/test_ml_algo/test_attention_memory.py tests/unit/test_ml_algo/test_dl_model_device.py tests/unit/test_automl/test_presets/test_tabular_compatibility.py 'tests/integration/test_sswarm_gpu.py::test_gpu_training_then_sswarm_on_cpu[fttransformer]'passed all 54 cases, including CUDA training followed by CPU inference.8e-8for predictions and3e-8for input and parameter gradients. The 10 new attention tests also passed on Python 3.10 with CPU PyTorch 1.13.1.pre-commit run --all-filespassed.codespellpassed on the changed files. The repository-wide check reports the same existing typo inlightautoml/dataset/utils.pyondevelopment.python check_docs.pypassed.pytest -x -q tests/unit tests/integrationwith the repository's warning filters stopped at the binary XGBoost holdout test after 72 passed and 4 skipped. The same test scored 0.6861 ondevelopmentand on this change, below its 0.69 threshold.Why / Goal:
Training on wide tables creates an attention map whose size grows quadratically with the number of features. At batch size 512 this can exhaust GPU memory for AutoInt or FT-Transformer while other networks in the same preset finish. Bounding the temporary attention blocks lets these models train at the configured batch size.
Related issues:
Related to #128 (AutoInt and FT-Transformer).