Skip to content

Reduce peak CUDA memory in AutoInt and FT-Transformer - #203

Open
SergeevVladislav wants to merge 2 commits into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/issue-128-tabular-attention-memory
Open

SergeevVladislav wants to merge 2 commits into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/issue-128-tabular-attention-memory

Conversation

@SergeevVladislav

Copy link
Copy Markdown

What was done:

AutoInt and FT-Transformer split large attention calculations along the query dimension. Each query still normalizes over every key. On PyTorch versions with non-reentrant checkpointing, backward recomputes one block at a time. AutoInt preserves its existing post-dropout weighting. FT-Transformer uses PyTorch's scaled dot-product attention when available during normal training. Direct calls to its attention layer still return the full attention map by default.

The table shows peak GPU memory allocated by PyTorch during one TabularAutoML.fit_predict fold on an RTX 5090 (32 GB). Each run used 2,048 training rows, 512 validation rows, one epoch, numeric features and batch size 512.

Model 300 features, before 300 features, after 1,000 features, before (12 GiB limit) 1,000 features, after (same limit)
AutoInt 2.54 GiB 0.46 GiB CUDA OOM 1.10 GiB
FT-Transformer 5.09 GiB 1.70 GiB CUDA OOM 5.45 GiB

Across three runs at 300 features, median fold time rose from 0.607 to 0.639 seconds for AutoInt and from 0.515 to 0.546 seconds for FT-Transformer. Both also trained under a 3 GiB limit with the PyTorch math attention backend. A V100 was unavailable for direct measurement.

Validation

  • pytest -q tests/unit/test_ml_algo/test_attention_memory.py tests/unit/test_ml_algo/test_dl_model_device.py tests/unit/test_automl/test_presets/test_tabular_compatibility.py 'tests/integration/test_sswarm_gpu.py::test_gpu_training_then_sswarm_on_cpu[fttransformer]' passed all 54 cases, including CUDA training followed by CPU inference.
  • Reloaded both models saved by the base revision. Maximum absolute FP32 differences on CUDA were below 8e-8 for predictions and 3e-8 for input and parameter gradients. The 10 new attention tests also passed on Python 3.10 with CPU PyTorch 1.13.1.
  • pre-commit run --all-files passed. codespell passed on the changed files. The repository-wide check reports the same existing typo in lightautoml/dataset/utils.py on development.
  • The Sphinx HTML build completed with seven warnings and python check_docs.py passed.
  • pytest -x -q tests/unit tests/integration with the repository's warning filters stopped at the binary XGBoost holdout test after 72 passed and 4 skipped. The same test scored 0.6861 on development and on this change, below its 0.69 threshold.

Why / Goal:

Training on wide tables creates an attention map whose size grows quadratically with the number of features. At batch size 512 this can exhaust GPU memory for AutoInt or FT-Transformer while other networks in the same preset finish. Bounding the temporary attention blocks lets these models train at the configured batch size.

Related issues:

Related to #128 (AutoInt and FT-Transformer).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant