IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference

[Paper] [Slides] [MLSys] [Artifact Evaluation]

This repository currently provides the artifact evaluation workflow and sample implementation for reproducing the main results in the paper. For reproduction details, see ArtifactEvaluation/.

Figure: IntAttention replaces the floating-point softmax island with an integer probability path.

Overview

IntAttention is a fully integer attention pipeline for efficient Transformer inference on edge devices. Conventional quantized attention still falls back to floating-point execution for the probability path, creating a dequantize -> softmax -> requantize path. IntAttention removes this detour and keeps the full attention path in integer arithmetic.

The target dataflow is:

• S8 x S8 -> S32 for query-key accumulation
• S32 -> U8 IndexSoftmax for probability generation
• U8 x S8 -> S32 for probability-value mixing

Compared with conventional INT8 attention pipelines, IntAttention reduces data type conversion overhead, memory traffic, and CPU latency while maintaining competitive accuracy.

Why It Matters

Quantized attention already accelerates the GEMM-heavy part of attention, but the probability path remains a major bottleneck on edge CPUs. After INT8 GEMM acceleration, the dequantize -> softmax -> requantize path can account for up to 65% of total attention latency, making probability normalization the next critical optimization target.

Figure: After INT8 GEMM acceleration, the probability path becomes the dominant bottleneck.

Method

At the core of IntAttention is IndexSoftmax, a hardware-friendly integer softmax replacement that consumes INT32 logits directly and produces U8 probabilities for integer value mixing.

IndexSoftmax is built from three components:

• Clipping: removes logits with negligible contribution after max subtraction
• LUT exponential: approximates exponentials in a bounded integer domain
• Integer normalization: rebuilds row-wise probabilities without floating-point conversion

This design is:

• Fully integer: no floating-point softmax island in the attention path
• Plug-and-play: no retraining or quantization-aware training required
• Edge-oriented: compact operators and lookup tables suitable for resource-constrained CPUs

Figure: IndexSoftmax replaces floating-point softmax with clipping, LUT exponential, and integer normalization.

Main Results

IntAttention delivers consistent improvements across both language and vision models on Arm-based platforms:

• Up to 3.7x speedup over FP16 attention
• Up to 2.0x speedup over conventional INT8 attention pipelines
• Up to 61% energy reduction over FP16 baselines
• Competitive accuracy on both LLM and ViT workloads

The reported results cover both RK3588S2 and Apple M2, with the Arm CPU implementation built on top of Arm Compute Library (ACL).

Figure: Throughput improvement on Apple M2.

Figure: Competitive accuracy on both language and vision tasks.

Citation

If you find this work useful, please cite:

@inproceedings{
zhong2026intattention,
title={IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference},
author={Wanli Zhong and Haibo Feng and Zirui Zhou and Hanyang Peng and Shiqi Yu},
booktitle={Ninth Conference on Machine Learning and Systems},
year={2026},
url={https://openreview.net/forum?id=CPCRITwAaP}
}