FlashAttention-4 Highlights Co-Design
The new FlashAttention-4 release demonstrates the power of hardware-software co-design. By pipelining kernel operations and tailoring memory access patterns, the algorithm achieves major speedups for AI workloads. The paper serves as a practical guide for how algorithmic insights can be translated into efficient digital architectures on FPGAs or ASICs.
The core bottleneck in Transformer models isn't just computation, but memory access. Standard attention mechanisms constantly move large intermediate matrices between the GPU's vast but slower High-Bandwidth Memory (HBM) and its small, extremely fast on-chip SRAM. This I/O overhead becomes the limiting factor for performance, a problem addressed by the entire FlashAttention series. Developed by Tri Dao and collaborators, the algorithm has evolved to target specific GPU architectures. The original FlashAttention introduced tiling to reduce HBM writes. FlashAttention-2 enhanced parallelism and work partitioning on A100 GPUs, roughly doubling performance. FlashAttention-3 was then optimized for the asynchronous features of Hopper (H100) GPUs to further boost hardware utilization. FlashAttention-4 directly confronts a trend called "asymmetric hardware scaling" seen in new accelerators like NVIDIA's Blackwell GPUs. On this new hardware, the throughput of Tensor Cores (for matrix multiplication) is increasing far more rapidly than the bandwidth of shared memory or the speed of special function units (SFUs) used for operations like softmax. To overcome these new bottlenecks, FlashAttention-4 redesigns the algorithm's software pipeline. It maximizes the overlap between matrix multiplication and the slower operations, for instance, by using otherwise idle FMA (fused multiply-add) units to perform a software emulation of the exponential function, assisting the dedicated but slower hardware units. This co-design pays off with significant performance gains on the latest hardware. On an NVIDIA B200 GPU, FlashAttention-4 can achieve up to 1605 TFLOPs/s, representing 71% of the chip's theoretical maximum utilization. This makes it up to 1.3 times faster than NVIDIA's own cuDNN library and 2.7 times faster than leading Triton implementations for the same task. The entire kernel is implemented in CuTe-DSL, a Python-based domain-specific language provided by NVIDIA's CUTLASS library. This approach allows for high-level, abstract programming consistent with CUTLASS while providing low-level control, and it reduces kernel compilation times by a factor of 20-30x compared to using traditional C++ templates. This relentless optimization of the attention mechanism is a key
