FlashAttention
An open-source PyTorch attention library that reduces GPU memory use for transformer models, with NVIDIA CUDA and AMD ROCm support.
FlashAttention is an attention library for developers training or running transformer models on their own GPUs. It computes exact attention while using less memory than standard PyTorch attention, allowing longer sequences.
- Lower attention memory use: Memory grows linearly with sequence length rather than quadratically; speed gains depend on the GPU and workload.
- Training support: Forward and backward computation supports model training, with causal attention, grouped-query and multi-query attention, rotary embeddings and ALiBi available through supported implementations.
- AMD attention backends: Composable Kernel and Triton implementations support AMD GPUs, with paged attention and variable sequence lengths available in the Triton backend.
- Model development: A GPT implementation, GPT-2 and GPT-3 training scripts, and optimized MLP, LayerNorm and cross-entropy layers support work beyond the attention operation itself.
The Python library works with PyTorch and is open source under BSD-3-Clause. NVIDIA CUDA support covers Ampere, Ada and Hopper GPUs, including RTX 3090, RTX 4090, A100 and H100. The CuTeDSL implementation targets Hopper and Blackwell hardware, including B200. AMD ROCm support covers compatible Instinct and RDNA GPUs; supported features differ between backends.