FlashInfer

An open-source GPU kernel library for LLM inference on NVIDIA hardware, with PyTorch compatibility and automatic backend selection.

FlashInfer is a GPU kernel library for developers building LLM inference systems on their own hardware or servers. It supplies the attention, matrix multiplication and mixture-of-experts operations used by model serving software.

Attention and cache management: Supports prompt processing, token generation and mixed batches, with paged caches for changing batches, shared-prefix reuse, sparse attention and DeepSeek's Multi-Latent Attention.
Backend selection: Automatically chooses an implementation for the hardware and workload from backends including FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM.
Quantized and expert models: Provides FP8 and FP4 computation and fused mixture-of-experts kernels, with routing methods used by DeepSeek-V3 and Llama-4.
Sampling and decoding: Includes Top-K, Top-P and Min-P sampling without sorting, plus support for speculative decoding.
Distributed inference: Supports multi-node NVLink and NVSHMEM for serving workloads spread across GPUs and servers.

FlashInfer is open source under Apache 2.0 and works with PyTorch, including torch.compile and CUDA Graphs. Serving projects such as SGLang and vLLM use it.

It runs on NVIDIA GPUs spanning Turing through Blackwell, including RTX cards, A100, H100 and Jetson Thor. Support varies by feature and GPU; FP4 matrix multiplication requires Blackwell hardware.