DeepEP
A GPU communication library for MoE training and inference on Linux, with FP8 dispatch and support for NVIDIA Hopper or newer GPUs.
DeepEP handles communication between GPUs for teams training or serving mixture-of-experts (MoE) models on their own servers. It moves tokens to the GPUs that hold the selected experts and combines their outputs.
- Training and inference: The same interface supports training, prompt processing and token generation, with communication designed for high throughput and low latency.
- FP8 dispatch: Token dispatch supports FP8, and communication can overlap with model computation rather than making each stage wait for the other.
- Expert load balancing: Dynamic expert replication distributes a busy expert's computation across additional GPUs, with NVLink transfers for expert weights and gradients.
DeepEP is open source under the MIT license. It runs on Linux with NVIDIA Hopper or newer GPUs and depends on PyTorch, CUDA and NCCL. It requires NVLink within a server and an RDMA network between servers. InfiniBand is a tested network option; RoCE compatibility is theoretical.
The CUDA toolkit and a host compiler must remain available at runtime because DeepJIT compiles GPU communication kernels for the device in use.
