XTuner

XTuner is an LLM training engine for researchers working with very large mixture-of-experts (MoE) models. Its focus is reducing memory pressure and improving distributed training efficiency when model experts receive uneven workloads.

• Dropless training: It trains without dropping tokens and reduces the extent to which teams need to distribute experts across machines compared with traditional 3D parallel approaches.
• Long sequences: Memory optimizations support long training sequences, and DeepSpeed Ulysses integration lets teams extend sequence length through sequence parallelism.
• Uneven expert workloads: The engine handles expert load imbalance during long-sequence training.
• Large-model training: XTuner supports MoE models up to a trillion parameters and uses fully sharded data parallel training, with hardware support that includes the Ascend A3 Supernode and NVIDIA H800.

XTuner is a Python project released under Apache 2.0. It's a training backend for teams with access to substantial accelerator hardware.