Megatron-LM

Megatron-LM is a Python framework for researchers training large language models on NVIDIA GPU infrastructure. It provides reference training scripts for experimenting with models and studying distributed training.

• Research training scripts: Preconfigured examples pair with Megatron Core's transformer components, so researchers can experiment without building an entire training framework.
• Distributed training: Tensor, pipeline, data, expert, and context parallelism let teams spread models and training workloads across GPUs.
• Mixed precision: Support for FP16, BF16, FP8, and FP4 gives teams different precision choices for GPU training.
• Checkpoint conversion: Megatron Bridge converts checkpoints between Hugging Face and Megatron in both directions and provides training recipes.

The framework targets GPU training infrastructure, with large-model benchmarks on NVIDIA H100 clusters. Megatron Core is the separate, composable library within the repository for developers building custom training systems.