Nanotron

Nanotron is a Python library for researchers and developers training language models on their own datasets and GPU infrastructure. It supports training on one machine or across multiple servers.

• Distributed training: Data, tensor and pipeline parallelism spread training across GPUs, with expert parallelism for mixture-of-experts models.
• Custom datasets: Custom data loaders and Datatrove integration let you connect your own data pipelines, while DoReMi examples explore faster training.
• Model examples: Llama and Mamba examples provide starting points, and Spectral µTransfer supports scaling up neural networks.
• Large model training: ZeRO-1, parameter sharding and custom module checkpointing support larger training jobs, alongside FP32 gradient accumulation.
• Checkpoint generation: You can generate text from saved checkpoints and distribute generation across multiple GPUs.

Nanotron runs on NVIDIA CUDA GPUs and uses PyTorch. It supports multi-node training with Slurm and includes CUDA event timing to measure GPU performance. Explicit tensor and pipeline parallelism APIs help developers inspect and debug distributed training. The library is open source under Apache 2.0.