GPT NeoX

An open-source LLM training framework that distributes work across GPU clusters and imports Llama, CodeLlama and Mistral checkpoints.

GPT-NeoX is EleutherAI's framework for researchers and teams training language models with billions of parameters on their own GPU infrastructure. It focuses on training from scratch, with support for fine-tuning existing checkpoints.

Distributed training: Spread training across GPUs with ZeRO and 3D parallelism, using a framework based on NVIDIA Megatron and DeepSpeed.
Model compatibility: Use predefined architectures including Pythia, PaLM, Falcon and LLaMA, import Llama 1, Llama 2, CodeLlama and Mistral-7b-v0.1 checkpoints, and export models to Hugging Face formats.
Training experiments: Support curriculum learning, use Hugging Face tokenizers, evaluate models with EleutherAI's Language Model Evaluation Harness, and track experiments through WandB, Comet or TensorBoard.
GPU optimizations: Use flash attention and optional Transformer Engine support, which provides optimized kernels for NVIDIA A100 and H100 GPUs.

GPT-NeoX is open source under Apache 2.0. It supports GPU clusters and supercomputers, with Slurm, MPI and IBM Job Step Manager compatibility, and has also run on AWS and CoreWeave infrastructure. Its focus is large-scale model training; EleutherAI recommends Hugging Face transformers for general inference with GPT-NeoX models.

LLM Fine-Tuning Training Frameworks Distributed execution Hugging Face integration