nanochat

nanochat is a toolkit for researchers and developers who want to train an LLM and inspect the pipeline themselves. Its small Python codebase covers the stages between preparing a tokenizer and chatting with the resulting model.

• Training pipeline: Train a tokenizer, pretrain and finetune a model, then evaluate it and run inference.
• Model scaling: Choose transformer depth and let the toolkit calculate related model dimensions and training settings for experiments at different scales.
• Research comparisons: Study scaling laws and compare training time against GPT-2 capability using the DCLM CORE benchmark.
• Chat interface: Talk to your trained model through a command-line interface.

The project is open source under the MIT license and targets a single GPU node, on your own hardware or a rented server. Its reference training setup uses NVIDIA H100 GPUs; A100 GPUs are also supported. CPU and Apple Silicon examples train much smaller models and produce limited results. Research runs can use Weights & Biases to track training metrics.