DistillKit

DistillKit is a Python toolkit for developers and researchers training smaller language models to learn from larger ones. It addresses the memory and storage demands of distillation, particularly when you want to reuse one teacher's outputs across several student models.

• Live or saved teacher outputs: Train with a teacher running alongside the student, or use previously captured outputs so both models don't need to fit in GPU memory together.
• Compressed training data: Reduce the space needed to store teacher predictions, with control over the tradeoff between storage size and reconstruction accuracy.
• Training objectives: Combine probability matching, ranking losses and hidden state alignment, with support for full-vocabulary predictions or a smaller selection of likely tokens.
• Cross-architecture distillation: Align hidden states between different model architectures and use mergekit-tokensurgeon when teacher and student tokenizers differ.

The toolkit runs on your own training hardware and is open source under Apache License 2.0. It builds on Hugging Face Transformers, TRL and Accelerate. Capturing your own teacher outputs requires vLLM; Arcee also provides captured datasets from Qwen3-235B, DeepSeek V3 and DeepSeek R1.

Here, online distillation means live teacher inference, while offline distillation means training from saved outputs. Live training needs enough VRAM for both models and their prediction data. DistillKit is a research release with best-effort community support.