TRL

TRL is a Python library for researchers and developers adapting foundation models with their own datasets. It builds on Hugging Face Transformers and supports training on a single GPU or across multiple machines.

• Supervised fine-tuning: Train language models or PEFT adapters on example responses from a custom dataset.
• Preference training: Use DPO with paired preferences or KTO with simpler desirable and undesirable feedback.
• Reinforcement learning: Train with GRPO and RLOO, and build reward models to score model outputs.
• Knowledge distillation: Train a model using knowledge from another model through a dedicated distillation trainer.
• Memory and scaling: Use PEFT with LoRA or QLoRA to reduce training memory needs, and Accelerate with distributed methods such as DeepSpeed for larger workloads.

TRL supports different model architectures and modalities within the Transformers ecosystem. Unsloth integration supplies optimized training kernels. Its Python trainers provide control over custom training workflows, while a command-line interface supports fine-tuning without writing code. The library is open source under Apache 2.0.