TRL tutorial: distilling smaller coding agents

Ben and Sergio explain how model distillation transfers task-specific behavior from a teacher to a student, with smaller coding agents as the practical example. They organize the methods around four questions: whether training uses text or logits, who generates the data, whether the teacher participates during training, and whether teacher and student are separate models.

The tutorial uses TRL's experimental GKDTrainer for both off-policy and on-policy distillation. In the off-policy example, lambda and beta are both zero: the student learns from fixed teacher data with forward KL loss. For the on-policy example, both values are one. The student generates completions, and a frozen teacher supplies token distributions for reverse KL loss on those trajectories.

The presenters describe experiments with a 0.6-billion-parameter student and a 4-billion-parameter teacher, using coding traces from Pi. A coding assistant prepares training scripts and compares three learning rates through Hugging Face Jobs. Sergio reports that the illustrative off-policy run used one A100. Falling loss curves demonstrate the training workflow; the session does not establish a coding benchmark improvement.

The final section introduces self-distillation through earlier checkpoints, privileged hints and better samples. The speakers also discuss teacher capability limits and explain why unavailable logits prevent closed teachers from supporting the specific logit-based approach shown.