TRL GRPO tutorial: rewards and coding agent training

This third Training Agents session explains reinforcement learning with GRPO after supervised fine-tuning and distillation. The presenters walk through sampling several completions for one prompt, scoring them with Python reward functions, and calculating advantages relative to the group's rewards. They explain why groups need varied scores to provide a useful learning signal.

The practical examples use TRL's GRPO trainer. A minimal experiment rewards responses near 20 characters long. A coding experiment uses short Python problems with unit tests and compares groups of 2, 8 and 16 generations. The example executes generated code in a process with a timeout; the presenter notes that stronger sandbox isolation would improve it.

A deliberately gameable reward gives credit for generating four Python code blocks rather than solving the problem. The presenters use it to explain why rising training rewards can fail to indicate better test performance. They recommend inspecting generated outputs alongside reward, entropy, completion length and KL curves, and describe KL penalties as an incomplete defense against reward hacking.

The training runs use Hugging Face Jobs, so this is a hosted training walkthrough rather than a local LLM setup. For experiment monitoring, the presenters use Trackio, which they say can also run locally.