TRL and Gemma 4 2B: coding agent fine-tuning tutorial

This session builds a supervised fine-tuning baseline for Gemma 4 2B using public coding-agent traces. The presenters use Codex to write the training script, launch parameter sweeps on Hugging Face Jobs, and review results. The traces come from Claude Opus 4.5 sessions in the Pi harness. The goal is to teach a smaller model to imitate tool calls and multi-turn responses.

The tutorial explains prompt/completion formatting and completion-only loss. Masking user tokens excludes them from the loss calculation so training targets the assistant's responses. A separate PyTorch and Transformers example explains the training loop; the speaker describes it as educational code rather than a production implementation. The main workflow uses TRL with LoRA and remote GPUs, so this is not a demonstration of training on local hardware.

Trackio groups runs and displays loss, entropy, and token accuracy. The speaker says its Python tool and web app can also run locally without an account. For anyone developing a coding assistant, the session stresses a limit of these metrics: lower held-out loss measures closer imitation of the traces, not reliable task completion. Inspect AI runs HumanEval and MBPP evaluations, but the presenters caution that these benchmarks may not match a specific use case. Reward-based training and RL environments are left for later sessions.