OpenEnv and TRL tutorial: training coding agents

This session explains how reinforcement learning changes when an AI agent can execute code, inspect results and try again. Instead of scoring one answer with a Python function, an environment maintains state across actions and supplies observations and rewards. The speakers separate the task, runtime and grading logic, and recommend evaluating an existing model before training it.

The OpenEnv walkthrough covers reset, step and state interfaces, Docker packaging, and a CLI for creating and sharing environments through the Hugging Face Hub. The presenter describes deployment on a local laptop or Kubernetes as well as hosted services. The demonstrations use Hugging Face infrastructure, so they do not establish a fully offline training workflow.

Two examples connect these ideas to TRL. The first uses Qwen3-1.7B on MBPP Python problems through GRPOTrainer and environment_factory. Its code execution environment uses a CPU without a GPU; that detail does not describe the model's training hardware. The second trains a model inside the OpenCode harness with AsyncGRPOTrainer in isolated sandboxes, scoring the fraction of hidden tests passed. A capture proxy records model calls while the harness controls its own execution loop.

The speakers report improved results in their experiments and stress the need for separate evaluations. They also discuss keeping verifiers outside the agent's sandbox to reduce reward hacking.