verl overview: RL design and DeepSeek V3 training

Hongpeng Guo and Ziheng Jiang present verl, an open source reinforcement learning framework for large language models. The talk explains its distributed architecture and recent development work rather than providing an installation walkthrough.

The hybrid controller design puts algorithm logic in a central process while multiple controllers handle distributed execution. The speakers describe support for training backends such as FSDP and Megatron-LM, alongside inference engines vLLM and SGLang.

For AI agent training, they explain why batched generation can leave hardware waiting for the slowest interaction. Async server mode handles samples on separate timelines. A token-in, token-out agent loop connects model generation to evaluation environments, with customizable tool logic and turn limits. Coding sandboxes provide compilation and test feedback over multiple rounds.

The speakers report a working DeepSeek V3 recipe for the 671-billion-parameter model using as few as 96 H100 GPUs. This is a cluster-scale training example. They also describe an ongoing refactor into composable engines for rollout, model operations, weight transfer and data movement, plus the agent loop.

The Q&A adds a limitation: in their experiments, moving rollouts from BF16 to FP8 affected final accuracy more than lowering training precision. Coding agents are described as the most common agent training use case.