ART and CoreWeave: RL fine-tuning tradeoffs explained

Kyle Corbitt, founder of OpenPipe and leader of CoreWeave's serverless training team, explains when reinforcement learning fine-tuning may justify its added complexity. He compares RL with supervised fine-tuning, describes GRPO's use of grouped rollouts instead of a separate critic model, and discusses later changes to clipping and length normalization.

For teams customizing an open source model, Corbitt recommends starting with a concrete problem. He says latency often drives customers toward smaller models for voice interactions and agentic search. If frontier models already handle a task well, his advice is to keep using them: customization slows iteration, and he doubts it would be worthwhile for the host's creative writing use case.

The practical discussion covers Ruler and evaluation rubrics. Corbitt recommends checking high- and low-scoring outputs, running short training rounds, and revising the judge before a longer run. His Hacker News title experiment illustrates reward hacking: the model repeated an unsupported headline until he added a judge to check it against the article.

Corbitt says ART can train on users' own local GPUs. CoreWeave's serverless option keeps the environment and dataset on the user's machine while remote GPUs handle training. He discusses latency tradeoffs when serving LoRA adapters, which can motivate merging weights into dedicated deployments.