Nemotron 3 Nano: RL training with Prime Intellect

This session examines hosted reinforcement learning for Nemotron 3 Nano in Prime Intellect Lab. Eli, an applied researcher at Prime, demonstrates software engineering task evaluations and explains how task sets, harnesses and runtimes fit together in Verifiers v1. The evaluation examples switch harnesses and use Prime sandboxes; training runs through a configuration file and the prime train command.

The most concrete example concerns invalid tool calls. Eli says an early run showed limited progress because Nano attempted calls outside its bash-and-edit harness. He added a -0.2 penalty for rollouts containing an invalid call. A successful task could therefore receive 0.8 rather than 1. He reports that later rollouts stopped showing this behavior. The rollout viewer provides context for interpreting the rising reward curve beyond the score alone.

The discussion also covers LoRA adapters, shared model infrastructure and rubric-based agentic judges. Eli treats bias from using a judge in the same model family as an open question and notes that judges can suffer reward hacking.

For readers considering their own servers, the deployment distinction matters: Lab uses Prime's managed compute, while Eli describes Prime RL as open source and available to run on users' compute. The session does not establish support for a cluster without outbound internet. Training times are examples, not guarantees; Eli puts the final demonstration run at roughly a day.