NeMo RL and Codex: star-counting experiments and SFT results
Learn how Codex sets up GPU training and builds a star-counting task with NeMo Gym. The presenter reports over 91% accuracy after switching to SFT.
Player not loading? Watch on YouTube
This tutorial follows a Codex coding assistant through an autoresearch campaign using NeMo RL and NeMo Gym. The presenter rents a single NVIDIA H100 with 80 GB of memory through NVIDIA Brev and connects a development environment to the remote machine. This is a cloud GPU demonstration; the hosts also discuss running similar workflows on other hardware.
The first prompt asks Codex to configure NeMo RL and complete a small GRPO smoke test. The presenter says the agent resolves CUDA and Docker issues, saves a YAML training recipe and creates a session handoff. Skills provide instructions for storage, approval checkpoints and preserving session state at least every 30 minutes or after major events. Weights & Biases tracks experiments.
Next, Codex builds a NeMo Gym environment for counting stars of a specified color and returning an answer in a strict format. The presenter reports that GRPO misses the 90% target, with a later comparison putting the best RL checkpoint at about 51% against roughly 30% for the base model. Switching to supervised fine-tuning reportedly exceeds 91% within 1 hour 37 minutes of a five-hour budget. These are results for this task, not a general ranking of training methods.
The discussion covers time limits, human review and separate evaluation sets. The speakers note that vague task definitions can make independent runs difficult to compare.
Design Blog
