OpenThoughts and Curator: LLM post-training overview

Mahesh Sathiamoorthy of Bespoke Labs explains how his team curates data and reinforcement learning environments for LLM post-training. He argues that data quality is a bottleneck for improving AI agent reliability on long tasks, and treats RL environments as another form of training data.

The talk centers on OpenThoughts, an open source reasoning dataset and research project. Its curation pipeline selects and mixes source questions, generates answers with teacher models, and tests filtering choices through ablations. Sathiamoorthy reports that sampling multiple answers per question helped, with one example using 16 answers. He suggests that varied reasoning traces explain the benefit. Stronger models were not always better teachers, and answer filtering did not consistently help in their experiments.

OpenThoughts Agents extends this work to agent trajectories and environments. The speaker says supervised fine-tuning contributed much of the improvement, while compute-intensive RL helped with the remaining gains. A Credit Karma example describes tagged training data intended to reduce errors involving numerical credit-card offer terms; he reports better compliance, latency and throughput.

Curator supports response generation from dataset prompts or collected logs, with Tinker and Fireworks integrations. The closing discussion covers environment quality, version tracking and sandbox orchestration, including checkpoints and rollback for long rollouts.