Ray and vLLM: scaling synthetic data at DatologyAI

Bogdan Gaza explains the infrastructure behind a DatologyAI run that he says generated about 12 trillion synthetic tokens across web, math and code. The talk covers large GPU clusters and training data production, rather than a desktop local LLM setup.

DatologyAI's BeyondWeb recipe starts with selected documents and rephrases or restructures them. Gaza argues that these source documents provide more useful seeds than asking a model to generate data from scratch. He presents benchmark results as evidence for the recipe's efficiency; those results concern the comparisons described in the talk.

The engineering discussion follows a move from separate Slurm research jobs and Kubernetes production pipelines to a shared workflow using Ray, KubeRay and vLLM. DatologyAI runs its H100 capacity on AWS EKS on HyperPod, while its broader pipeline targets Kubernetes environments and S3-compatible storage.

Four operational problems drive the practical advice. Batching S3 metadata requests reduced estimated processing time from 9–11 days to about two hours. Smaller partitions and checkpointing limited lost work after GPU failures. Dedicated resource pools helped coordinate CPU and GPU scheduling across clusters. Gaza also reports about 40% more throughput from inference parameter sweeps, including batch size and speculative decoding settings. These gains describe DatologyAI's workload, not guaranteed results for other deployments.