NeMo Curator

A self-hosted Python toolkit that filters and deduplicates AI training datasets across text, images, video and audio, with NVIDIA GPU support.

NeMo Curator is an open source Python toolkit for ML engineers and data teams preparing AI training datasets. It provides repeatable curation pipelines that can run on a laptop or across a multi-node Ray cluster.

Text curation: Detect languages, classify content, filter for quality and remove exact, fuzzy or substring duplicates.
Image selection: Generate embeddings, remove duplicates, detect NSFW content and filter images by aesthetics.
Audio datasets: Assess quality, tag audio and identify speakers to prepare speech and audio-language training data.
LLM processing: Run an OpenAI-compatible inference endpoint inside a pipeline for classification and synthetic data generation.
Reusable data recipes: Use Nemotron-CC pipelines for Common Crawl extraction, filtering, deduplication and synthetic data generation.

Curator runs on your own infrastructure and uses NVIDIA RAPIDS with Ray for distributed GPU workloads. It supports Docker, Slurm and Kubernetes deployment. Video and audio pipelines need system codec libraries, which the published Docker container includes.

CPU-only text processing is available. The bundled GPU text example targets Linux x86_64 with CUDA, about 16 GB of GPU memory and internet access to download a Hugging Face model. Those requirements apply to that example rather than every pipeline. Curator is licensed under Apache 2.0.