DataTrove

An open-source Python library that filters and deduplicates LLM training data locally or on Slurm and Ray clusters, with reusable processing blocks.

DataTrove is a Python library for teams preparing large text datasets, including LLM training corpora. It combines reusable processing blocks with custom cleanup logic, so teams can apply their own rules across large workloads.

Text preparation covers reading and writing datasets, extracting text from HTML, filtering documents, tokenization and deduplication.
Custom processing lets you add your own functionality alongside the supplied blocks instead of fitting every dataset into a fixed cleanup process.
Local and cluster execution runs the same processing steps on a local machine, across multiple machines, or on Slurm and Ray clusters, with parallel execution across CPU workers.
Job recovery and statistics tracks completed tasks so interrupted jobs can resume unfinished work, and collects statistics about the dataset during processing.

DataTrove is open source under Apache 2.0. It supports local and remote storage through fsspec, so processing can use files on your own hardware or access an external storage system. Its relatively low memory use and staged processing suit large datasets.

Parallel processing works across input files. DataTrove doesn't automatically divide a single file among tasks, so one large input file limits how much of that work can run in parallel.