Data-Juicer

An open-source AI data processing framework cleans and curates multimodal datasets on local machines or Ray clusters, with CUDA acceleration.

Data-Juicer is a Python framework for researchers and teams preparing datasets for AI training, evaluation, agents or document retrieval. It processes data on your own machine or across a Ray cluster.

Multimodal datasets: Clean, filter, deduplicate and generate synthetic data across text, images, audio and video.
Reusable pipelines: Combine processing steps into reproducible YAML recipes that you can version and share, and add custom operators for your own data requirements.
Agent records: Clean tool interaction traces, organize context, remove identifying information and assess quality, with HTML reports for inspecting problematic examples.
Retrieval preparation: Extract and normalize data, split documents into semantic chunks and profile datasets before using them for retrieval-augmented generation.
Instruction-based refinement: The Juicer model accepts natural-language cleaning, filtering and semantic-tagging requirements and produces structured outputs.

Built-in tracing helps you inspect processing behavior when debugging or auditing a pipeline. For larger datasets, Data-Juicer supports adaptive parallelism and CUDA acceleration, and combines compatible processing stages to reduce overhead.

The framework is open source under Apache 2.0. Local execution and distributed Ray processing are supported; Alibaba Cloud PAI also integrates it into cloud data processing products.