DataFlow
An open-source LLM data preparation system that generates, cleans, and evaluates datasets with reusable pipelines and a visual workspace.
DataFlow is a Python system for researchers and teams preparing data for LLM training or RAG. It turns noisy PDFs, plain text, and question-answer datasets into data suited to a specific domain or training task.
- Data generation and cleaning: Generate text, math, and code datasets, then evaluate, filter, and refine the results for pre-training, supervised fine-tuning, or reinforcement learning.
- Document extraction: Convert PDFs into question-answer datasets, including visual question-answer pairs from book PDFs.
- Reasoning and SQL datasets: Add reasoning, categories, and difficulty estimates to existing question-answer pairs, or build Text2SQL data with queries, explanations, and schema context.
- Reusable workflows: Combine existing processing operators with custom ones, swap underlying language models, and reuse pipelines to compare data preparation strategies.
- Visual pipeline editing: The companion DataFlow-WebUI provides drag-and-drop editing and AI agent assistance for building, managing, and running workflows through a visual workspace and natural-language interaction.
DataFlow is open source under Apache 2.0. Its operators accept JSON, JSONL, and CSV data, and its Python and Git foundations support sharing custom operators and tracking pipeline changes.
Logotyper
