Distilabel

Distilabel is a Python framework for engineers building synthetic datasets and model-generated feedback with local LLM backends such as Ollama, llama.cpp and Hugging Face Transformers. It also connects to cloud services, including OpenAI and Anthropic; requests through those integrations go to the remote provider. Where inference runs depends on the backend you choose.

Pipelines combine data generation with model judging for tasks such as classification, extraction, instruction following and dialogue. They can generate examples for fine-tuning or add AI feedback to existing data. The framework includes methods based on published research for generating and assessing datasets, including preference data used in model training.

Local model integrations also include vLLM and MLX. A shared API gives pipelines a common way to work with different model providers, so generation and feedback stages can use the backend that suits the task.

Outlines and Instructor integrations support structured generation when examples need a defined shape. Dataset processing includes text clustering through UMAP and Scikit-learn, sentence embeddings through sentence-transformers, and duplicate detection with MinHash.

Ray support distributes pipeline work across resources, and the framework supports fault tolerance for larger jobs. Generated datasets can be exported to Argilla for review. Distilabel is open source under Apache 2.0, and community collaborators maintain the project.