Argilla

Argilla is an open-source data annotation and feedback tool for AI engineers and domain experts who build training and evaluation datasets. You can run your own Argilla server or deploy it on Hugging Face Spaces. It's licensed under Apache 2.0.

The work centers on reviewing data: experts label examples and contribute feedback, while AI suggestions help with annotation. Filters and semantic search help reviewers find relevant records and focus their attention on data that needs checking. A shared interface gives domain experts a place to contribute their judgment alongside the engineers developing the models.

Argilla supports text classification and named entity recognition, as well as LLM projects involving RAG, preference tuning and reinforcement learning from human feedback (RLHF). Teams can also collect feedback for multimodal projects such as text-to-image models. Its Python SDK connects dataset work to programmatic workflows for ongoing evaluation and model improvement.

Human review can work alongside AI-generated feedback. For example, teams have used Argilla with distilabel to curate training data, and its filters helped identify a data generation bug in the UltraFeedback dataset. These uses make it relevant to teams that need to inspect and correct generated datasets before fine-tuning models.

The project is in maintenance mode: maintainers commit to bug fixes and patches, but aren't developing new features.