Label Studio
Open source data labeling and AI evaluation platform that runs locally or on your server, with custom annotation interfaces and model-assisted labeling.
Label Studio is a self-hosted platform for teams preparing training data or evaluating AI outputs through human review. It handles text, images, audio, video and time series in the same application, including tasks that combine several data types. The open source edition uses the Apache 2.0 license and runs locally or on your own server, with Docker deployment and browser access. A separate hosted cloud edition runs on the provider's infrastructure.
For LLM and agent evaluation, reviewers can compare answers side by side, apply custom rubrics and benchmarks, and collect preferences, rankings or corrections for RLHF and fine-tuning. It also supports reviewing agent traces through connections to observability tools. RAG evaluation covers retrieval relevance and grading generated answers against their source material.
Its annotation tools cover object detection and segmentation, document OCR, named entity recognition, audio transcription and speaker diarization. Custom layouts and templates let teams adapt the review interface to their data and criteria. Connected machine learning models can supply preliminary labels and predictions for people to check or compare.
Teams can manage multiple datasets as separate projects and associate annotations with user accounts. Data can come from local files, Amazon S3 or Google Cloud Storage, with exports for model training. A REST API, Python SDK and webhooks connect labeling work to training, active learning and evaluation pipelines.
