TensorZero
Self-hosted LLM gateway with evaluation, A/B testing, and Ollama support. Open source under Apache 2.0; archived and no longer maintained.
TensorZero is a self-hosted platform for developers building LLM applications. The project is archived and no longer maintained. It combines a model gateway with tools for inspecting responses, evaluating workflows, and improving prompts using production data and human feedback.
The gateway connects to local backends such as Ollama, vLLM, SGLang, and TGI, as well as cloud providers including OpenAI, Anthropic, and AWS Bedrock. It supports OpenAI-compatible APIs and works with the OpenAI SDK. Inference and feedback records stay in your own database; calls to cloud models go to the selected provider.
Its Rust gateway supports tool use, structured JSON output, embeddings, batch requests, and image or file inputs. Routing, retries, fallbacks, and load balancing help applications handle provider failures. You can also track usage and costs and give clients model access without sharing provider API keys.
Evaluation covers individual responses and complete workflows, using heuristics or LLM judges. You can replay past requests against different prompts or models, build datasets from recorded interactions, and compare variants through adaptive A/B tests. Optimization includes automated prompt engineering with GEPA, supervised fine-tuning, and inference strategies that use examples selected dynamically.
TensorZero is open source under Apache 2.0. Its UI includes an interactive prompt playground, and it exports OpenTelemetry traces and Prometheus metrics to external monitoring tools.
