TensorZero

TensorZero is a self-hosted platform for developers who need to compare and improve models in LLM applications. The project is archived and no longer maintained.

• Model gateway: Connect applications to Ollama, vLLM, SGLang and TGI or cloud providers such as OpenAI and Anthropic through a shared API compatible with the OpenAI SDK.
• Request handling: Use tool calls, structured JSON output, embeddings and image or file inputs, with routing, retries and fallbacks to handle failed requests.
• Response inspection: Review model responses and human feedback in the UI, build datasets from recorded interactions and replay requests against different prompts or models.
• Evaluation and experiments: Compare individual responses or complete workflows with heuristics and LLM judges, then test prompt and model variants through adaptive A/B tests.
• Optimization: Use production metrics and feedback for automated prompt engineering with GEPA, supervised fine-tuning and inference strategies that select examples dynamically.

TensorZero is open source under Apache 2.0. Inference and feedback records live in your own database; requests to cloud models go to the selected provider. An interactive playground supports prompt testing, and OpenTelemetry traces and Prometheus metrics connect it to external monitoring tools.