TensorZero with Ollama: setup and model routing tutorial

Uploaded on June 26, 2025, this tutorial demonstrates a self-hosted TensorZero gateway on Ubuntu. The local model is Qwen 3 with 1.7 billion parameters through Ollama; the other provider is OpenAI. The source does not specify TensorZero, Ollama or Docker versions, so the setup describes the software available at the time of upload.

The presenter installs Ollama, downloads the model and sets an OpenAI API key for the paid API option. Docker is a prerequisite. He then prepares a working directory, downloads a Docker Compose file and creates a TOML configuration with two model variants. Each has a weight of 0.5, intended to split traffic evenly between the providers. His demonstration uses an NVIDIA H100 with 80 GB of memory, which he describes as excessive for this example rather than a minimum requirement.

After starting the containers, he opens the interface on localhost port 4000 and sends API requests for a technology haiku. The dashboard records both variants and exposes input and output token counts alongside inference timing. He also discusses session records, custom functions and basic supervised fine-tuning support.

The presenter reports extra latency with the Ollama route and suggests container-to-host networking as a possible cause. He does not establish that explanation or confirm the routing algorithm. The example combines a local LLM with a cloud API, so it does not demonstrate a wholly offline workflow.