LiteLLM + Pi Coding Agent: local model router setup

This tutorial builds a local LLM router with LiteLLM and Pi Coding Agent. The presenter connects models running in Ollama and oMLX through one gateway, then assigns requests to different models according to task complexity. The examples include Qwen3.5 4B, Gemma 4, and a configured Qwen3.8 27B model.

The setup uses Docker Desktop, a Docker Compose file for LiteLLM and PostgreSQL, and a YAML configuration containing model endpoints and routing rules. The presenter explains how the container reaches model servers on the host machine, then maps simple, medium, complex, and reasoning tasks to model tiers.

Two classification approaches receive attention. The heuristic classifier scores signals such as prompt length and code patterns. The LLM classifier uses a separate model to judge complexity. In the demonstrated configuration, a Gemma 4 E2B classifier runs through Ollama, with heuristic fallback and Qwen3.5 4B as the default route.

Dashboard prompts and repository questions in the coding assistant demonstrate model selection. The presenter argues that routing can avoid using the largest model for every task, but does not provide a controlled performance comparison. Several loaded models still occupy substantial memory: the demonstration reports about 86 GB in use. Routing alone does not establish a smaller memory requirement.