Bifrost setup: route and monitor LLM requests

This tutorial sets up Bifrost as a self-hosted proxy between an application and several model providers. The presenter runs it locally with Docker, opens the dashboard on port 8080 and configures OpenAI and Anthropic API keys. Running the proxy locally does not mean the demonstrated models run on the same machine: requests go to provider services and a fine-tuned model deployed on Modal.

The security walkthrough covers admin credentials and optional virtual keys with spending limits. The demonstration leaves virtual-key enforcement disabled; the presenter recommends enabling it when tracking usage and budgets for individual users. Provider settings also restrict which models a key can access. After initially selecting Claude 3.5, the presenter reports that it is no longer supported and switches to Claude Haiku 4.5.

For the custom endpoint, the tutorial uses vLLM's OpenAI-compatible interface and registers the deployed model name, summarizer. A client lists available models and switches between providers. The presenter explains that the LoRA deployment exposes both the base model and its adapter.

The dashboard shows request counts, input and output tokens, costs and logs. Bifrost lacks pricing for the custom model in this setup, so the presenter explains how to supply a custom pricing URL. The logs also expose a slow request that the presenter attributes to a Modal container cold start.