BMW's Ray and vLLM gateway vs Claude API costs

Thomas Riedl explains BMW's AI Gateway architecture and its move from managed APIs to self-hosted inference. The platform routes requests to Azure AI, Amazon Bedrock or BMW's own model servers. Its hosting stack uses KubeRay and vLLM on AWS EKS, with vLLM Omni adding support for diffusion models and multimodal outputs.

The engineering details show the hardware tradeoffs behind the service. Riedl describes serving Qwen 3.6 with 4-bit weights on a 96 GB GPU. MiniMax needs two RTX Pro Blackwell cards, an 8-bit KV cache and a context reduction from 256,000 to 128,000 tokens. GLM 5.2 runs across eight GPUs and required runtime updates and architecture patches in the setup presented.

A coding assistant pilot with fewer than 100 developers compares these models with Claude's Haiku, Sonnet and Opus classes. BMW's cost calculation assumes discounted GPU pricing, hosting for 14 hours each weekday and 150 million tokens per developer per month. Under those assumptions, Riedl places break-even usage at 1.2 billion monthly tokens for Qwen, 1.5 billion for MiniMax and 2.5 billion for GLM. He reports savings of up to 50% for the Opus model class after roughly a month of the pilot.

These are BMW's reported results. Riedl also describes GPU availability constraints, model-specific tuning and the operational work required to serve open source models.