RouteLLM overview: GPT-4 routing costs and comparisons

RouteLLM is an open source framework for deciding which model should answer a prompt. This overview, uploaded July 8, 2024, explains its preference-data approach and reported evaluations. It does not demonstrate installation or specify a software release version.

The speaker describes a two-model setup: a cheaper model handles suitable queries, while a stronger model handles the rest. Training examples pair prompts with judgments about which model gave the better response, including ties. The presentation covers four router approaches and reports that adding data from an LLM judge improved their results against random routing.

For GPT-4 and Mixtral 8x7B, the speaker cites cost reductions above 85% on MT-Bench, 45% on MMLU and 35% on GSM8K while retaining 95% of GPT-4 performance. These are reported benchmark results, not guaranteed savings for every workload.

A further comparison uses Claude 3 Opus and Llama 3 8B with the same routers, without retraining. The speaker proposes combining a local LLM with cloud models, but presents this as a possible architecture rather than a tested local deployment. A separate sponsored segment introduces Langtrace for tracing and evaluating LLM applications.