NVIDIA Switchyard: routing Nemotron vs DeepSeek

Tim tests NVIDIA Switchyard, an open source router that sits between an AI client and model endpoints. His local AI setup pairs Nemotron 3.5 Lightning on an RTX 3090 with 24 GB of VRAM with DeepSeek V4 Flash across two GX10s. GitHub Copilot sends the coding requests through Switchyard.

He chooses Stage routing, which can switch models as a task progresses. The initial configuration uses Nemotron by default, a confidence threshold of 0.5 and the last three tool results. Early tests produce frequent timeouts. Telemetry reveals that many requests reach DeepSeek because they exceed Nemotron's configured 32K context window, rather than because the router judges the work too difficult.

After increasing context to 128K, Tim reports that DeepSeek requests on one task fall from 22 of 48 to three of 35, with no context fallbacks. He also reports a separate vLLM configuration fix that raises Nemotron's generation speed from roughly 44 to 198 tokens per second.

In a longer unit-test task, almost 88% of routing decisions stay on Nemotron. Tim says the build, lint and tests pass, although his benchmark rejects the run because of a detector bug. Another run stalls. These are results from his setup, and routing cannot rescue a stuck AI agent once it reaches the most capable model configured.