NVIDIA Dynamo and Gcore: distributed inference explained

NVIDIA, Gcore and Orange Business discuss distributed inference for enterprises running models across GPU data centers. The session explains cluster routing and customer-controlled infrastructure rather than walking through a desktop installation.

Gcore describes Everywhere AI as a Kubernetes-native platform for self-hosted inference, with support for air-gapped environments. Its presenter says it can deploy on public clouds, VMware, OpenStack, OpenNebula or bare metal, and connect remote GPU workers to a control plane. A portal, APIs, SDKs and Terraform provide ways to manage deployments.

NVIDIA Dynamo handles inference orchestration, including separate prefill and decode workers. Gcore adds routing across sites based on latency, cache location, endpoint health and utilization. The speakers clarify that this layer shares KV cache metadata, not the cache contents. NVIDIA describes transferring the cache itself between workers as an alpha feature in the main branch at the time of the discussion.

The Gcore portal demo covers dedicated models, API keys and token usage. Orange Business then presents a mock-up for choosing deployments by sovereignty, latency and energy criteria. Its projected capacity and deployment screens illustrate a proposed workflow. The closing discussion covers enterprise coding workloads and industrial maintenance, followed by API and SDK access; it does not demonstrate a CLI.