Seldon Core

Seldon Core 2 is an AI model serving framework for teams running production machine learning and LLM applications on Kubernetes. It can run on your own infrastructure or in a cloud environment. Its focus is managing individual models and connected applications within the same deployment system.

Models can share inference servers, reducing the need for separate infrastructure for each model. Memory overcommit lets teams deploy a collection of models whose combined memory requirements exceed available memory, rather than reserve capacity for every unused model. Autoscaling applies to both models and application components, with built-in or custom scaling logic.

For applications with multiple stages, Seldon Core supports composable pipelines that use Kafka to stream data between components. Custom components can add application logic, LLMs, drift detection or outlier detection to those pipelines. This makes it relevant to teams whose serving needs extend beyond a single prediction endpoint.

It also supports model comparisons. A/B tests route traffic between candidate models or pipelines, while shadow deployments let teams evaluate candidates alongside an existing deployment.

Seldon Core 2 uses the Business Source License 1.1. It permits non-production use and a limited production grant for non-profit educational institutions; other production use needs commercial terms.