Seldon Core

Seldon Core is an AI model serving framework for teams running machine learning models and LLM applications on Kubernetes. It manages individual models and applications with multiple processing stages.

• Shared model serving: Host multiple models on shared inference servers, with memory overcommit for model collections whose combined requirements exceed available memory.
• AI pipelines: Connect application components through Kafka data streams and add custom logic, LLMs, drift detection or outlier detection.
• Autoscaling: Scale models and application components using built-in or custom logic.
• Model experiments: Route traffic between candidate models or pipelines for A/B tests, or evaluate candidates through shadow deployments.

Seldon Core can run on your own infrastructure or in a cloud Kubernetes environment. Pipelines use Kafka to carry data between components.