Mosec

Mosec is a self-hosted model serving framework for developers turning trained machine learning models into APIs. It runs on Linux and macOS, on your own hardware or cloud servers, and is open source under the Apache 2.0 license. Its main focus is handling concurrent requests while making better use of CPU and GPU resources.

Dynamic batching groups requests from separate clients into a single inference batch, then returns each result to its caller. Multi-stage pipelines let CPU preprocessing, GPU inference and other work run in separate processes. That helps keep a GPU supplied with work when a service mixes model inference with slower preparation or input/output tasks.

You write model handlers in Python. Mosec doesn't require a particular ML framework, so developers can reuse inference code they already test outside a server. Rust handles the web layer and task coordination with asynchronous I/O. Examples cover PyTorch and JAX, Stable Diffusion image generation, passage reranking and an OpenAI-compatible embedding service.

A single service can expose multiple models through separate routes. Request validation can generate OpenAPI documentation, while shared-memory communication supports passing data between pipeline stages. For server deployments, Mosec includes model warmup, graceful shutdown and Prometheus metrics, and works with Kubernetes or other container orchestration systems. GPU allocation controls let separate model replicas use different GPUs.