BentoML

BentoML is a Python framework for developers who want to turn AI models into services on their own hardware or servers. It handles model serving and application logic for LLMs, image generation, audio, and embeddings.

• Model APIs: Expose custom or fine-tuned models through REST APIs using frameworks and runtimes such as PyTorch, Transformers, JAX, vLLM, SGLang, and TRT-LLM.
• Multi-model workflows: Combine models into pipelines, including retrieval-augmented generation, with custom application logic and task queues.
• Serving performance: Use adaptive batching, model parallelism, and distributed serving to make use of CPU and GPU resources, including spreading large models across multiple GPUs.
• Deployment packaging: Package code, models, and dependencies together and generate Docker images for reproducible deployments across environments.

BentoML is open source under Apache 2.0. You can develop and debug locally, then run services in Docker on infrastructure you control. Example workloads include Llama and DeepSeek for text, Stable Diffusion for images, SentenceTransformers for embeddings, and WhisperX for audio.

The broader Bento platform adds deployment automation, monitoring, access controls, and autoscaling, with deployment on your own cloud, on-premises infrastructure, or Kubernetes. BentoCloud is the hosted alternative: it supplies cloud compute with NVIDIA and AMD GPUs, and workloads deployed there run in the cloud.