Paddler

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

• Model serving: Run LLMs and vision-language models through the built-in llama.cpp engine on CPUs or GPUs, with support for embeddings, function calling and grammars.
• Load balancing and scaling: Distribute inference requests across machines, add serving capacity dynamically and integrate with autoscaling tools. Request buffering supports scaling from zero hosts.
• Model management: Swap models dynamically and use the web admin panel to adjust chat templates and inference parameters.
• Monitoring and testing: Monitor the serving fleet, test inference in a browser and collect observability metrics.

Paddler is open source under the Apache 2.0 license. It runs on your own infrastructure and packages its serving components in a self-contained binary. Its deployment consists of a load balancer and inference agents, which can run across separate machines.