RamaLama

An open-source local LLM runner serves models with llama.cpp or vLLM, using Podman or Docker containers by default and CPU or GPU inference.

RamaLama runs AI models for local chat or a self-hosted REST API. It's for developers who want a container workflow for trying models on their own hardware and serving them in production.

Model sources and packaging: Use models from Hugging Face, ModelScope, Ollama or OCI registries such as Docker Hub and Quay, package models as OCI images, and convert Safetensors to quantized GGUF.
Inference backends: Run models with llama.cpp or vLLM; Apple Silicon Macs can also use MLX on the host, outside containers.
Container isolation: Models run in rootless Podman or Docker containers by default, with read-only model mounts; local chat containers have networking disabled.
Model comparison: Inspect model details, benchmark inference performance and measure perplexity to assess how well a model predicts text.

RamaLama is open source under the MIT license. It runs on Linux and macOS, and on Windows through WSL2 with Docker or Podman. It detects supported GPUs and selects a matching runtime image, falling back to the CPU when needed. Hardware support includes NVIDIA and AMD GPUs and Apple Silicon, plus Intel and Moore Threads GPUs and Ascend NPUs on Linux.

Inference runs on your hardware. Model and runtime downloads use remote registries, and temporary data written inside the default containers is deleted when the session ends. MLX requires macOS and Apple Silicon and doesn't support container execution.