wllama

A local LLM library runs GGUF models in the browser through llama.cpp, with CPU or WebGPU inference and an OpenAI-compatible API.

wllama lets developers add local LLM inference to browser apps without a separate inference server. It brings llama.cpp to JavaScript and TypeScript projects through WebAssembly, so models run on the user's device.

Chat and embeddings: Generate text and chat responses, or create embeddings for comparing text, through a built-in, typed OpenAI-compatible API.
Image and audio input: Use models that accept image and audio files, alongside support for tool calling.
CPU and GPU inference: Run models without a GPU or use WebGPU acceleration, with partial GPU offloading when a model won't fit entirely in video memory.
GGUF model loading: Load models from Hugging Face or other URLs, including split GGUF files that download in parallel.
Responsive browser apps: Model inference runs in a worker so it doesn't block interface rendering, and the library selects single-thread or multithread execution based on browser support.

The library is free and open source under the MIT license, with no runtime dependencies. Inference happens locally; loading model files from Hugging Face or another remote host involves a download. Its OpenAI-compatible API doesn't require an OpenAI inference service.

Browser support affects GPU acceleration and multithreading. Multithread execution requires cross-origin isolation headers from the host. Individual model files have a 2 GB limit, so larger models must use split files.