WebLLM
An open-source local LLM engine that runs models in the browser with WebGPU acceleration, an OpenAI-compatible API and custom MLC models.
WebLLM is an engine for developers building browser-based AI assistants and Chrome extensions. It runs language models on the user's device, so applications don't need a separate inference server to process prompts.
- OpenAI-compatible interface: Connect applications through familiar API calls and display responses as the model generates them with streaming output.
- Structured results: Generate JSON that follows a custom schema for applications that need machine-readable answers.
- Model choice: Run supported Llama, Phi, Gemma, RedPajama, Mistral and Qwen models, or use custom models in MLC format.
- Browser integration: Build your own interface or Chrome extension, with Web Worker and Service Worker support to keep model computation separate from the interface.
WebLLM is a TypeScript package and a companion project to MLC LLM. It is open source under Apache 2.0. Inference runs inside the browser using WebGPU acceleration, which requires a browser environment that supports WebGPU.
Browser storage can cache model files for reuse. Optional integrity checks verify downloaded configuration, WebAssembly and tokenizer files against supplied hashes before loading them.
Logotyper
