node-llama-cpp

node-llama-cpp is a library for developers adding local AI to JavaScript and TypeScript applications. It uses llama.cpp to run GGUF models on your hardware and works in Node.js, Bun and Electron.

• Structured output: Enforce a JSON schema during generation and use structured decisions with any model.
• Function calling: Give models access to application functions that retrieve information or perform actions.
• Search and retrieval: Generate embeddings and rerank results for applications that search and compare documents.
• Text generation: Build chat, text completion and fill-in-the-middle features, with support for LoRA adapters.
• Model evaluation: Try conversations in a terminal chat interface or the example Electron app before integrating a model into your application.

The library supports macOS, Linux and Windows, including Apple Silicon and Windows on Arm. Metal, CUDA and Vulkan provide GPU acceleration, and it adapts to the available hardware automatically. Prebuilt native binaries are available for the supported desktop platforms.

TypeScript support helps developers integrate model calls into an existing codebase. Chat templates, including Jinja templates, and context management support longer conversations. The project is open source under the MIT license.