picoLLM

picoLLM is an on-device LLM inference SDK for developers adding text generation to their apps. Inference works offline, but a Picovoice account and AccessKey are required, and key validation needs an internet connection.

• Local text generation: Process prompts on the user's hardware without sending them to a cloud inference service.
• Compressed models: Run models using Picovoice's compression method, which learns how to allocate precision across model weights instead of using a fixed allocation.
• Model choices: Use Gemma, Llama 2, Llama 3, Llama 3.2, Mistral, Mixtral and Phi, with base, instruction and chat variants available.
• App integration: Build with Python, C, .NET and Node.js SDKs, plus Android, iOS and web integrations.

The engine runs on CPU or GPU across Linux, macOS and Windows, including Apple Silicon and Intel Macs and Windows on ARM. It also supports Raspberry Pi 4 and 5, Android and iOS. Browser execution works in Chrome, Safari, Edge and Firefox.

picoLLM is free for open-weight models, and its SDK repository uses the Apache 2.0 license. Model downloads come through Picovoice Console. Local inference still requires contact with Picovoice's license servers to validate the AccessKey.