OlliteRT

OlliteRT lets you use an Android phone as a private AI server for apps on your local network. It's for people who want to run models locally for chat clients or home automation without a cloud inference service.

• Client compatibility: Connect Open WebUI, OpenClaw and other clients through OpenAI-compatible APIs or the Anthropic Messages API.
• Text, vision and audio: Use supported models for text generation, image understanding, audio transcription and reasoning, with streaming responses.
• Model choice and benchmarks: Download models from Hugging Face or import .litertlm files, then compare their performance on your phone. Choices include Gemma 4, Gemma 3n, Gemma 3, Qwen 2.5 and DeepSeek-R1.
• Monitoring and home automation: View live device and server statistics, export Prometheus metrics to Grafana, and control the server through a Home Assistant REST API. Home Assistant conversation and voice use require custom integrations.

Inference runs on the phone's CPU or GPU through Google's LiteRT-LM runtime. Prompts stay on your hardware, and the app collects no telemetry or analytics. It requires Android 12 or later, an arm64-v8a device and at least 6 GB of RAM; multimodal models need 8 GB or more.

OlliteRT is free and open source under Apache License 2.0 and builds on Google AI Edge Gallery. It supports .litertlm models rather than GGUF, loads one model at a time and processes requests sequentially.