Edge AI for Jetson and Raspberry Pi

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

TensorFlow Model Optimization Toolkit is an open-source Python library for developers preparing TensorFlow and Keras models for deployment on mobile, IoT and other edge devices, or in the cloud. It helps reduce inference latency and cost, and adapt models to limits on memory, processing power, energy use and storage.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

Moondream is a vision language model for developers building apps that interpret images, such as warehouse or manufacturing footage. Its downloadable models can run on your own hardware, including offline in an air-gapped environment.

OpenVoiceOS is a voice assistant platform for people building smart speakers or adding voice control to their own devices. Its core is a fork of MycroftAI/mycroft-core, and most classic Mycroft skills remain compatible.

py-xiaozhi is a Python AI voice client for people who want to try Xiaozhi conversations on a computer or build voice interaction into hardware projects. It builds on the xiaozhi-esp32 project and lets you use a desktop microphone and speakers without dedicated Xiaozhi hardware.

jetson-stats monitors NVIDIA Jetson boards and controls their power and cooling settings. It's for developers running workloads, including local AI, who need to inspect resource use and adjust how their board operates.

LiteRT Torch is a Python library for converting PyTorch models into .tflite files for the LiteRT runtime. Developers can prepare models on Linux and deploy them into Android, iOS and IoT applications that run inference on the device. Code is licensed under Apache 2.0.

MediaPipe is a toolkit for developers adding on-device AI to applications. It provides pretrained models and task APIs for applications that need to process images, video, text or audio on the user's device.

PicoClaw is a self-hosted personal AI assistant that can use local models through Ollama or vLLM. It's aimed at people who want an agent on their own hardware, including small Linux boards and Android phones. Hosted providers are optional, but choosing OpenAI, Anthropic, Google Gemini or DeepSeek sends model requests to those services and requires internet access and API keys.

MiniCPM is a family of compact language models for developers building local assistants and coding agents. It targets on-device workloads and hardware with limited computing resources.

NVIDIA Cosmos provides world models for developers building robots, autonomous vehicles and industrial vision systems. It supports reasoning about physical scenes, generating training data and learning actions before real-world deployment.

jetson-containers is a Docker container build system for developers running local AI on NVIDIA Jetson. It assembles GPU software and its dependencies into containers for workloads that need a compatible CUDA stack.

Cactus is an on-device AI engine for developers adding automation to mobile apps, wearables and other small devices. Its Needle model turns requests into tool calls locally, including when the device has no internet connection.

ncnn is a C++ inference framework for developers adding on-device AI to mobile, desktop and embedded applications. It focuses on a small memory footprint and doesn't require third-party runtime dependencies.

OlliteRT lets you use an Android phone as a private AI server for apps on your local network. It's for people who want to run models locally for chat clients or home automation without a cloud inference service.

Triton Inference Server, offered by NVIDIA as Dynamo-Triton, is a self-hosted AI inference server for teams deploying models in applications. It brings models from different frameworks into one serving system across on-premises, cloud and edge hardware.

Vosk is an offline speech-to-text toolkit for developers building voice interfaces or transcription software. It processes speech on the device or server running it, without sending audio to a cloud recognition service.

LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.

Gemma is Google DeepMind's family of open-weight AI models for developers building applications on their own hardware. It includes compact models for mobile and IoT devices and larger models for reasoning on personal computers and servers.

PaddleDetection is a computer vision toolkit for developers building image and video analysis systems. It covers training, adapting and deploying models for uses such as inspection and traffic analysis.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

Roboflow Inference runs computer vision models on images and camera feeds for teams building inspection, monitoring and image analysis systems. You can host it on your own computer, server or edge device.

NullClaw runs AI agents on your own hardware or server for coding work and automated background tasks. It's aimed at developers who want an assistant that can carry out multi-step jobs with controls over its access and actions.

Willow is a voice assistant for people who want to control home automation through hardware and speech processing they own. It provides an alternative to Amazon Echo or Google Home on Espressif's ESP32-S3-BOX hardware family.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

Needle is an AI model for developers building device assistants that turn requests into actions. It prioritizes automation over general chat, with smaller model variants for devices that have limited resources.

Max Headbox is a local AI agent for people who want a voice-controlled assistant that can carry out actions on a Raspberry Pi. It processes voice and runs models on your own hardware without relying on cloud AI providers.

Apache TVM is a machine learning compiler for developers optimizing models for deployment on their own hardware. It compiles model computations into modules that applications can embed, with targets spanning data center GPUs and edge devices. Code is licensed under Apache 2.0.

taOS is a self-hosted AI agent platform for people who want to run assistants on their own computers and keep their conversations, files, and memory there. It brings agent management and local model access into a browser-based desktop.

Ultralytics YOLO is a Python computer vision library for developers building applications that interpret images and video. Models can run on hardware you control for tasks such as visual inspection, inventory monitoring and robotics perception.

Wyoming Satellite connects a microphone and audio playback device to Home Assistant for a self-hosted voice assistant. The project is archived and no longer maintained; Linux Voice Assistant replaces it using the ESPHome protocol.