Tools tagged with "Streaming inference"

Pocket AI Lab runs AI models on an iPhone and helps you judge which ones the phone can handle. It's for people who want private chat on their device and want to check memory and storage before downloading model weights.

RealtimeSTT is a Python library for developers building voice assistants, dictation tools and apps that turn live audio into text. It can show text while someone speaks and produce a final transcript when they finish.

FastRTC is a Python library for developers building voice assistants, webcam AI apps, and other interactive audio or video tools. It connects live media streams to Python functions so developers can focus on how their application responds.

Haystack is a Python framework for developers building AI agents, document search and applications that answer questions using their own data. Its pipelines give teams control over the information a model receives and the steps that produce an answer.

llama-swap is a local AI proxy for people who want several models available to their apps without keeping them all loaded. It starts the server a request needs and replaces the active server when another model is requested.

Pipecat is an open-source Python framework for developers who want to run voice and video agent applications locally or on their own servers. It hosts the conversation pipeline, while speech and model processing depend on the services you choose. Integrations such as Soniox, OpenAI and Cartesia send the relevant requests to those providers.

GitAgent is an AI agent framework for developers who want to version an agent's behavior and saved memory alongside its tools. Agent definitions live in a Git repository, where teams can review changes, fork agents and maintain different behaviors on separate branches.

OlliteRT lets you use an Android phone as a private AI server for apps on your local network. It's for people who want to run models locally for chat clients or home automation without a cloud inference service.

PaddleSpeech is a speech and audio toolkit for developers and researchers building applications on their own machines or servers. Built on PaddlePaddle, it supports pretrained models alongside training and evaluation workflows.

Seamless Communication is Meta's family of AI models for developers and researchers building multilingual speech applications. It covers translation and transcription, with dedicated models for conversations and expressive speech.

OpenHarness SDK is a TypeScript toolkit for developers building coding assistants and other AI agents into their own applications. It gives the application control over conversation history, tool permissions and task delegation.

GenericAgent runs tasks on your computer using cloud models to decide what to do. It's for people who want an AI agent to work across websites, files and desktop apps, then save successful workflows for reuse. Model calls require API keys and an internet connection, with requests going to the selected provider. It supports Claude, Gemini, Kimi and MiniMax.

Kokoro-FastAPI generates speech on your own machine or server using Kokoro-82M. It's for developers adding narration to reading apps, audiobooks or voice assistants through an OpenAI-compatible speech API.

RealtimeTTS is a Python text-to-speech library for developers building voice assistants and other apps that need spoken responses before the full text is ready. It accepts both complete text and incoming LLM output.

LiveKit Agents is a framework for developers building voice assistants, phone agents and applications that combine speech, video and text. Python and Node.js agents interact with users through LiveKit rooms connected to web, mobile or telephone clients.

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Sandbook is an offline text-to-speech reader for iPhone and iPad, for people who want to listen to books and documents without uploading them. It runs Kokoro-82M on the device through Apple MLX, with the speech model and voices included in the app. It needs no account and has no ads or tracking.

Whisper Notes transcribes interviews, lectures and meetings for people who need recordings to stay on their own device. It works without an internet connection or account.

PocketFlow is a Python framework for developers building AI agents and workflows with their own model connections. Its utility examples include an Ollama function for local LLM calls. Developers supply that function rather than using a built-in model wrapper. Cloud API examples send model requests to their providers. FastAPI examples show chat applications with streamed responses and background jobs.

WhisperLiveKit is a self-hosted speech-to-text server for meeting transcripts, live captions and apps that need speech recognition as people talk. It processes audio on your own machine or server.

MLX Omni Server lets developers use local models through both OpenAI-compatible and Anthropic-compatible APIs on an Apple Silicon Mac. It's for applications built around those SDKs that need inference on hardware the developer controls. Apple's MLX framework provides hardware acceleration, and all AI processing stays on the Mac.

Prompty is a prompt format and developer toolkit for building AI applications. It keeps prompt text, model settings, inputs and tool definitions in a portable Markdown file that developers can share across editors and language runtimes.

Conduit is a native mobile AI client for people who want access to their self-hosted models and agents from an iPhone or Android phone. It can connect through Open WebUI or directly to a model endpoint.

NobodyWho Chat is an on-device AI assistant for people who want help with writing, brainstorming, coding and questions without sending their conversations to a hosted model. Language-model inference runs on the phone, and chat works offline without an account.

Unlimited-OCR is an OCR model for developers building document extraction workflows on their own hardware or servers. It handles individual document images and multi-page inputs, including PDFs converted to page images.

LocalVocal turns speech into live captions inside OBS for streamers and people recording video. It processes speech on your computer, so transcription can work offline without sending audio to a cloud service.

Pico MLX Server is a macOS app for people who want to run MLX models locally and use them with an existing AI chat client. It gives MLX Server a graphical interface and exposes local models through an OpenAI-compatible API. The project is open source under the MIT license and requires macOS Sonoma or later.

Hermes Browser Extension puts an existing Hermes Agent in a browser side panel for people who want to ask about the pages they're reading. Page context works through local or remote API connections; Hermes Cloud and dashboard-ticket connections are chat-only.

MOSS TTS generates speech for developers and creators working on narration, dubbing and other long audio content. It supports consistent voices across long passages and can clone a speaker from reference audio without separate training for that voice.

OpenVINO Model Server is a self-hosted inference server for developers and teams serving AI models on their own machines or servers. It uses OpenVINO to run workloads on x86 CPUs, including Xeon, with acceleration for Intel integrated and discrete GPUs and NPUs. It's open source under Apache 2.0.

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

FunASR is a speech-to-text toolkit for developers building transcription services and voice applications. It supports offline recordings and live speech on local hardware or self-hosted servers, with capabilities that depend on the model and runtime.

Vosk is an offline speech-to-text toolkit for developers building voice interfaces or transcription software. It processes speech on the device or server running it, without sending audio to a cloud recognition service.

Fish Speech is a self-hosted text-to-speech system for creators producing narration and developers building voice applications. Its S2 Pro model supports directed speech delivery and dialogue with multiple speakers.