Tools tagged with "Streaming inference"

FreeLLMAPI gives developers one API for experimenting with cloud models and local backends. The router runs locally, but cloud inference still sends requests to the providers you enable.

Free Claude Code lets developers use coding agents with local models or a choice of cloud providers. It connects existing clients to a shared model catalog rather than requiring a separate coding workflow.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

AIClient2API is a self-hosted AI API proxy for developers connecting chat apps and coding assistants to multiple model providers. It exposes services such as Antigravity, Codex, Grok, and Kiro through interfaces those apps already support.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

VoxInput is an open-source dictation tool for Linux and macOS that types spoken words into the active application. It's for people who want speech input across their desktop, including applications without their own dictation feature. On Linux, it also works in the system console.

Coqui TTS is a text-to-speech toolkit for developers and speech researchers who want to generate voices locally or train their own models. The idiap fork builds on coqui-ai/TTS.

CosyVoice is a text-to-speech system for developers building speech applications on their own hardware or self-hosted servers. It can generate speech in a reference speaker's voice, including in another language, without training a separate model for each speaker.

Fisper is a voice-to-text app for Apple Silicon Macs that puts dictated text into the app you're using. It runs Whisper locally and works offline after the initial model download. It's for people who want to dictate in everyday apps or talk to terminal-based AI agents without sending their audio to a cloud service.

Tellama is a local LLM chat app for Android users who want conversations to stay on their phone. Chat with an installed compatible model works offline, and you don't need an account.

OpenZiti LLM Gateway is a self-hosted API proxy for developers and teams connecting AI applications to inference servers across private networks and cloud services. Its optional zrok integration uses OpenZiti for identity-based access and end-to-end encryption, so clients and backends can connect across network boundaries without opening inbound ports.

Agently is a Python AI agent framework for developers building assistants and service backends with local Ollama models or hosted model APIs. It runs within your application, without requiring a separate orchestration service. Model requests go to the endpoint you choose, so using a remote provider sends the request data outside your machine.

Anything Analyzer is a desktop protocol analyzer for developers and security researchers investigating APIs, authentication and request signing. It brings browser and app traffic into one session for inspection and AI analysis.

JesType is a paid dictation app for people who want to write emails, documents, and other text by speaking while keeping their audio on their own computer. It runs on Apple Silicon Macs with macOS 13.4 or later and 64-bit Windows PCs running Windows 10 or later. Intel Macs aren't supported.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

Transcribe Offline turns recordings and live speech into text for people who need to review conversations without uploading audio to a cloud service. It pairs transcript editing with audio playback in a desktop app.

whisper.cpp runs OpenAI's Whisper speech recognition models on your own hardware. It's for developers adding transcription to applications and people who need offline speech-to-text.

PocketPal AI runs language models on phones and tablets for people who want an assistant they can use without sending conversations to a server. Core chat works offline after a model download and doesn't require an account.

LM Studio runs downloaded language models for conversations and tasks on your computer. Its Bionic agent helps people create documents, write software and carry out work using local or frontier open models.

SpeakoFlow lets you dictate into other apps and ask an AI assistant for help without leaving your work. It's for people who want voice typing, spoken conversations and meeting notes with processing on their own computer.

opencodex is a local AI gateway for developers who want to choose which models power their coding assistants. It connects Codex, Claude Code, Claude Desktop and Grok Build to alternative providers while keeping each tool's native interface.

RAG Web UI is a self-hosted app for asking questions about your own documents. It's a learning project, and its maintainers advise against commercial use and state that it isn't ready for production.

audio.cpp runs audio models locally for developers building speech tools and people generating or processing audio on their own hardware. It brings multiple audio tasks into one C++ runtime built on ggml.

MOSS TTSD generates spoken dialogue from scripts for podcasts, audio dramas, dubbing and commentary. Its focus is maintaining distinct voices across long conversations, including speaker changes and overlapping speech.

Phantasm is an iPhone AI chat app for people who want to use models running on their own machines. The phone provides the chat interface, while a backend you control handles inference.

Sokuji translates speech for people taking part in meetings across languages. Other participants hear your translated voice as ordinary call audio, so they don't need to install anything.

parrot.nvim is an AI coding and text editing assistant for Neovim users who want to request changes and chat without leaving their editor. It builds on an early fork of gp.nvim and lets you choose what context goes to the model.

Argmax OSS Swift is a speech AI SDK for developers building iOS and macOS apps that process audio on the user's device. Its separate libraries cover transcription, identifying speakers and generating spoken audio.

MCP Agent is a Python framework for developers building AI agents that use tools through Model Context Protocol (MCP). It handles connections to MCP servers and provides reusable workflows for coordinating agents across a task.

DeepSeek-OCR is an OCR model for developers processing document images and PDFs on their own hardware. It also supports research into visual text compression, representing text-bearing images with compact sets of vision tokens.

Ollama JavaScript is a library for developers adding model chat, generation or retrieval features to JavaScript applications. It connects those applications to an Ollama server, including one running locally.

Miko is an AI companion for people who want to talk with an animated character on their Windows desktop. It pairs a local language model with a customizable 3D avatar and spoken replies.

VibeVoice is a family of voice AI models for developers and researchers building transcription and speech generation applications. It can capture who spoke and when, or produce spoken responses from incoming text.

Claw Admin is a web dashboard for people managing OpenClaw Gateway and Hermes Agent. It puts agent conversations, behavior controls and activity monitoring in one browser interface.