Tools tagged with "Streaming inference"

ESPnet is a speech AI toolkit for researchers and developers who want to run pretrained models on their own hardware or train speech systems. Its recipes cover data preparation through evaluation, with published results for comparison.

Orpheus TTS generates speech for developers building voice applications or creating custom voices. Its Llama-based models support local inference and let you control how spoken text sounds.

Marbor is a self-hosted AI inference gateway for operators serving models across their own GPU fleet. It places requests using live hardware and model state to reduce cold loads and avoid deployments that can't serve the requested model.

FluidVoice turns speech into text in Mac apps, including email, documents, chat, terminals and code editors. It suits people who want local dictation with optional editing before text reaches the active app.

TypeWhisper is a macOS dictation and transcription app for people who want to turn speech into text across their apps. Local speech models keep audio on your Mac; optional cloud transcription sends it to the selected provider.

Agent Service Toolkit is a Python template for developers building their own AI agent services. It brings together agent logic, an API and a browser chat interface so you don't have to build each part from scratch.

ARIA turns audio playing on a Windows PC into live subtitles for videos, games, browser playback and calls. It's for people who need captions across apps or want translations alongside the original speech.

WebLLM is an engine for developers building browser-based AI assistants and Chrome extensions. It runs language models on the user's device, so applications don't need a separate inference server to process prompts.

ChatTTS Colab turns text into speech for people producing narration or scripts with different character voices. It builds on ChatTTS and provides a browser interface for comparing voices and generating long audio passages.

LLM Router is a self-hosted AI gateway for teams connecting applications to local LLM servers and cloud providers. It puts a shared interface in front of your models so clients can use the same gateway across backends.

Prismos AI is a desktop assistant for people working with private documents and technical files. It uses local Ollama models to answer questions and retain knowledge for later conversations.

Screenpipe records computer activity so people can revisit past work and give AI agents context from screens and conversations. Capture history stays on your device by default.

Claudeck is a self-hosted browser workspace for developers who want to manage Claude Code conversations and coding tasks together. It combines ongoing chats with agent workflows and repository tools.

Front Porch AI is a desktop app for character chat and roleplay with local LLMs. It's for people who want to keep their conversations on their own machine while building stories with characters that retain memories and changing emotional states.

Agent Squad is a framework for developers building assistants that divide work among specialized AI agents. It handles agent selection and conversation context so users can switch topics without starting a separate chat.

llmcord is a self-hosted Discord bot for people who want to share AI conversations with friends or a community. It connects Discord chats to local models or remote providers. Conversation history stays in Discord, even when the model runs on your own hardware.

Vercel AI SDK is a free TypeScript library for developers building apps and agents that use local models through Ollama or LM Studio. It runs within your application; those backends run the models. Its default model route uses Vercel AI Gateway, which sends requests to a cloud service, while the documented local integrations connect to your own model server.

Personal LLM runs AI chat on your phone for people who want to ask questions without sending them to a cloud model. Chat, photo analysis and document questions work offline after you download a model and any required image support files.

WhisperLive turns speech into text as audio arrives, using OpenAI's Whisper models. It's for developers and teams who want to run transcription on their own machine or server and connect it to live audio or recorded files.

Yap is a voice dictation app for people who prefer speaking to typing emails, messages or prompts on their Mac. It transcribes speech on-device and inserts the result into the text field you're using.

CleanStream filters unwanted speech in live OBS audio for streamers and people recording spoken content. It runs speech recognition on your machine with Whisper through whisper.cpp, and introduces a playback delay while it decides which audio to keep.

Parrot is an open-source desktop app that reads selected text aloud in other applications. It's for people who prefer listening to written material or need text-to-speech for accessibility. It runs on macOS, Windows and Linux, and builds on Handy by CJ Pais as an MIT-licensed fork.

Blades is an AI agent framework for Go developers building conversational applications and automated workflows. It lets developers combine model connections, application tools and conversation memory within their own software.

Tel-Agent is an AI receptionist for businesses that want phone calls and customer messages handled by one agent on their own server. Recordings and transcripts stay on your machine, but its voice setup uses Deepgram, a cloud LLM and ElevenLabs, requiring internet access and your own API keys.

Talk DAT! is a free, open-source dictation app for people who want to turn speech into formatted writing across their desktop apps. It runs on Windows 10 and 11 and Apple silicon Macs with macOS 14 or later. Downloaded models let it work offline without an account, keeping speech and text on your computer.

OllamaSharp is a C# library for developers building .NET applications with Ollama. It gives those applications access to chat, embeddings and model management through the Ollama API.

Dictate Keyboard is a voice keyboard for people who prefer speaking to typing in Android apps. It builds on FlorisBoard and lets you choose where speech recognition happens.

Aiursoft Ollama Gateway gives teams controlled access to shared local LLM servers. It connects AI applications and coding assistants to Ollama or OpenAI-compatible backends through a central gateway.

Rig is a Rust library for developers building AI agents and LLM applications. It provides a shared interface across model providers and vector stores for applications that need to work with different backends.

icefall is a speech model training toolkit for developers and researchers building recognition systems on their own hardware. Its Python recipes use k2 and lhotse to train, fine-tune, and evaluate models across speech datasets.

Thuki is a floating AI assistant for Mac users who want help with the text or screen they're already viewing. It appears above other apps, including fullscreen windows, and automatically brings selected text into the conversation.

Deltafin runs full Kimi K3 on a single device for researchers and developers exploring large models on their own hardware. It preserves the original expert weights and complete expert bank, but responses are slow and long conversations take substantially longer than short completions.

PasteGuard is a privacy proxy for teams that use AI with customer records, client files or project secrets. It runs locally or on your own server and checks requests before they reach a model provider.

PageLM is a self-hosted AI study platform inspired by NotebookLM. Students, educators and researchers can use it to turn their materials into resources for revision and exam practice.