Tools tagged with "Streaming inference"

Open Glean is a self-hosted AI workspace for searching files, notes, saved webpages and knowledge synced through Hydra DB connectors. Its browser app runs on your machine or server, including in Docker, but retrieval uses the remote Hydra DB API by default and requires a Hydra key. You can change the Hydra backend endpoint. Local answer models require explicit enablement and are disabled by default.

ReScript is a video and audio editor for people cutting interviews, podcasts and voice notes through text. Deleting words from a transcript removes the matching sections of the recording.

gp.nvim brings AI chat and prompt-based editing into Neovim for developers and writers who want to work with their existing buffers. You can ask about selected text or code, then apply the response inside the editor.

Unmute gives text LLMs a spoken conversation interface using Kyutai speech recognition and speech synthesis. It's for developers who want to self-host voice chat while choosing their own language model.

OpenBrowserAgent is an installed browser extension for Chrome, Firefox and Safari that combines page-focused chat with browser automation. It connects to local Ollama endpoints or model APIs you configure. Requests send selected browser context and messages to that endpoint; a remote model connection sends that material outside the browser.

TypingMind is an AI chat frontend for people who want to use different model providers in one workspace. It brings conversations, project materials and custom AI agents into a shared interface.

Oriveo Community Edition is a free, open-source AI chat client for people who want local models and cloud providers in the same app, with their conversation history stored on their own device. It runs on iOS, Android and the web, and you can self-host the web client. It doesn't require an Oriveo account.

VoiceStreamAI turns live browser audio into text on a server you control. It's for developers adding streaming transcription to an app or building a speech-to-text service around Whisper.

Codex Router runs on your computer and connects the Codex App and CLI to external model providers or your own self-hosted endpoints. It's for developers who want a different model within their existing coding workflow. Cloud routes send model requests to remote services and need internet access; paid providers still charge for access or usage.

LangChain is an open-source framework for developers building AI agents and LLM applications, including apps that run models locally through Ollama. Its Python ChatOllama integration connects to open-weight models such as gpt-oss. You write the application in code, while Ollama handles local inference.

Strands Harness SDK is an open source toolkit for developers building AI agents who need control over their behavior. It handles the agent loop while letting you choose its components.

VoxCPM is a local text-to-speech system for developers and audio creators who want to generate speech with voices they can design or clone. It supports multilingual narration and control over how a voice delivers the text.

Ray Serve is a Python library for developers building inference APIs around machine learning models. It's useful when a response needs several models or application steps, rather than a single model call.

Meetily records calls and turns them into live transcripts and meeting summaries. It's for people who want to keep meeting recordings on their own computer without adding a recording bot to the call.

Klee is a native macOS AI chat app and agent for people who want an assistant to work with files on their Mac. It runs models locally through MLX, and downloaded models can chat offline without an account or API key.

MLX LM is a Python package for developers and researchers who want to generate text and adapt language models on Apple Silicon Macs. It uses MLX for local inference and training, with access through Python or a terminal.

Talkio puts several AI models in one conversation so you can compare answers or have different roles discuss the same problem. It also supports ordinary one-to-one chats.

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.

KittenTTS is a Python text-to-speech library for developers adding local speech generation to apps or producing spoken audio from text. It supports voice cloning without model training and doesn't require a GPU.

Local OpenAI LLM for Home Assistant connects conversation agents and AI tasks to models served through an OpenAI-compatible API. It's for Home Assistant users who want to choose their own model server for voice conversations and generation tasks.

Minutes is a meeting notetaker for people who want searchable records of calls, voice memos and everyday dictation on their own computer. You can read the notes in the app or let an AI assistant search the conversations you choose to share.

sherpa-onnx is a toolkit for developers building speech features that work offline on their own devices. It covers transcription, speech generation and audio analysis, with processing kept local.

Haplo AI runs language models on iOS and macOS for people who want to ask questions, write code or examine photos without sending that material to a hosted assistant. After downloading a model, chat works offline without an account.

Instructor is a library for developers who need to turn text into structured data using an LLM. Its focus is extraction, with typed outputs that applications can use and validation rules that check the model's responses.

Speaches is a self-hosted speech server for developers building transcription services and voice applications. Its OpenAI-compatible API lets existing clients and SDKs work with speech models running on hardware you control.

LunarGate is a self-hosted AI gateway for teams whose apps use multiple model providers. It gives those apps one OpenAI-compatible endpoint and keeps routing, retries and fallback rules in your infrastructure. The gateway is open source under the MIT license and runs on macOS and Linux, with a container image also available.

Magenta RealTime is a local AI music model and synthesis engine for musicians playing AI instruments and developers building interactive music apps. It generates a continuous audio stream on supported Apple Silicon Macs.

Agent Chat UI gives developers a web interface for talking to their LangGraph agents and returning to existing conversations. It connects to an agent server you choose, so you can use it with a local development server or a remote deployment.

Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.

Void is a desktop AI code editor for developers who want to choose the models behind their coding assistant. It builds on VS Code as a fork; the project is archived and no longer maintained.

Chainlit's Python framework lets developers self-host conversational AI applications and provide a browser chat interface around their own application logic. Its OpenAI assistant example uses the OpenAI Python SDK’s standard connection, sending model requests to OpenAI’s cloud service. Self-hosting the chat server and supplying model inference are separate choices.