Tools tagged with "Streaming inference"

50 tools

LangChain

5 videos

An MIT-licensed Python framework for AI agents and LLM apps, with model, tool and data integrations. LangChain.js serves JavaScript and TypeScript.

147.3KUpdated 1 day agoMIT

#Human approval#RAG#Streaming inference

LangChain is an MIT-licensed open-source framework for developers building AI agents and applications powered by LLMs. It provides a shared interface for models, tools and data connections, so developers can change providers or test workflows without rebuilding the whole application.

Agent FrameworksLLM Libraries and SDKsRAG Frameworks and Pipelines

Ollama JavaScript

JavaScript client for Ollama in Node.js and browsers. Connect to local models or Ollama's cloud with an open-source, MIT-licensed library.

4.4KUpdated 2 days agoMIT

Web#LoRA#Multimodal input#Ollama integration

Ollama JavaScript connects Node.js and browser applications to models running through Ollama. It's for developers building chat interfaces, AI agents or other apps that need a local LLM backend. The library is open source under the MIT license, with TypeScript types and an API that follows Ollama's REST interface.

LLM Libraries and SDKs

PocketPal AI

1 video

A mobile AI assistant that runs GGUF models on iOS and Android. Core chat works offline after a model download and needs no account.

8.5KUpdated 2 days agoMIT

iOS · Android#GGUF#Hugging Face integration#llama.cpp backend

PocketPal AI is an open source assistant for people who want to run language models on a phone or tablet. It works on iOS, iPadOS and Android. Once you've downloaded a model, you can chat offline without an account, and your prompts, replies and documents stay on your device. The app is licensed under MIT.

AI Characters and RoleplayLocal Model RunnersMobile Chat AppsOn-Device and In-Browser AIText-to-Speech and Audiobooks

Speaches

Self-hosted speech API for transcription, translation and speech generation. Runs via Docker on CPU or GPU with faster-whisper, Kokoro and Piper.

3.7KUpdated 5 months agoMIT

Docker#OpenAI-compatible API#Streaming inference

Speaches is a self-hosted speech server for developers who want transcription, translation and speech generation on their own hardware. Its OpenAI-compatible API lets applications use local speech models through tools and SDKs built for OpenAI's API. The project is open source under the MIT license.

Inference Servers and ClustersSpeech-to-Text and TranscriptionText-to-Speech and AudiobooksTranslation

llama-swap

1 video

A local AI proxy that switches models on demand through OpenAI and Anthropic compatible APIs. Runs on macOS, Windows, Linux and FreeBSD under MIT.

5.8KUpdated 2 days agoMIT

macOS · Windows · Linux · Docker · Web#GGUF#Image-to-image#llama.cpp backend

llama-swap is a self-hosted proxy for people running several AI models on their own hardware. It starts the model server a request needs and swaps out another when necessary, so you don't have to keep every model loaded or manage separate API connections in your apps.

LLM Gateways and RoutersLocal Model Runners

Chainlit

An open-source Python framework for AI chat apps you can self-host, with customizable interfaces, OAuth and integrations including LangGraph and HuggingFace.

12.5KUpdated 1 month agoApache-2.0

Web#LLM tracing#Multi-user access#Streaming inference

Chainlit is an open-source Python framework for developers building conversational AI apps with their own application logic. You can run the app on your own server and give users a browser chat interface. It's a framework for creating an app, rather than a ready-made chatbot with a fixed model.

AI Bots for Chat AppsLLM Libraries and SDKsSelf-Hosted Chat Web UIs

MLX LM

2 videos

A Python package for local LLM inference and fine-tuning on Apple Silicon, built on MLX with Hugging Face model support and an MIT license.

7.2KUpdated 1 day agoMIT

macOS#Batch processing#Distributed execution#Hugging Face integration

MLX LM is an open-source Python package for generating text and fine-tuning language models locally on Apple Silicon Macs. Built on MLX, it suits developers and researchers who want to work with models through Python or a terminal, including adapting models to their own tasks. The package uses the MIT license.

Inference Libraries and BindingsLLM Fine-TuningLocal Model RunnersQuantization and Model ConversionTerminal Chat Clients

OllamaSharp

.NET library for local LLM apps using Ollama, with streaming chat, embeddings and model management. Open source under MIT; also supports Ollama cloud.

1.4KUpdated 2 months agoMIT

#Multimodal input#Ollama integration#Streaming inference

OllamaSharp is a C# library for developers building .NET applications around Ollama. It connects to Ollama on your own machine or a remote server and covers the full Ollama API, including model management alongside chat and embeddings. It's open source under the MIT license.

LLM Libraries and SDKs

Instructor

An MIT-licensed LLM data extraction library that validates structured outputs with Pydantic and works with Ollama, llama-cpp-python, vLLM and cloud APIs.

14KUpdated 3 weeks agoMIT

#llama.cpp backend#Ollama integration#Streaming inference

Instructor is an open-source library for developers who need structured data from local LLMs or cloud models. It turns natural-language input into typed objects that applications can use, with validation and retries built into the extraction process. Its focus is data extraction.

LLM Libraries and SDKs

Coqui TTS (idiap fork)

Open-source text-to-speech toolkit for local speech generation, voice cloning and model training on Linux, macOS and Windows, licensed under MPL-2.0.

2.3KUpdated 4 months agoMPL-2.0

macOS · Windows · Linux · Docker#Multilingual#Streaming inference#Voice cloning

Coqui TTS (idiap fork) is a local text-to-speech library for developers and speech researchers who want pretrained voices or tools to train their own models. It builds on coqui-ai/TTS, continuing the original unmaintained project. The Python toolkit is open source under the Mozilla Public License 2.0 (MPL-2.0).

Inference Libraries and BindingsText-to-Speech and AudiobooksTraining FrameworksVoice Cloning and Voice Changers

WebLLM

2 videos

A local LLM engine that runs models in the browser with WebGPU acceleration, OpenAI API compatibility, and an Apache 2.0 license.

19.2KUpdated 2 weeks agoApache-2.0

Web · Browser Extension#OpenAI-compatible API#Streaming inference#Structured output

WebLLM runs language models directly in a user's browser, using WebGPU for GPU acceleration. It's an open-source engine for developers building web-based AI assistants and Chrome extensions that process prompts on the user's device rather than an inference server. The project uses the Apache 2.0 license.

Inference Libraries and BindingsLLM Libraries and SDKsOn-Device and In-Browser AI

Haystack

An open-source Python AI framework for self-hosted agents and document search, with local model support and an Apache 2.0 license.

26.6KUpdated 1 day agoApache-2.0

Docker#Guardrails#Hugging Face integration#Hybrid search

Haystack is a Python framework for developers building self-hosted AI agents, document search, and apps that answer questions using their own data. Its modular pipelines let teams control which information reaches a model and inspect how retrieval, memory, tools, and generation contribute to an answer. It's open source under Apache 2.0.

Agent FrameworksLLM Libraries and SDKsRAG Frameworks and Pipelines

Ollama Python

Python client for Ollama with local and cloud model access, streaming chat, embeddings, and async support. Open source under the MIT license.

10.6KUpdated 2 days agoMIT

#Batch processing#Ollama integration#Streaming inference

Ollama Python connects Python applications to models running through Ollama on your own machine or to Ollama's cloud service. It's for developers adding local LLM features to scripts, chat applications, or other Python projects. The library is open source under the MIT license and requires a running Ollama service for local use.

LLM Libraries and SDKs

Vercel AI SDK

2 videos

A free TypeScript library for AI apps and agents, with streaming, provider switching, and chat UI hooks for React, Next.js, Svelte, and Vue.

27.1KUpdated 3 hours ago

#Agent Skills#Streaming inference#Structured output

Vercel AI SDK is a TypeScript library for developers building chatbots, generative interfaces, and AI agents. It gives an application one way to work with models from providers such as OpenAI, Anthropic, and Google, so teams can change providers without rebuilding the parts of their app that handle responses. It suits developers who want control over the application they build while choosing how it connects to model services.

Agent FrameworksLLM Libraries and SDKs