LLM Client Libraries and SDKs

Bifrost is a self-hosted AI gateway for developers and teams whose applications use multiple model providers. It puts Ollama, custom model deployments, and cloud services behind one OpenAI-compatible API, so applications can switch models without maintaining a separate integration for each provider.

Graphiti is a self-hosted Python framework for developers building AI agents that need to remember changing facts. It builds knowledge graphs from conversations, structured records and unstructured text, so an agent can query current information or recover what was true earlier. It's open source under Apache 2.0.

Cua gives AI agents access to computers they can inspect and operate, with tools for desktop automation, local virtual machines, and hosted fleets. It's for developers building agents that work across native apps and browsers, or evaluating how well those agents complete computer tasks. You bring the agent and model.

Context7 brings current library documentation and code examples into an AI coding assistant's context. It's for developers who want answers grounded in the libraries and versions they're actually using, rather than an assistant's older training data. It works with Claude Code, Codex, Cursor, Devin Desktop and Antigravity.

Pydantic AI is a Python SDK for developers building AI agents into their own applications. Its main draw is Pydantic validation across agent tools and results, so an agent can return structured data that application code can check and use. The SDK is MIT licensed.

LiteLLM gives platform teams one place to manage access to LLMs across providers. Its self-hosted AI gateway puts cloud services and internal or locally hosted models behind an OpenAI-compatible API, so applications can change models without changing their integration. Developers can also use its Python SDK directly.

LangGraph is a free, MIT-licensed Python framework for developers building AI agents that need to keep state and handle complex tasks. It gives teams control over how an agent moves between steps, where people can intervene, and what happens when a long-running task is interrupted.

Mem0 is a memory layer for developers building AI agents and assistants that need to recall earlier interactions. It retains user, session, and agent context across conversations, so an assistant can remember preferences or a support bot can refer to past tickets. The self-hosted code is open source under Apache 2.0; Mem0 also has a managed service.

Transformers is a Python library for developers and researchers who want to run pretrained AI models or train their own on hardware they control. It covers language, images, audio, video and multimodal work through a shared way of defining models. The library runs in a local Python environment; pretrained checkpoints are available from the separate Hugging Face Hub.

LLM Guard is a Python security toolkit for developers building applications around large language models. It checks prompts and generated responses for risks such as prompt injection, sensitive data exposure and harmful language. The project is archived and no longer maintained, including its associated models on Hugging Face.

Rebuff is a prompt injection detector for developers building LLM applications that accept untrusted input. It combines checks for suspicious prompts with a record of past attacks and tests for leaked prompt content. The project is archived and no longer maintained.

Vigil is a self-hosted security scanner for developers and researchers who want to check LLM inputs and responses for prompt injection, jailbreak attempts, and other suspicious content. It combines several detection methods and includes attack signatures and datasets, so teams can assess known threats without building every detector themselves. It is experimental alpha software for research and is open source under Apache 2.0.

RouteLLM is a self-hosted Python framework for developers who want to split requests between a stronger LLM and a cheaper model. It judges which prompts need the stronger model, so an application doesn't have to send every request to its most expensive provider. You control the cost-quality tradeoff through a routing threshold.

picoLLM is an on-device inference SDK for developers building apps that run compressed language models on users' hardware. It generates text locally, so prompts don't need to go to a cloud inference service. Its main distinction is Picovoice's compression method, which learns how to allocate precision across model weights rather than applying a fixed allocation.

DataDreamer connects LLM prompting, synthetic data generation, and model training in one Python library. It's for researchers and developers who want to build datasets and use them to fine-tune or align models in reproducible workflows. The library is open source under the MIT license.

Composio connects AI assistants and custom agents to apps such as Gmail, Slack, GitHub, and Linear. It's for people who want their assistant to act on requests across apps, and developers who don't want to maintain each integration themselves. Its CLI gives coding agents a local interface; the standard setup uses Composio's hosted authentication and execution service. It requires an account and internet access.

Vocode is an open source Python library for developers building voice AI agents, with a self-hosted telephony server and support for live conversations through a computer's microphone and speakers. It connects speech recognition, an LLM, and speech synthesis in one library. The code uses the MIT license.

Rivet is a desktop visual programming environment for developers building AI agents and applications with complex LLM workflows. Its editor runs on macOS, Windows and Linux, and its TypeScript library executes the resulting graphs inside your own application. The project uses the MIT license.

LMQL is a programming language for developers who need model calls and ordinary Python logic in the same program. It lets you define rules for generated text, including types, length limits, allowed answers and stopping phrases. Those rules apply during generation, so you can constrain intermediate responses as well as the final output.

pgai keeps search embeddings in sync with PostgreSQL data for developers building RAG applications and AI agents. It's a Python library with database components and workers you can self-host, including in Docker. The project is archived and no longer maintained or supported. Its code is open source under the PostgreSQL License.

chromem-go is a vector database that runs inside your Go application, so developers can add semantic search or retrieval augmented generation (RAG) without maintaining a separate database server. It stores text alongside embeddings and retrieves related documents for use in LLM answers. Its focus is ordinary application workloads rather than collections containing millions of documents.

Quivr Core is a Python framework for developers adding document-based AI answers to their own applications. It combines file ingestion with retrieval-augmented generation (RAG), so a model can answer questions using material from your documents. It supports local models through Ollama as well as cloud APIs from OpenAI, Anthropic, and Mistral.

Nexa SDK is an on-device AI inference framework for developers building applications that process text, images or audio on users' hardware. It runs models locally across CPUs, GPUs and NPUs, with a shared interface for different backends. Its scope includes language and vision models, speech recognition, speech synthesis and image generation.

LangChainGo is a Go implementation of LangChain for developers building LLM applications in their own software. It connects Go programs to model backends, including Ollama for local LLM use and cloud services such as OpenAI and Gemini. It's a library, so its audience is developers who want to build an application rather than use a ready-made chat interface.

Unstructured is a local document processing library for developers building LLM applications and document ingestion pipelines. It turns PDFs, Word documents, HTML, emails and images into document elements that applications can use. The Python library is open source under Apache 2.0 and runs on your own hardware, including through Docker images for x86_64 and Apple Silicon.

Prompt flow is an MIT-licensed, open-source toolkit for developers who build LLM applications and need to test their behavior before deployment. Its development tools run locally, while an optional cloud version in Azure AI supports team collaboration. Feature development has ended.

LiteRT is Google's open-source framework for developers building AI into apps that run on users' own devices. It succeeds TensorFlow Lite and covers model conversion, optimization and local inference. It's licensed under Apache 2.0.

CopilotKit is a self-hostable SDK for developers building AI agents into web and mobile apps, Slack, or Microsoft Teams. Agents can display interactive charts and forms using an app's own components, read shared app state, and take actions through frontend tools, APIs, or MCP tools.

ScrapeGraphAI is an AI web scraping tool for developers who want to describe the data they need in plain language. Its open-source Python library runs on your own infrastructure under the MIT license. A separate managed API runs in ScrapeGraphAI's cloud.

Curator is a Python library for developers preparing LLM training datasets or extracting structured records from existing data. It supports local inference through Ollama and vLLM alongside cloud model APIs, so the same data pipeline can use models on your hardware or a hosted provider. It's open source under Apache 2.0.

llama.rn brings llama.cpp into React Native apps so developers can run local LLM inference on iOS and Android. It's an MIT-licensed library for building AI features into a mobile app, with model processing on the device. It uses GGUF models and requires React Native's New Architecture.

OpenLLMetry adds LLM tracing to the OpenTelemetry monitoring stack a team already uses. It's for developers who need to follow model calls alongside database activity and API requests in their AI applications. The extensions run within your application and send standard OpenTelemetry data to your chosen monitoring destination.

any-llm is a Python library for developers who want the same application to work with local LLM servers and cloud providers. It connects to Ollama and custom OpenAI-compatible endpoints, alongside OpenAI, Anthropic, Mistral and Azure / Microsoft Foundry. A shared interface reduces the provider-specific code needed to try another model or change where inference runs.

rerankers is a Python library for developers building search and retrieval systems who want to compare reranking models without rewriting their integration each time. It takes a query and candidate documents, then ranks their relevance through a shared interface across local models and hosted services. It's open source under Apache 2.0.

Distilabel is an open-source Python framework for engineers building datasets to train or evaluate AI models. It pairs synthetic data generation with LLM feedback, so a pipeline can create examples and judge their quality. It uses the Apache 2.0 license.

LLPhant is an MIT-licensed PHP framework for adding language models, embeddings and vector databases to Symfony and Laravel applications. It requires PHP 8.1 or later and is installed through Composer.