LLM Client Libraries and SDKs

LongMemory gives LLM applications and AI agents persistent memory across sessions. Developers can use it to retain project decisions and conversation context while distinguishing current facts from information that was true earlier.

FlashRAG is an open source Python toolkit for researchers and developers comparing retrieval-augmented generation (RAG) methods or building their own. It combines implemented research methods with prepared benchmark datasets and reported results, so experiments don't have to start with a separate codebase for each approach. It uses the MIT license.

Guardrails AI is an open-source Python validation framework that runs inside your application or as a self-hosted service. It's for developers who need to check LLM inputs and responses before their apps use them. It provides a validation layer around model interactions; it doesn't supply local model inference.

ContextGem is a Python framework for developers extracting structured information from documents such as contracts, invoices and reports. You describe what you need in natural language, and it handles extraction prompts and data modelling for text and image content.

OllamaSharp is a C# library for developers building .NET applications with Ollama. It gives those applications access to chat, embeddings and model management through the Ollama API.

NVIDIA NeMo Data Designer is a Python framework for developers and data teams creating synthetic datasets with controlled variation and relationships between fields. It can start with an empty dataset or build on your own seed data. The framework is open source under Apache 2.0.

CascadeFlow is a library for developers who need to control AI agent spending and behavior during a run. It works inside the application, assessing model calls, tool calls and sub-agent handoffs before deciding how execution should continue.

LazyLLM is a Python framework for developers building applications with multiple AI agents. It brings application workflows, model training and evaluation together so teams can refine a prototype using task-specific data.

PandasAI is a Python library that executes data analysis code in your environment and lets analysts ask questions in plain language. It can isolate that execution in Docker. Its documented LiteLLM integration uses OpenAI's gpt-4.1-mini, so model requests go to a cloud service and require an API key.

Rig is a Rust library for developers building AI agents and LLM applications. It provides a shared interface across model providers and vector stores for applications that need to work with different backends.

Rivet AgentOS is a runtime for developers building applications where AI agents need a working environment for code and tools. It runs inside an existing backend process, with WebAssembly isolation instead of a separate virtual machine for each agent.

txtai is a Python framework for developers building search, chat with their data, and AI agents on their own hardware. It combines a searchable knowledge store with model workflows, so applications can retrieve context and act on it.

LlamaIndex is a Python framework for developers building AI agents and applications that answer questions using their own data. It provides the connections and retrieval tools needed to give a language model relevant context from documents and other sources.

Jsonformer is a Python library for developers who need structured JSON from language models running through Hugging Face Transformers. It controls the JSON structure while the model supplies the values, reducing reliance on prompts that ask a model to produce correctly formatted output.

LLMLingua is a Python library for developers building LLM applications that need to fit lengthy input into a model's context window or reduce token costs. It uses smaller models to remove less useful text before the prompt reaches the answering model.

llama.rn is a library for developers adding local LLM features to React Native apps on iOS and Android. It brings llama.cpp inference onto the device, so an app can process model requests on the user's hardware.

FastRTC is a Python library for developers building voice assistants, webcam AI apps, and other interactive audio or video tools. It connects live media streams to Python functions so developers can focus on how their application responds.

Haystack is a Python framework for developers building AI agents, document search and applications that answer questions using their own data. Its pipelines give teams control over the information a model receives and the steps that produce an answer.

GitAgent is an AI agent framework for developers who want to version an agent's behavior and saved memory alongside its tools. Agent definitions live in a Git repository, where teams can review changes, fork agents and maintain different behaviors on separate branches.

Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.

OpenHarness SDK is a TypeScript toolkit for developers building coding assistants and other AI agents into their own applications. It gives the application control over conversation history, tool permissions and task delegation.

Memori's BYODB edition is a memory library for developers who want agents to retain context in their own SQLite file or self-managed database. It generates embeddings locally, but its default Advanced Augmentation sends conversation messages and summaries to Memori's cloud API for memory extraction. The documented OpenAI connection also sends model requests to OpenAI.

CDAF saves reusable text descriptions alongside videos so AI agents don't have to analyze the same footage for every task. It's for developers building video agents and editors that repeatedly search or ask questions about a clip library.

Chonkie is a library for developers preparing documents for retrieval-augmented generation (RAG). It brings document preparation and ingestion into one pipeline, with optional components to keep dependencies small.

Mem0 provides a self-hosted memory server and libraries for developers whose AI applications need to remember user preferences and earlier interactions. The code is open source under Apache 2.0. Its default language and embedding models are OpenAI's gpt-5-mini and text-embedding-3-small, so self-hosting still sends model requests to OpenAI and requires internet access.

llama-cpp-python is for developers adding local LLM inference to Python applications or connecting existing API clients to models on their own hardware. It brings llama.cpp into a Python library and a self-hosted server.

NVIDIA NeMo Agent Toolkit is an open-source Python library for developers building self-hosted servers that expose AI tools and workflows through MCP or A2A. It also measures and evaluates agent workflows across frameworks. Its introductory NVIDIA NIM workflow uses hosted inference and requires an NVIDIA account and API key.

LangChain4j is a Java library for building chatbots, assistants and AI agents that use local models through Ollama. It runs within JVM applications, while Ollama handles model inference on your hardware using a CPU or GPU. You can also connect to OpenAI or Google Vertex AI, in which case model requests go to the remote provider.

Graphiti is a Python framework for developers building AI agents that need to remember information as it changes. It turns conversations, structured records and unstructured text into a queryable graph with a history of facts.

PocketFlow is a Python framework for developers building AI agents and workflows with their own model connections. Its utility examples include an Ollama function for local LLM calls. Developers supply that function rather than using a built-in model wrapper. Cloud API examples send model requests to their providers. FastAPI examples show chat applications with streamed responses and background jobs.

UQLM is a Python library for developers and researchers assessing the reliability of LLM answers. Its confidence scores estimate the likelihood of errors or hallucinations; they don't verify that an answer is factually correct.

Prompty is a prompt format and developer toolkit for building AI applications. It keeps prompt text, model settings, inputs and tool definitions in a portable Markdown file that developers can share across editors and language runtimes.

ScrapeGraphAI extracts data from websites and documents based on a plain-language request. Developers can use its self-hosted Python library or call a separate managed cloud API.

Curator is a Python library for developers building LLM training datasets or extracting structured records from existing data. It supports pipelines that generate data in stages and process many model requests.