LLM Client Libraries and SDKs

Needle is an AI model for developers building device assistants that turn requests into actions. It prioritizes automation over general chat, with smaller model variants for devices that have limited resources.

FastMCP is a Python framework for developers who want AI agents to use their applications and data through the Model Context Protocol (MCP). It covers both publishing tools and connecting to them, with support for servers that run locally or remotely.

Mirascope is an open-source development library for building AI applications with local models or cloud providers. Its documented local backends include Ollama, vLLM and LM Studio, plus direct inference through MLX on Apple Silicon Macs. Developers can use a shared model interface while keeping inference on their own hardware or sending requests to OpenAI, Anthropic or Google.

npcpy is a Python library for developers and researchers building AI agents and applications with language models. It supports local LLM backends as well as cloud providers, so projects can choose where model requests run.

Headroom reduces the context that LLM apps and coding agents send to a model. It's for developers whose prompts fill up with repetitive tool results, logs, file contents and retrieved documents. Compression runs locally, but model requests still go to the chosen LLM provider.

LLM Sandbox is a Python library for developers whose AI agents or applications need to execute generated code. It puts execution in containers on infrastructure you control, with support for local deployments and Kubernetes clusters.

Spring AI is a Java framework for developers adding AI features to Spring applications. It connects models with application data and APIs for tasks such as answering questions about documents.

node-llama-cpp is a library for developers adding local AI to JavaScript and TypeScript applications. It uses llama.cpp to run GGUF models on your hardware and works in Node.js, Bun and Electron.

Context7 is a locally runnable MCP server and CLI that connect coding assistants to its hosted documentation service. It's for developers who want library references and code examples in their assistant's context. The local component is the connection to that service: the API backend, document parsing and crawling engines remain private hosted components.

picoLLM is an on-device LLM inference SDK for developers adding text generation to their apps. Inference works offline, but a Picovoice account and AccessKey are required, and key validation needs an internet connection.

LangChain is an open-source framework for developers building AI agents and LLM applications, including apps that run models locally through Ollama. Its Python ChatOllama integration connects to open-weight models such as gpt-oss. You write the application in code, while Ollama handles local inference.

LM Format Enforcer is an open-source Python library for developers who need a local LLM to return structured data their applications can parse. It constrains text during generation, so a prompt's formatting request doesn't have to carry the whole burden. The library uses the MIT license.

chromem-go is a vector database for Go developers adding semantic search or retrieval augmented generation (RAG) to an application. It runs inside the app, so it doesn't need a separate database server.

Strands Harness SDK is an open source toolkit for developers building AI agents who need control over their behavior. It handles the agent loop while letting you choose its components.

MemMachine gives developers building AI agents a way to remember earlier conversations and user preferences. Its memory persists across sessions, agents and model changes, so user history can carry over when an application switches models.

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.

Megatron-LM is a Python framework for researchers training large language models on NVIDIA GPU infrastructure. It provides reference training scripts for experimenting with models and studying distributed training.

LangExtract is a Python library for developers who need to turn notes, reports or other text into structured data. It links extracted details to their original passages so people can check the results against the document.

Instructor is a library for developers who need to turn text into structured data using an LLM. Its focus is extraction, with typed outputs that applications can use and validation rules that check the model's responses.

Genkit is Google's framework for developers building AI agents and chatbots into web and mobile apps. It brings model connections, application workflows and testing tools into one framework.

Outlines is a Python library for developers who need LLM responses in a defined format. It constrains generation to produce structured data for tasks such as extracting service tickets from emails or limiting an answer to allowed choices.

Acontext gives AI agents reusable memory from past tasks. It's for developers who want agents to retain successful approaches, mistakes, and user preferences in files they can inspect and correct.

pgai is a Python library for developers building RAG applications and AI agents around PostgreSQL data. The project is archived and no longer maintained or supported.

LLM Guard is a Python security toolkit developers can deploy as a self-hosted API to check prompts and model responses in their applications. It provides screening and sanitization rather than running the application's LLM. The project is archived and no longer maintained, including its associated Hugging Face models.

Chainlit's Python framework lets developers self-host conversational AI applications and provide a browser chat interface around their own application logic. Its OpenAI assistant example uses the OpenAI Python SDK’s standard connection, sending model requests to OpenAI’s cloud service. Self-hosting the chat server and supplying model inference are separate choices.