LLM Client Libraries and SDKs

search2ai is a self-hosted search gateway for developers adding web access to AI agents and chat clients. It connects your chosen search services through one interface, with backup providers when requests fail.

Distilabel is a Python framework for engineers building synthetic datasets and model-generated feedback with local LLM backends such as Ollama, llama.cpp and Hugging Face Transformers. It also connects to cloud services, including OpenAI and Anthropic; requests through those integrations go to the remote provider. Where inference runs depends on the backend you choose.

KeyBERT is a Python library that extracts keywords and phrases by comparing their meaning with the document they come from. It's for developers and researchers who want to add keyword extraction to text processing projects without training a model from scratch. The library is open source under the MIT license.

LLPhant is a PHP framework for developers adding AI capabilities to Symfony and Laravel applications. It supports local model services and cloud providers, so you can choose where model requests run.

Ragas is a Python library for developers evaluating LLM applications and retrieval-augmented generation systems. Its non-LLM metrics compute text comparisons in your Python environment without a model, including string similarity, BLEU and ROUGE. Its model-based quickstart uses OpenAI's GPT-4o and an API key, sending evaluation inputs to OpenAI's cloud service. Local text scoring and cloud model judgments are separate evaluation options.

EverOS is a Python library and memory runtime for developers building agents that need context across sessions and applications. It keeps conversations, files and task histories in readable Markdown that you can edit and version with Git.

M Flow is a self-hosted memory engine for developers whose AI agents need to recall past conversations and work preferences. It runs in Python or Docker and supports Ollama for local model use, alongside providers such as OpenAI and Anthropic. Choosing a cloud provider sends model requests to that service.

Memind gives AI agents long-term memory for projects, user preferences and past tasks. Developers can use it to carry context between sessions while keeping links to the material each memory came from.

Composio's local CLI connects coding agents such as Claude Code to a hosted service for app actions and authentication. It's for people adding integrations to an assistant or building their own agents. The CLI runs on your machine, but requires a Composio account and internet access; model processing depends on the connected agent and its provider.

LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.

PyOD is a Python library for detecting unusual records and behavior in data on your own machine or server. A shared fit/predict interface lets developers and data scientists compare classical and deep learning anomaly detectors without changing the surrounding application for each algorithm. Its code uses the BSD 2-Clause license.

Prompt flow is a Python toolkit for developers who want to run and evaluate LLM application workflows in their own development environment. Its chatbot examples call OpenAI or Azure OpenAI remotely using API credentials. Workflow execution and evaluation can run locally, while those model requests leave the machine.

fx is a coding agent for developers who prefer working in a terminal and researchers who want to embed an agent in their own software. Its shell-style interface keeps tool activity alongside the conversation and preserves terminal scrollback.

FinGPT is a financial LLM framework for researchers and developers building tools that interpret market news and company information. It includes trained models, financial datasets, and tools for adapting models to specific finance tasks.

CopilotKit is a self-hostable SDK for developers putting AI agents inside their apps. Its open-source SDK uses MIT, and its enterprise runtime and Intelligence platform can also run on your infrastructure. Copilot Cloud provides managed hosting; where model requests go depends on the agent backend you connect.

Quivr is a content search and monitoring engine for developers building applications around incoming material and topics they want to follow over time. It's at an evaluation stage, and the project advises against production use because its API may change without notice.

ExtractThinker is a Python library for developers who want to extract structured data from documents using models running locally through Ollama. It also supports OpenAI, Anthropic and Groq. With a cloud provider, extraction sends document content to that service, requires provider credentials and may incur charges.

mcp-use is a TypeScript framework for developers building MCP servers and apps for ChatGPT, Claude, and AI agents. It brings tool definitions, interactive interfaces, and testing into one development framework.

E2B provides isolated execution environments for developers building coding assistants, computer-use agents and background jobs. You supply the model and agent logic; it supplies the machine where the agent works.

PageIndex is a document retrieval engine for developers building question answering over long PDFs, including financial reports, legal documents and technical manuals. It uses an LLM to select relevant sections from a document tree without a vector database or document chunking.

rerankers is a Python library for developers building search and retrieval systems. It ranks candidate documents against a query and lets you compare reranking approaches through a shared interface.

Ollama Python is a client library for developers adding model conversations to Python applications. It connects to a running Ollama service for local use or to Ollama's cloud service for larger models.

Unstructured is a document processing library for developers preparing files for LLM applications. It turns text documents and images into structured output that applications can work with.

RankLLM is a Python toolkit for search researchers and developers who want to compare how AI models reorder retrieved documents by relevance. Its main focus is listwise reranking, where a model considers several candidates together. It supports local inference and optional cloud APIs, and it's open source under the Apache 2.0 license.

Pydantic AI is a Python SDK for developers building agents into their applications. It uses Pydantic validation to give application code structured results and check the arguments agents pass to tools.

Langroid is a Python framework for developers building applications where AI agents divide up a task. Each agent can maintain its own conversation and use documents or tools, then delegate work to other agents.

Semantic Kernel is an SDK for developers building AI agents and coordinated workflows inside applications. It lets developers choose local model backends or cloud providers for those applications.

mergekit combines pretrained language models into one model for developers and researchers who want to combine specialized skills or balance model behaviors. It works directly with model weights, without additional training or access to the original training data.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

Safetensors is a file format and library for developers storing and loading AI model weights on their own hardware or servers. It addresses the arbitrary code execution risk of pickle-based PyTorch files.

Gradio builds browser interfaces around machine learning models, APIs and Python functions. It's for developers and researchers who want people to try an AI application without writing a separate JavaScript or CSS frontend.

LLM is a command-line tool and Python library for developers who want to use language models in terminal workflows or applications. It supports models running on your machine alongside remote providers such as OpenAI, Anthropic and Google.

Rivet is a desktop visual AI agent builder for developers working with connected prompts and application logic. Teams can inspect complex LLM workflows as graphs and run those graphs inside their own applications.

Bifrost is an AI gateway for developers and teams building applications that use local LLMs or multiple cloud providers. It gives those applications a shared API for model access and a central place to manage usage.