DSPy

DSPy is a Python framework for developers building language model applications with local backends such as Ollama on a laptop or SGLang on a GPU server. It connects application code to those model servers, which handle inference on your hardware. It also supports cloud providers, so where model requests go depends on the backend you choose.

Developers compose model tasks into modular Python programs and use optimization algorithms to improve prompts and model weights. This covers classifiers, retrieval-augmented generation pipelines and agent loops. The framework puts application structure and optimization in the same programming environment.

Its documented local integrations include Llama 3.2 through Ollama and Meta-Llama-3-8B-Instruct through an OpenAI-compatible SGLang endpoint. DSPy can also connect to other OpenAI-compatible endpoints. Cloud choices include OpenAI, Anthropic and Google's Gemini services, which require their respective credentials and receive the model requests sent to them.

A program can use multiple language models, with different models assigned to different parts of its work. DSPy caches model calls by default and keeps interaction histories with inputs, outputs and usage metadata. These records let developers inspect model behavior while comparing runs.

DSPy is open source under MIT. Its model interface supports both Chat Completions and the Responses API for models that use those endpoints.