Player not loading? Watch on YouTube
This architecture walkthrough follows a Python application that calls an LLM as it grows into a service other applications can use. The speaker presents a replaceable set of tools, explaining when each layer becomes useful. Running the application on a laptop is the starting point; the discussion does not demonstrate local model inference or offline operation.
FastAPI exposes an HTTP endpoint, while Uvicorn runs the server. Asynchronous Python handles waits for model calls, databases and tools. Pydantic defines input contracts and validates structured model responses where the model supports them. For workflows with several steps, LangGraph manages state, routing and checkpoints, including pauses for human approval. The speaker notes that a single prompt and response may need no orchestration framework.
PostgreSQL holds durable application data, and pgvector adds embedding storage and similarity search for RAG. Redis supplies caching and temporary state. LiteLLM provides a common interface across model providers, with routing, retries and fallbacks; the speaker warns that fallback models still need testing.
Langfuse and Opik help investigate requests through traces and manage evaluations. Pytest checks predictable software behavior, while representative evaluation questions test answer quality and retrieval after changes. uv and Ruff support project maintenance. Deployment, autoscaling and cloud infrastructure receive only brief treatment, as do authentication, security and secret management.
