Ollama and Python: build RAG from scratch
Learn to build a local RAG pipeline with Ollama, Qwen 3 4B Instruct and NumPy, using paragraph chunks and cosine similarity instead of a vector database.
Player not loading? Watch on YouTube
This tutorial builds a RAG pipeline in plain Python, which the presenter describes as under 50 lines of code. A fictional cafe handbook supplies the documents. The example retrieves relevant passages and adds them to a question before asking a local LLM for an answer. It changes the prompt rather than training the model.
The setup uses Ollama to run models locally: Nomic Embed Text creates embeddings, while Qwen 3 4B Instruct generates answers. The presenter gives download sizes of 274 MB and about 2.5 GB respectively, and recommends testing response speed on your own machine. Python requests calls Ollama's local server on port 11434; NumPy handles the vector calculations. The project uses uv for setup and execution.
The code splits Markdown files at blank lines, producing 11 chunks from the sample handbook. It embeds the chunks and each question, ranks matches with cosine similarity, and passes the three closest passages into the answer prompt. LangChain and a prebuilt vector database are outside this implementation.
The final section explains what the example leaves out. It recomputes embeddings when it runs; persistent vector storage could avoid that work. The presenter also discusses overlapping chunks, preserving tables and code, reranking results, and evaluation questions for measuring answer quality.
The video includes a paid TripleTen segment, separate from the implementation.