Ollama and PrivateGPT: offline chat and RAG setup

This tutorial starts with Ollama for running a local LLM, then adds PrivateGPT to answer questions about personal documents. The speaker describes the initial model as uncensored and usable offline after download. Its name is not stated in the transcript, so the demonstration does not establish that every model behaves the same way.

The Ollama section covers downloads for Mac, Linux and Windows, pulling a model and starting a chat. The speaker then explains RAG, or retrieval augmented generation, as a way to give a model access to documents outside its training data. This part demonstrates document retrieval rather than model fine-tuning.

For PrivateGPT, the walkthrough uses Ubuntu through Windows Subsystem for Linux. It covers cloning the repository, setting up pyenv with Python 3.11.14, installing Poetry and adding its bin directory to PATH. Poetry must also use the intended Python version. The setup includes UI, embedding and vector-store dependencies, plus Ollama inside WSL. Llama 3.1 is the model selected in the demonstrated configuration.

The interface opens at localhost:8001. The speaker uploads a PDF and asks about its contents; the answer includes references to pages 26 and 4. The interface also has search and summary modes, while basic chat ignores uploaded files.