AnythingLLM + Ollama: local document RAG setup

This tutorial sets up AnythingLLM with Ollama on Windows 64-bit to answer questions about documents stored on the presenter's computer. The presenter explains RAG, or retrieval augmented generation, and the roles of the language model, embeddings and vector database. The aim is a local AI workflow that avoids sending documents to a cloud provider.

The setup uses Qwen 3.5 with a download shown as 2.7 GB. The presenter enables "Expose Ollama to Network," selects Ollama in AnythingLLM, and keeps the default embedding and vector database settings. Model choice depends on available resources; the presenter recommends testing several models against the language and documents involved.

Examples include summarizing a PDF, finding invoices from 2016 and retrieving a password from a Docker Compose file. The password query needs clarification before it succeeds. The tutorial then covers uploading files to a workspace and granting directory access through Agent Skills. The presenter permits reading and searching while leaving write actions disabled, warning that an agent could otherwise alter or delete files.

Results have practical limits. The small model may misunderstand requests, and the final directory query returns 32 files after roughly three to four minutes.