localGPT 2.0 preview: document chat and RAG setup

Uploaded July 15, 2025, this guide covers the localGPT 2.0 preview on the localgpt-v2 branch. The speaker describes his open source project as a way to chat with documents using local models without sending documents to external APIs. These claims and setup details refer to that preview, rather than a confirmed current release.

The walkthrough shows PDF indexing with hybrid search, embedding selection and optional sentence-level chunking. Document overviews help the model decide whether to retrieve evidence or answer from its own knowledge. Context enhancement adds summaries based on nearby chunks. The speaker demonstrates question decomposition and reranking, and explains sentence pruning. An optional answer check adds another LLM call, which he says can take 5 to 10 seconds.

Setup uses Ollama, Node.js/npm and a Python environment. He demonstrates starting services together or separately, then opening the frontend on localhost port 3000. Docker deployment is another documented option, but this guide focuses on direct setup.

The preview supports PDFs only. The speaker developed it on macOS and warns of possible Windows and Linux bugs. He identifies formatting and streaming issues, notes that cached answers can complicate repeated tests, and explains that poor PDF parsing can affect retrieval. His advice is to compare indexes with different settings and ask explicit document-specific questions.