Build a Python RAG app with Groq and Sentence Transformers

This tutorial explains retrieval-augmented generation through an open-book exam analogy, then builds a Python example in Google Colab. The speaker presents RAG as a way to supply information beyond a model's training data, including private company documents, and reduce unsupported answers. The explanation also covers why a limited context window makes passing thousands of PDFs into a single prompt impractical.

The implementation uses a sample Acme return and support policy rather than importing PDFs. It tokenizes the text with tiktoken's cl100k_base encoding, creates 60-token chunks with a 10-token overlap, and converts them back to text. Sentence Transformers loads all-MiniLM-L6-v2 to produce normalized embeddings with 384 dimensions. The example stores six chunks and their vectors in memory instead of using a vector database.

For each question, the code uses the same embedding model, ranks chunks by cosine similarity and joins the top three into context. A system prompt asks the model to answer only from that context and decline when it lacks an answer. Generation uses Groq's API with Llama 3.1 8B Instant, so the demonstrated workflow requires an API key and is not offline. The speaker tests policy questions and an unrelated question that produces a refusal; these examples do not establish that the prompt prevents every hallucination.