LLM tutorial: local models, shell pipelines and RAG

Simon Willison explains how his LLM command-line tool fits into Unix pipelines: text enters as a prompt or context, and the response becomes input for another command. He demonstrates installation with Python tooling or Homebrew, API key configuration, conversation continuation and model selection.

Uploaded on June 12, 2024, the talk covers the tool's capabilities at that time. The supplied metadata gives no separate recording date or LLM release number. Model examples include Claude 3, GPT-4o and Mistral 7B Instruct; the setup demonstrations should be read in that dated context.

For viewers who want to run models locally, Willison demonstrates GPT4All and Ollama plugins, then shows llamafile bundling a model with its runtime. His downloaded Mistral example runs offline on his laptop. He explains that interactive chat avoids loading the model again for every prompt. His own machine is an M2 Max with 64 GB of RAM, rather than a stated minimum requirement.

The later examples combine curl and jq to summarize Hacker News threads, pass project files into prompts, and use ShotScraper to extract web search results for RAG. LLM stores prompt history and embeddings in SQLite, which he browses with Datasette. The demonstrated blog embeddings use OpenAI’s API; local storage does not make that example offline. He then pipes semantic-search results into another prompt. Some live demonstrations fail, including the llamafile plugin call. Structured output and function calling remain planned features in this presentation.