LM Studio and Ollama: setup and model sizing

This introduction explains how to get a local LLM running with LM Studio or Ollama. The presenter describes LM Studio as a desktop app for browsing, downloading and chatting with models. Ollama provides a terminal workflow and a local server compatible with much of the OpenAI API. She explains that some existing apps can use that endpoint by changing their base URL and model name.

The hardware discussion separates memory capacity from speed: a model must fit in memory, while compute and memory bandwidth affect how quickly it responds. Quantization reduces the memory needed for model weights. The presenter's starting points are small models for 8GB machines, roughly 12–14 billion parameters around 16GB, and quantized models around 20–30 billion parameters with 24GB or 32GB. Context capacity and performance still depend on the model and runtime.

The coding assistant section explains how a harness gives a model access to files and commands. A later section distinguishes retrieval for changing documents from fine-tuning for behavior and stable domain patterns, then introduces LoRA, QLoRA and Unsloth.

The presenter recommends testing an application with Wi-Fi off to confirm offline operation. She also cautions that demanding tasks may suit frontier cloud models better, since local hardware can limit both response quality and speed.