Gemma, LM Studio and Ollama: a local AI overview

This Google-sponsored episode explains how to run models locally and choose software for a first experiment. The speaker separates the model files, Hugging Face model discovery, the runner and the workflow built around them. He also explains parameters, tokens, context windows, quantization and GGUF, with Q4 as his suggested starting point for lower memory use.

For Gemma 4, he recommends E4B for initial desktop experiments and E2B for older or slower machines. His hardware guidance is approximate: start small with 8 GB of RAM, try E4B or smaller quantized models with 16 GB, and consider larger workflows with 32 GB. Larger models still depend on the available hardware.

The setup discussion covers LM Studio chat and its localhost server, Ollama's command-line runner and API, and Google AI Edge with LiteRT-LM for on-device applications. A brief comparison covers Llama, Qwen, DeepSeek, GLM, Mistral and Phi, including license and deployment considerations.

The proposed first workflow turns 10 support tickets into a memo. The speaker recommends checking quotations, missed complaints and formatting against a cloud model before considering fine-tuning. He proposes hybrid processing with a private local first pass and human approval, then outlines business ideas for home health documentation, offline field reports and outbound draft review.