Player not loading? Watch on YouTube
This architectural walkthrough follows a website's rewrite button to explain how WebLLM moves generation onto the visitor's GPU. The speaker describes a local LLM workflow that avoids inference provider charges for those requests, while retaining model delivery and hosting costs.
The browser first downloads weights, a compatible model library, tokenizer assets and configuration. WebLLM uses the browser's Cache API by default, but cached files still need runtime preparation and working memory. Browser eviction can remove them. Fresh generation can work offline after loading; reopening the entire website also requires a strategy for loading its page and application code.
WebGPU handles GPU operations, while WebAssembly supports CPU work. A Web Worker helps keep the editor responsive without reducing model memory requirements. The speaker recommends testing whether rewrites preserve conditions such as "if the tests pass." The 1-billion-parameter example compares 2 GB of raw 16-bit weights with 0.5 GB at 4 bits, excluding metadata and runtime memory.
Compatibility checks include a secure context, normally HTTPS, and a usable GPU adapter. The walkthrough recommends measuring first and return visits, battery costs and output quality. Local processing alone does not establish that an app is private: analytics or document saving may still transmit text. A proposed hybrid fallback asks users to choose the server route before sending their selection.
