Needle 3: CPU function calling demo and limits

Needle 3 from Cactus is the focus of this tutorial on on-device automation. The speaker describes a 53 million parameter model with a 35 MB file that produces structured function outputs from text. In the home automation demo, he reports a response time of about 66 milliseconds. He also cites two function calls in under 97 milliseconds on a CPU without a GPU; these figures are presented as demonstration results and claims, not guarantees for every device.

The Python walkthrough uses a Google Colab notebook. Functions define typed inputs, docstrings for function selection, and a decorator that exposes them to the model. A request to dim living room lights to 30 and lock the front door becomes separate calls with extracted arguments. The speaker contrasts this process with token-by-token generation in a local LLM and says Needle 3 does not need an additional JSON parser.

The limitations receive substantial attention. Conversation history can cause incorrect calls unless the state resets between requests. Missing default values can leave a selected function without usable results. A clear invoice description works in the extraction example, while messier message text fails. The speaker also finds intent classification weaker than the System 1 model he compares it with.

The architecture discussion covers a simple attention network and model slicing, including a reported 13 MB two-layer option. An embeddings example explores cosine similarity for retrieval, which the speaker treats as a possible use rather than a settled recommendation.