MLC LLM and React Native: on-device inference tutorial

This tutorial introduces @react-native-ai/mlc for running on-device inference in React Native apps on iOS and Android. The speaker explains how MLC LLM builds optimized code for selected models, using memory planning, operator fusion and hardware-specific optimizations. He describes GPU acceleration through drivers such as OpenCL and Vulkan, plus CPU optimizations for devices without suitable GPU support.

The repository's demo app shows how to select a model, download it from Hugging Face and prepare it for inference. The initial setup requires an internet download; the speaker says later runs take a few seconds to prepare. Developers can use the supplied model configurations or bring their own models. The demonstration generates a response with Llama 3.2's 3-billion-parameter model.

Once the model is ready, the tutorial uses Vercel AI SDK's generateText, streamText and generateObject APIs. Text generation returns a complete response, while streaming displays incoming tokens in a growing buffer. For structured output, the example passes a Zod schema with name and description fields. The speaker warns that models can return invalid JSON and trigger parsing errors, particularly models with fewer parameters.

The package's own API handles model discovery and preparation. The AI SDK provides the interaction API, which the speaker presents as a consistent interface across MLC and Apple's built-in model runtime. His positive performance assessment is qualitative; the tutorial provides no benchmark figures.