On-device AI: MLX, Gemma and Whisper use cases

Software Mansion's AI team discusses on-device inference through features in familiar apps. The speakers weigh privacy, offline access and reduced server work against model downloads, battery drain and heat. Their examples illustrate possible benefits; the discussion also describes services that send more complex work to a cloud backend.

Snapchat lenses show the camera-processing use case, with PyTorch models exported to TFLite or ONNX. Apple Photos provides an example of image and text embeddings for semantic search. Siri is described as combining local speech processing with private-cloud routing for harder tasks.

For language models, the speakers discuss fine-tuning small Gemma models for specific jobs and using MLX for efficient device execution. Their reported Grammarly throughput is an account of another implementation, not a benchmark performed in the video. Whisper Tiny is discussed as a compact model for real-time speech transcription, alongside a brief Parakeet mention.

The conversation also covers Chrome translation and summarization APIs, messaging-app scam detection, video-quality prediction and image cutouts. React Native ExecuTorch appears as the team's application library. Wearable translation closes the discussion, including the iPhone's role in AirPods processing and an acknowledged quality limitation.