Voxtral Transcribe and TTS: local use and API tutorial

This tutorial explains how Voxtral Transcribe and Voxtral TTS fit around an LLM in a conversational application. Geoffrey, a Mistral AI developer relations engineer, walks through transcription and speech generation with playground demos and notebook code, then builds a French-to-English voice translation example.

The session covers newer 4B Voxtral speech models with open weights, rather than the earlier Small 24B checkpoint. He demonstrates browser transcription and explains how to run models locally with Transformers, noting that his CPU example is slow and that GPU inference is more efficient. For Voxtral 4B TTS 2603, he points to vLLM serving examples and says a Transformers implementation is not yet available at the time of the session.

The transcription examples cover complete-file processing, called offline here, and real-time streams. They explore word and segment timestamps, speaker diarization, and context biasing with up to 100 supplied terms. The TTS section covers preset voices, streaming output, and voice cloning using a 10 to 30 second reference recording.

For latency, Geoffrey proposes two transcription connections with different delays: a fast draft followed by a more accurate replacement. His translation app processes completed sentences through an LLM, generates speech for each sentence, and appends the audio. The reported timings describe these demos rather than guaranteed performance on other hardware.