Pavankumar Reddy Muddireddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system.IN PARTNERSHIP WITH MISTRAL AI:---This episode was produced in partnership with Mistral AI.Mistral AI: https://mistral.ai/---The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, passed to the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model becomes a dual-stream decoder that consumes audio and emits text at once, at a target delay down to 160ms, with slower streams in parallel for anything that can wait for more context.On generation, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places FSQ and flow matching in it. Tim presses on the priors underneath: why a mel spectrogram instead of raw waveform, what noise augmentation buys, and when acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes. Diarisation is emitted autoregressively inside the transcript rather than by a separate head, which makes streaming diarisation fragile — less context, late speaker changes, invented extra speakers. And because the architecture commits to what it has already predicted, one out-of-distribution mistake compounds into looping or skipped segments, which is what DPO corrects: the negative supervision pre-training and SFT cannot give.The last third is the argument Tim keeps returning to. Customers running voice agents over millions of sessions describe scaffolding, not a solved problem, with a sharp drop outside the top few languages. Cascades survive because each component stays separately adaptable, observable and constrainable. And voice alone is cognitive debt: absorbing information and deciding in one serial stream is harder than glancing at a menu. Voice becomes ubiquitous beside a screen, not instead of one.---TIMESTAMPS:00:00:00 Cold open00:00:46 Why Mistral moved into audio00:09:27 Inside Voxtral: trunk, encoder, dual streams00:20:22 Speech that works in real time00:30:52 How a voice becomes tokens00:39:59 Flow matching, FSQ and the new codec00:52:51 When speech models lose the speaker01:03:23 Correcting hallucinations with preferences01:12:12 Controlling synthetic speech01:20:06 Why cascades still win01:29:25 Speech in the wild01:33:46 Audio models as interfaces01:37:54 Why voice still needs a screen---REFERENCES:paper:[00:01:42] Mistral 7Bhttps://arxiv.org/abs/2310.06825[00:09:38] Voxtralhttps://arxiv.org/abs/2507.13264[00:14:41] Whisper: Robust Speech Recognitionhttps://arxiv.org/abs/2212.04356[00:19:11] Voxtral Realtimehttps://arxiv.org/abs/2602.11298[00:21:52] Delayed Streams Modeling (Kyutai)https://arxiv.org/abs/2509.08753[00:30:52] Voxtral TTShttps://arxiv.org/abs/2603.25551[00:32:38] SoundStream neural audio codechttps://arxiv.org/abs/2107.03312[00:34:59] Flow Matching for Generative Modelinghttps://arxiv.org/abs/2210.02747[00:37:03] EnCodec: High Fidelity Neural Audio Compressionhttps://arxiv.org/abs/2210.13438[00:37:42] Moshi and the Mimi codechttps://arxiv.org/abs/2410.00037[00:39:05] Finite Scalar Quantization (FSQ)https://arxiv.org/abs/2309.15505[01:03:33] Direct Preference Optimization (DPO)https://arxiv.org/abs/2305.18290dataset:[00:46:14] Mozilla Common Voicehttps://commonvoice.mozilla.org/en/datasetsorganization:[00:50:47] Hugging Facehttps://huggingface.co/

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.