Where Speech Models Actually Learn

Speech model fine-tuning recipes usually decide which layers get gradients before training even begins. Research reveals that encoders already capture nuanced semantics, speaking styles, and accent features that decoders systematically discard. Examining empirical benchmarks across Whisper and Parakeet, this discussion unpacks why freezing the acoustic encoder degrades multilingual accuracy, how high-rank LoRA rivals full fine-tuning, and why targeting specific attention heads can stop hallucinations without sacrificing transcription accuracy. Key sources: • Using fine-tuning and Min Lookahead beam search to improve Whisper (arXiv, full text read; retrieved 2026-11-08) — https://arxiv.org/abs/2309.10299 • Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models (arXiv, full text read; retrieved 2026-11-08) — https://arxiv.org/abs/2609.00727 • NVIDIA Parakeet TDT 0.6B v2 model card, created 2025-04-15, last modified 2026-06-29 (retrieved 2026-11-08) — https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2 Episode 6494 — myweirdprompts.com/n/6494 (permanent short link: myweirdprompts.com/889841) Listen on Android: MWP FM Player — myweirdprompts.com/apk