Why Speech Models Forget Other Voices When Tuned on Yours

Fine-tuning a speech recognition model on a single speaker's voice often quietly destroys its ability to transcribe other accents, longer audio, and basic formatting. Examining evidence from Whisper adaptations in Hebrew and Swiss German, this audit covers how catastrophic forgetting erodes timestamps and structure even when target accuracy improves. It breaks down why general regression testing sets are critical, where low-rank adaptation can still drift, and how replay data paired with knowledge distillation protects baseline capabilities. Key sources: • Vander Eeckt & Van hamme, Continual Learning for Monolingual End-to-End Automatic Speech Recognition, EUSIPCO 2022, v4 revised 2023-03-09 . Retrieved 2026-11-08. — https://arxiv.org/abs/2112.09427 • Vander Eeckt & Van hamme, Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition, ICASSP 2023, v3 revised 2023-03-09 . Retrieved 2026-11-08. — https://arxiv.org/abs/2203.16082 • Fine-tuning Whisper on Low-Resource Languages for Real-World Applications (Swiss German case study) . Retrieved 2026-11-08. — https://arxiv.org/abs/2412.15726 Episode 6493 — myweirdprompts.com/n/6493 (permanent short link: myweirdprompts.com/744916) Listen on Android: MWP FM Player — myweirdprompts.com/apk