Why Speech Recognition Confidence Scores Lie

Speech recognition models like Whisper and Parakeet compute internal probabilities for every token, but turning those numbers into trustworthy word-level confidence is surprisingly brittle. Softmax saturation, pooling choices across sub-words, decoding fallbacks, and blank frames in CTC architectures distort confidence scores across the board. The mechanics behind word confidence calculations, why fine-tuning often worsens calibration, and how recent beam search inspection techniques attempt to quantify a model's true uncertainty are explored. Key sources: • NVIDIA Technical Blog, Entropy-Based Methods for Word-Level ASR Confidence Estimation . Retrieved 2026-11-08. — https://developer.nvidia.com/blog/entropy-based-methods-for-word-level-asr-confidence-estimation/ • NVIDIA Parakeet TDT 0.6B v2 model card, created 2025-04-15, last modified 2026-06-29, CC-BY-4.0 . Retrieved 2026-11-08. — https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2 • OpenAI Whisper `timing.py`, `WordTiming` dataclass and `find_alignment`, showing `np.mean(text_token_probs[i:j])` word probability . Retrieved 2026-11-08. — https://raw.githubusercontent.com/openai/whisper/main/whisper/timing.py Episode 6497 — myweirdprompts.com/n/6497 (permanent short link: myweirdprompts.com/817356) Listen on Android: MWP FM Player — myweirdprompts.com/apk