6. Attention decoders and Whisper

CTC predicts each frame independently. An attention encoder-decoder does the opposite: it writes the transcript one token at a time, and each token is conditioned on all previous tokens and on the whole audio. This is the architecture of machine translation applied to speech, first shown to work at scale by Listen, Attend and Spell in 2015, and it is what Whisper uses.

Encoder-decoder

The encoder is the Transformer from page 4. The decoder is a stack of blocks like this:

y = y + MaskedSelfAttention(LayerNorm(y))       # over tokens produced so far
y = y + CrossAttention(LayerNorm(y), enc_out)   # queries from tokens, keys/values from audio
y = y + FeedForward(LayerNorm(y))

Masked self-attention lets token i see tokens 0..i-1, so the decoder is a language model. Cross-attention is where the decoder reads the audio: for each token it computes attention weights over the encoder frames and takes a weighted sum. When the model is working, those weights form a diagonal band, token i attending to the frames where it was spoken. Whisper’s word timestamps come from reading that band in a few selected attention heads and running dynamic time warping over it.

Training uses teacher forcing: feed the true previous tokens, predict the next one, cross-entropy loss. No alignment is needed because cross-attention learns it. Decoding is autoregressive: run the decoder, pick a token, append it, run again, until end-of-text.

Whisper’s multitask prompt

Whisper folds language identification, task selection and timestamps into the token stream. The decoder’s first tokens are a prompt, not output:

<|startoftranscript|> <|hi|> <|transcribe|> <|notimestamps|> नमस्ते ... <|endoftext|>

• <|startoftranscript|> always first.
• <|hi|>: one of 99 language tokens. If you pass language=None, Whisper runs the decoder one step on the first 30 s window and reads the distribution over language tokens. That is language ID. If you pass language="hi", the token is forced and the model does not guess. For Hinglish audio, forcing hi tends to produce Devanagari and forcing en produces Latin script (page 8).
• <|transcribe|> or <|translate|>: the latter outputs English regardless of the source language.
• <|notimestamps|>: if omitted, the model interleaves timestamp tokens (<|0.00|>, <|0.02|>, … in 20 ms steps) before and after each segment. That is how segment-level timestamps are produced, and it is also how Whisper decides where to cut a long file: the last timestamp in a window becomes the start of the next window.
• <|nospeech|>: predicted when the window contains no speech. Its probability is the no-speech score used for VAD-like filtering.

In vaak, backend.transcribe(audio, language="hi") sets the language token; language=None uses detection. result.language reports what was used.

Why Whisper is accurate, and why it hallucinates

The decoder is a language model trained on 680 000 hours of transcript text. This gives Whisper fluent punctuation, casing, and resistance to noise. It also gives it the ability to produce fluent text that has nothing to do with the audio. Known failure modes:

Hallucination on silence or music. Given a window with no speech, the decoder still has to emit something. It produces plausible sentences, often subtitles-style boilerplate such as “Thank you for watching” or, in Hindi, common phrases from its training data. Mitigation: VAD before Whisper (page 11), and the no_speech_threshold check.

Repetition loops. Once the decoder emits “the the the”, the language model prior makes the next “the” likely, and it continues until the token limit. More common in low-resource languages and on long windows. Mitigation: compression-ratio check (below), repetition_penalty, and no_repeat_ngram_size in HF generate.

Window boundary errors. Whisper sees 30 s at a time. A word cut by a boundary is often dropped or doubled. Mitigation: cut on silence with VAD, or use the timestamp-based seeking that faster-whisper does by default.

Prompt drift. Whisper can be conditioned on the previous window’s text. This helps consistency and hurts when the previous window was a hallucination, because the error propagates. faster-whisper’s condition_on_previous_text=False trades consistency for safety.

Temperature fallback

Whisper’s reference decoder tries greedy or beam decoding at temperature 0 first. It then checks two heuristics on the output:

• gzip compression ratio of the text above 2.4 means it is repetitive.
• average token log-probability below -1.0 means the model was unsure.

If either fails, it re-decodes the window with sampling at temperature 0.2, then 0.4, 0.6, 0.8, 1.0, until the checks pass or temperatures run out. Higher temperature breaks repetition loops at the cost of more random errors. faster-whisper exposes this as temperature=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0], compression_ratio_threshold=2.4 and log_prob_threshold=-1.0.

from vaak import get_backend, load_audio

audio = load_audio("noisy_hindi_call.mp3")
backend = get_backend("faster-whisper:large-v3-turbo")
result = backend.transcribe(audio, language="hi")
for s in result.segments:
    print(f"[{s.start:7.2f} -> {s.end:7.2f}] {s.text}")

If you see the same sentence repeated across several segments, that is a loop that the fallback did not catch. Re-run with VAD enabled, or split the file and transcribe the pieces.

Decoding cost

Each output token requires a full decoder pass over all previous tokens (with a key-value cache, the cost is linear per step). A 30 s Hindi window is often 80 to 150 tokens, so the decoder runs that many times. This is why Whisper’s RTF is dominated by the decoder, and why beam search with 5 beams is noticeably slower than greedy. faster-whisper reduces the cost by running the decoder in CTranslate2 with int8 or float16 weights and batched beams.

The distilled and turbo variants

large-v3-turbo keeps the 32-layer encoder of large-v3 but shrinks the decoder from 32 layers to 4, then fine-tunes. Decoding becomes several times faster with a small accuracy loss on English. The loss on Hindi and other lower-resource languages is larger; check the benchmarks before choosing it.

Next: Transducers: RNN-T and TDT.