9. Decoding and language models

The network gives probabilities. Decoding turns them into one transcript. This is where accuracy, speed and controllability trade against each other, and where you can add domain knowledge without retraining anything.

Greedy

Take the best symbol at each step. For CTC this is argmax per frame plus collapse; for transducers it is the frame loop on page 7; for Whisper it is argmax per decoder step. Greedy is the default for every vaak backend except faster-whisper, which uses a beam of 5. Greedy is fast and usually within a few percent relative of beam search for a well-trained model.

Beam search

Keep the k best partial hypotheses (the beam) instead of one. At each step extend every hypothesis with every possible symbol, score, keep the top k. Beam search finds sequences whose first token was not the best choice but whose total probability is higher. For attention decoders this helps on long outputs where an early mistake commits the whole sequence. Cost is roughly k times greedy, mitigated by batching the beams.

Length matters. Longer sequences have lower probability simply because they have more factors, so a beam search over a decoder will prefer short outputs and may stop early. Length normalisation (divide the log-probability by length, or by length^alpha) counters this. Whisper uses a length penalty of 1.0 by default in faster-whisper.

# faster-whisper exposes the beam directly (vaak passes kwargs through)
from vaak import get_backend, load_audio
audio = load_audio("clip.wav")
b = get_backend("faster-whisper:small", beam_size=5, best_of=5, patience=1.0)
print(b.transcribe(audio, language="hi").text)

Shallow fusion with an n-gram language model

CTC models have no idea which words exist. An n-gram language model does: it stores the probability of each word given the previous n-1 words, estimated by counting a text corpus. KenLM is the standard tool: it builds an ARPA or binary model from a text file in minutes and queries it in microseconds.

Shallow fusion adds the LM score to the acoustic score during beam search:

score = log P_acoustic(tokens | audio) + alpha * log P_LM(words) + beta * word_count

alpha weights the LM (typically 0.3 to 1.0), beta is a word insertion bonus that counters the LM’s preference for fewer words. Both are tuned on a dev set. For CTC the fusion happens at word boundaries during prefix beam search; pyctcdecode implements this for HF wav2vec 2.0 models, and NeMo has an equivalent for Conformer-CTC.

# pip install pyctcdecode kenlm
from pyctcdecode import build_ctcdecoder
import numpy as np

labels = ["", " ", "a", "b", "c"]            # index 0 must be the CTC blank, here ""
decoder = build_ctcdecoder(labels, kenlm_model_path="hindi_3gram.bin", alpha=0.5, beta=1.0)
logits = np.random.randn(100, len(labels)).astype(np.float32)   # (T, C), from the model
print(decoder.decode(logits, beam_width=64))

Gains from a domain n-gram are large for CTC (often 10 to 30 % relative WER on in-domain text) and small or negative for Whisper, which already carries a strong LM in its decoder. For Hindi, the LM text must use the same script and normalisation as the model’s output, or fusion will fight the acoustics.

Hotwords and contextual biasing

A customer’s product names, a doctor’s drug list, the attendees of a meeting: these are words the model has rarely seen. Options, from cheapest to strongest:

• Whisper initial prompt. Pass the words as the text of a fake previous segment (initial_prompt="Kathbath, IndicVoices, FLEURS"). The decoder’s self-attention sees them and is more likely to reproduce their spelling. Cheap, works on names and spelling, unreliable, and the prompt eats into the 448-token context.
• Hotword boosting in beam search. Add a bonus when a hypothesis matches a prefix of a hotword (pyctcdecode hotwords=, hotword_weight=). Works for CTC and transducer models at decode time.
• LM biasing. Build a small n-gram from in-domain text and interpolate it with a general one.
• Fine-tuning. The strongest and the most expensive. Page 13.

Timestamps and word alignment

Each family produces time information differently.

CTC. The frame where a character spikes is its time. Multiply the frame index by the frame duration (40 ms for 4x subsampling) and you have a character-level timestamp, usually within one frame of the truth. Word times come from the first and last character of each word. No extra model needed.

Transducer / TDT. The frame index at which a token was emitted is its time. TDT’s duration prediction gives a start and end per token.

Whisper. Segment timestamps come from the timestamp tokens the decoder emits (page 6). Word timestamps are not predicted; they are read off the cross-attention weights of a handful of alignment heads using dynamic time warping, then shifted by a fixed offset. They are usable but drift on long words and fail when the model hallucinates. faster-whisper’s word_timestamps=True does this.

Forced alignment. If you already have the correct text and only want timings, run a CTC model with the target fixed and compute the best path with torchaudio.functional.forced_align. This is how subtitle tools and the tiny model on page 12 can produce character-level timings. Multilingual CTC alignment models (such as the MMS-based aligner in torchaudio) handle Devanagari.

In vaak, every backend fills result.segments with start, end and text in seconds; word-level timing is backend specific and exposed through backend kwargs.

Confidence

Decoders also give you how sure the model was. Average token log-probability, the no-speech probability (Whisper), and the margin between the best and second-best beam are all usable confidence signals. They are poorly calibrated across models and languages, so compare within one model only. In practice, a Whisper segment with average log-prob below about -1.0 is worth flagging for review.

Where the time goes

Decoder	Cost per utterance	Parallel over batch
CTC greedy	one argmax over (T, C)	yes
CTC beam + n-gram	T x beam x LM lookups, CPU	partly
Transducer greedy	T joint evaluations plus one per token	per utterance
Whisper greedy	one decoder pass per token	per window
Whisper beam 5	about 5x greedy	beams batched

On a GPU, CTC greedy decoding of an hour of audio takes well under a second. Whisper beam search on the same hour takes minutes. Page 10 shows how to measure this honestly.

Next: Metrics: WER, CER and RTF.