8. Tokenization and multilingual speech

The output head predicts tokens, not words. What a token is decides how long the output sequence is, how rare words are handled, and how badly a model does on scripts it rarely saw. For Hindi this page explains most of the gap you will see in the benchmarks.

Characters, subwords, bytes

Characters. The vocabulary is every letter in the language plus space. Small (Hindi needs roughly 70 to 130 symbols depending on how you count matras and conjunct marks), never out-of-vocabulary, long output sequences. wav2vec 2.0 fine-tuning and the toy model on page 12 use characters.

Subwords (BPE, SentencePiece). Start from characters and repeatedly merge the most frequent adjacent pair until the vocabulary reaches a target size, typically 128 to 1 024 for CTC and transducer models and 50 000 for Whisper. Frequent words become one token; rare words split into pieces. Shorter output sequences mean fewer decoder steps and fewer CTC frames needed. SentencePiece is the usual tool because it works on raw text without a pre-tokenizer and treats space as an ordinary symbol (shown as ▁). NeMo Conformer and Parakeet models use SentencePiece unigram or BPE.

Byte-level BPE. Whisper uses the GPT-2 tokenizer design: the base alphabet is the 256 byte values, so any Unicode string can be encoded and there is never an unknown token. The multilingual Whisper vocabulary was refit on the training transcripts, so common Hindi words have their own tokens, but less common ones fall back to byte pieces. A Devanagari character is 3 bytes in UTF-8, so a rare Hindi word can cost 10 or more tokens, and each one is a decoder step where the model can go wrong.

from transformers import WhisperTokenizer
tok = WhisperTokenizer.from_pretrained("openai/whisper-small", language="hindi", task="transcribe")
for text in ["hello world", "नमस्ते दुनिया", "कृत्रिम बुद्धिमत्ता"]:
    ids = tok(text, add_special_tokens=False).input_ids
    print(len(ids), text, tok.convert_ids_to_tokens(ids))

The English sentence encodes to 2 tokens. The Hindi ones encode to several each, with byte fragments like à¤ visible in the token strings. That is the byte-level fallback at work.

Devanagari details that matter

Devanagari is an abugida. Each consonant carries an inherent vowel; vowel signs (matras) modify it; the virama (halant, ्) removes it to form conjuncts. In Unicode a syllable such as क्ष is three code points: क + ् + ष. Consequences for STT:

• Character error rate counts code points, so one wrong matra is one error, while WER counts the whole word as wrong.
• The same sound can be written several ways: ज़ as one precomposed code point (U+095B) or as ज + nukta (U+091C U+093C). Unicode NFC normalisation maps both to the decomposed form, which is why vaak.metrics.normalize_text applies NFC before scoring. It does not fix genuinely different spellings, such as a nukta that one annotator wrote and another omitted.
• Chandrabindu vs anusvara (ँ vs ं), and ि vs ी, are both frequently confused in transcripts and in reference texts. Models learn whatever their training data did.

Hinglish and code-switching

Spoken Hindi in India mixes English freely: “मुझे कल meeting में join करना है”. A transcript can write the English words in Latin script, in Devanagari (मीटिंग), or in a mix, and reference datasets disagree with each other. The result:

• Whisper with <|hi|> forced tends to produce everything in Devanagari. With <|en|> forced it produces romanised Hindi. With auto-detect it picks one per 30 s window and may flip between windows.
• IndicConformer models trained on Devanagari-only transcripts cannot output Latin script at all.
• WER against a reference that used Latin script for “meeting” will count a correct Devanagari “मीटिंग” as a substitution.

For evaluation this is a normalisation problem (page 10). For products, decide the script convention first and pick a model or fine-tune to match it.

Language identification

Whisper performs language ID by reading the decoder’s distribution over language tokens after the start token (page 6). It is good on clean 30 s windows and poor on short clips, on code-switched speech, and between closely related languages (Hindi vs Urdu is a known confusion because they are the same spoken language in different scripts). CTC and transducer models have no language ID; indic-conformer:<lang> is a per-language model and you choose. vaak’s language argument accepts a code or None; the REST API accepts language=auto. For Hindi content, pass hi explicitly whenever you know it.

Why Hindi WER is high

Published Whisper numbers for Hindi are several times worse than for English. Several causes stack:

1. Data volume. Whisper’s training set is dominated by English. Hindi is a small slice. Self-supervised models like XLS-R help by learning the acoustics from unlabelled audio, but the mapping to text still needs labelled hours.
2. Spelling variants. Hindi has no single enforced spelling for many words. References from different annotators differ; the Vaani benchmark measured this by collecting three transcripts per utterance and found that single-reference WER overstates errors.
3. Numerals. “पच्चीस” may be written as a word, as 25, or as २५. Whisper tends to output digits; many references spell numbers out. Every mismatch is a word error.
4. Tokenization cost. More tokens per word means more decoder steps and more chances for a one-byte error, which then corrupts the whole word.
5. Transliteration and loanwords. English words in Devanagari have no agreed spelling (मीटिंग / मीटींग).
6. Segmentation. Compounds and postpositions can be written joined or separated (कर रहा vs कररहा). WER counts a word boundary difference as two errors.
7. Acoustic conditions. Much Hindi benchmark data is spontaneous, phone-recorded speech; English benchmarks like LibriSpeech are read audiobooks.

This is why vaak reports CER alongside WER for Indic languages, and why a Hindi WER of 20 % can correspond to a transcript that a reader finds nearly perfect.

Punctuation and casing

Whisper outputs punctuation and casing because its training transcripts had them. CTC and transducer models usually output lowercase (or script-only) text with no punctuation, because their training transcripts were normalised. Neither is wrong; scoring must normalise both the same way.

Next: Decoding and language models.