10. Metrics: WER, CER, RTF

A benchmark number is only meaningful if you know how it was computed. This page defines the metrics vaak reports and shows where they mislead.

Word error rate

Align the hypothesis to the reference with the minimum number of edits (Levenshtein distance over words), then:

WER = (S + D + I) / N

S substitutions, D deletions, I insertions, N words in the reference. WER can exceed 100 % when the hypothesis is much longer than the reference.

Worked example, English:

REF: the  cat  sat  on  the  mat
HYP: the  cat  sat  on       mat  today
                            D         I

One deletion, one insertion, N = 6, WER = 2 / 6 = 33.3 %. Note that a single missing short word and a single extra word together cost a third of the sentence.

Worked example, Hindi:

REF: मुझे  कल  बाज़ार  जाना  है
HYP: मुझे  कल  बाजार  जाना  हैं
                S              S

Two substitutions, N = 5, WER = 40 %. The first error is a missing nukta (ज़ vs ज), the second a stray chandrabindu. A Hindi reader would call this transcript correct. WER calls it 40 % wrong. This is the whole argument for CER.

Character error rate

Same formula, over Unicode code points instead of words, including spaces:

CER = (S + D + I) / number of characters in the reference

For the Hindi example: the reference has 22 code points including spaces (count them with len() after NFC normalisation). Two code points differ, so CER is about 9 %. CER rewards partially correct words, is insensitive to word-segmentation disagreements, and tracks how usable a transcript is for an Indic language much better than WER. vaak reports both; for Hindi, read CER first.

CER has its own blind spot: a wrong matra on a function word is as bad as a wrong matra on a name, and it says nothing about whether the words exist.

jiwer

# pip install jiwer
import jiwer

ref = "the cat sat on the mat"
hyp = "the cat sat on mat today"
out = jiwer.process_words(ref, hyp)
print(out.wer, out.substitutions, out.deletions, out.insertions)   # 0.333 0 1 1

ref_hi = "मुझे कल बाज़ार जाना है"
hyp_hi = "मुझे कल बाजार जाना हैं"
print(jiwer.wer(ref_hi, hyp_hi), jiwer.cer(ref_hi, hyp_hi))

vaak wraps the same computation with its normaliser:

from vaak.metrics import wer, cer, normalize_text

r = normalize_text("मुझे कल बाज़ार जाना है।", lang="hi")
h = normalize_text("मुझे कल बाजार जाना हैं", lang="hi")
print(wer(r, h), cer(r, h))

Normalisation pitfalls

WER between two systems is only comparable if both outputs and the reference pass through the same normaliser. Things that silently change a number by several points:

• Punctuation and case. Whisper outputs “Hello, world.” and a CTC model outputs “hello world”. Without normalisation the CTC model is penalised for every punctuation mark. vaak lowercases Latin text and strips punctuation including the Devanagari danda । and ॥.
• Unicode form. ज़ can be one code point (U+095B) or two (U+091C U+093C). NFC normalisation maps both to the same form. Without it, identical-looking text scores as an error.
• Numerals. vaak leaves digits as they are. If a reference has “पच्चीस” and the model writes “25”, that is counted as an error, and it is reported as such because fixing it requires an inverse text normaliser that is itself a source of errors. Whisper’s English normaliser does convert numbers; results computed with it are not comparable to results without it. Benchmark tables must say which was used.
• Whitespace and zero-width characters. Devanagari text from the web contains zero-width joiners and non-joiners. Strip or keep them consistently.
• Script choice for code-switched words. No normaliser fixes “meeting” vs “मीटिंग”. Report it as a known source of error.

Rule: never compare a WER from one paper with a WER from another unless the normaliser is the same. The vaak benchmark page labels each number either “reproduced by vaak” (our normaliser) or “published by ” (theirs).

Real-time factor

RTF = processing_time / audio_duration

RTF 0.1 means one hour of audio in six minutes. Lower is faster. result.rtf reports it for every transcription, measured from the start of feature extraction to the end of decoding, excluding model loading.

RTF depends on everything: GPU, batch size, precision (float16 vs int8), beam size, audio length (Whisper pads to 30 s, so short clips look slow), and whether the GPU was warm. Compare RTF only within one hardware line of the benchmark table, and only with the model loading time (load_s) stated separately.

Latency versus throughput

They are different numbers and products need different ones.

• Latency is how long one user waits for one result. For a file upload it is load + processing; for streaming it is the delay between a word being spoken and it appearing (page 11). Latency is governed by the sequential parts: decoder steps, chunk size, lookahead.
• Throughput is audio hours per wall-clock hour across all users. It is governed by batching. A CTC model can batch 64 utterances in one forward pass; a Whisper decoder can batch beams but still runs one step per token.

A system can have excellent throughput and poor latency (batching Whisper on a large GPU) or the reverse (a streaming transducer on CPU). vaak bench reports RTF for single-utterance processing, which is closer to a latency measurement. Peak VRAM (peak_vram_gb) is reported because it bounds how many copies or how large a batch fits on one card.

Reading a benchmark row

{"model": "faster-whisper:large-v3-turbo", "dataset": "fleurs-hi", "n": 200,
 "wer": 0.31, "cer": 0.12, "rtf": 0.05, "peak_vram_gb": 3.1, "load_s": 4.2}

Read this as: on 200 FLEURS Hindi clips with vaak’s normaliser, about one word in three differs from the reference but only about one character in eight does; it processes audio 20x faster than real time on the stated GPU, needs 3.1 GB, and takes 4 s to load. Whether 31 % WER is acceptable depends on whether the differences are nuktas or missing words. Look at the CER, then look at the actual outputs.

Next: Streaming.