Benchmarks

Every number on this page comes from src/data/benchmarks.json. Each row carries one of two badges:

• reproduced by vaak means vaak bench produced the number on the hardware listed, and the raw JSON is in bench/results/.
• published by <source> means the number was copied from a paper, model card or leaderboard. The badge links to the source. We did not reproduce it.

There are no hand-typed numbers on this page.

Results

Model	Dataset	n	WER	CER	RTF	Peak VRAM (GB)	Load (s)	Provenance
faster-whisper:small	fleurs-en	30	8.1%	3.4%	0.341	–	1.5	reproduced by vaak
faster-whisper:base	fleurs-en	30	11.6%	5.2%	0.137	–	0.8	reproduced by vaak
faster-whisper:tiny	fleurs-en	30	14.2%	6.0%	0.065	–	0.5	reproduced by vaak
faster-whisper:small	fleurs-hi	30	61.1%	25.5%	0.832	–	1.9	reproduced by vaak
faster-whisper:base	fleurs-hi	30	105.9%	88.2%	0.232	–	36.4	reproduced by vaak
faster-whisper:tiny	fleurs-hi	30	109.0%	96.5%	0.246	–	1.0	reproduced by vaak

Observations from the first run

The first run is a laptop CPU smoke test over 30 FLEURS clips per language. It is too small to rank models, but two things in it are worth knowing before you pick a Whisper size for Hindi:

• Whisper tiny and base do not really transcribe Hindi. On most clips they either write the Hindi in Latin letters (“Kochi Arangu mein arster kendra kota”) or translate it into English. Every word then counts as an error, which is how WER goes above 1.0.
• Whisper small writes Devanagari and gets the gist, but still misspells roughly one word in two and one character in four on read speech. That is the gap IndicConformer and a Hindi LoRA are meant to close; those rows arrive with the GPU run.
• English is the easy case: small reaches about 8 percent WER on the same hardware.

Per-utterance hypotheses for every row are in bench/results/*.jsonl if you want to see the errors yourself.

Methodology

Datasets

Key	Source	Language	Notes
fleurs-hi	google/fleurs, hi_in split, streamed	hi	read speech, Wikipedia sentences
fleurs-en	google/fleurs, en_us split, streamed	en	read speech
cv-hi	mozilla-foundation/common_voice_17_0, hi	hi	crowd-sourced, needs a HF token
kathbath-hi	ai4bharat/Kathbath, Hindi	hi	read speech, Indian accents

Each dataset yields {"id", "audio", "text", "lang"} with audio resampled to 16 kHz mono. The --limit N flag takes the first N utterances of the test split so runs are cheap and repeatable. The n column in the table is that count.

Text normalisation

References and hypotheses go through the same normalize_text(text, lang) before scoring:

1. Unicode NFC normalisation.
2. Lowercase for Latin script.
3. Strip punctuation, including the Devanagari danda । and double danda ॥.
4. Collapse whitespace.

Hindi digits are left as they are. No number-to-word conversion is applied in either direction.

Metrics

• WER (word error rate): (substitutions + deletions + insertions) / reference words, computed on whitespace-split tokens after normalisation. Lower is better. Values above 1.0 are possible.
• CER (character error rate): the same edit distance on characters. More forgiving of tokenisation differences, which matters for Devanagari where word boundaries vary between writers.
• RTF (real-time factor): processing_time / audio_duration. 0.1 means one minute of audio takes six seconds. Lower is faster. Model load time is excluded and reported separately as load_s.
• peak_vram_gb: peak GPU memory during transcription, as reported by torch.cuda.max_memory_allocated or the backend’s equivalent. CPU-only backends report null.

Hardware and cost

Each run records the GPU, CPU, provider and hourly price in the hardware block. The first planned run targets an RTX 4090 on RunPod community cloud. The price we will attribute is the one shown on the RunPod pricing page at the time of the run; see the RunPod guide for the budget calculation.

Cost per audio hour is not a column in the JSON. You can derive it: price_usd_per_hr * rtf.

Reproduce it

pip install "vaak[all]"
vaak bench --models faster-whisper:small,indic-conformer:hi --datasets fleurs-hi,fleurs-en --limit 200

This writes bench/results/<timestamp>.json. Then:

python bench/report.py

merges every file in bench/results/ into website/src/data/benchmarks.json and this page updates on the next build. Reference numbers live in the reference array of that file with "reproduced": false and a source URL.

What the numbers do not say

• A WER difference of one or two points on 200 utterances is noise.
• FLEURS is read speech. Expect worse results on conversational or noisy audio.
• vaak does not train these models. The weights come from Whisper (OpenAI), AI4Bharat, NVIDIA, Meta and the Vosk project. We only measure how they behave through vaak’s wrappers on the hardware listed.