Skip to content
vaak

Vaani Benchmark V1.0 (2026)

This summary is based on the paper’s abstract and the parts of the text we could access at the time of writing. Where we are not sure, we say “see paper”.

In one paragraph

Section titled “In one paragraph”

Vaani Benchmark V1.0, from the Vaani project at IISc and ARTPARK, is a Hindi evaluation set built from spontaneous speech collected across 102 districts of India. Speakers described images, so every recording is paired with the image that prompted it, which makes the set usable for image retrieval with speech or text queries as well as for speech recognition. Each utterance carries three independent transcriptions. The authors evaluate a range of current ASR systems and show that scoring against a single reference overstates their errors: when all three references are used, measured WER drops substantially. They also report that text-based image retrieval outperforms audio-based retrieval on the same data.

Key idea

Section titled “Key idea”

Multiple references change the number, and for Hindi the change is large. Spontaneous Hindi has no single correct transcript: spelling variants, nukta and chandrabindu usage, numerals, word segmentation and the script used for English loanwords all vary between competent annotators. A single reference therefore counts annotator preference as model error. By collecting three transcripts per utterance and scoring each hypothesis against the closest one (or otherwise combining them; see paper for the exact scoring protocol), the benchmark separates genuine recognition errors from convention mismatches. Geographic breadth matters for the same reason: a model evaluated only on studio Hindi from one city has not been evaluated on Hindi.

Why it matters for vaak

Section titled “Why it matters for vaak”

This paper is the clearest published evidence for two choices in vaak’s benchmark methodology. First, Hindi WER against one reference is pessimistic, and the gap between systems on one reference may be smaller than it looks; vaak reports CER alongside WER and normalises references and hypotheses identically to reduce, but not remove, this effect (see the metrics tutorial). Second, the systems compared in the paper include open models that vaak wraps (IndicConformer and Whisper large-v3) alongside commercial APIs, so the paper’s ranking is a useful cross-check for vaak’s fleurs-hi and kathbath-hi results, keeping in mind that the paper’s numbers are published, not reproduced by vaak. The dataset itself is openly licensed and is a candidate for a future vaani-hi entry in bench/datasets.py.

Numbers to remember

Section titled “Numbers to remember”

From the abstract and accessible text:

  • 102 districts; three transcriptions per utterance.
  • About 10.9 hours of audio, 5 050 utterances, 3 701 images, 1 062 speakers (see paper to confirm these totals and the split definitions).
  • Ten ASR systems evaluated. The average WER across them falls from 24.80 % with a single reference to 17.76 % with three references.
  • Released under CC BY 4.0 on the Hugging Face hub as ARTPARK-IISc/Vaani-Benchmark-V1.0.

Per-system WER, the retrieval metrics, and the exact multi-reference scoring rule: see paper.

Citation

Section titled “Citation”

Sujith Pulikodan, Agneedh Basu, Saurabh Kumar, Pranav Bhat, Pavan Kumar J, Nihar Desai, Prasanta Kumar Ghosh. “Vaani Benchmark V1.0: An Inclusive Multimodal Benchmark Dataset for Hindi.” arXiv:2606.21408, June 2026 (v2 September 2026). https://arxiv.org/abs/2606.21408