Skip to content
vaak

Whisper (2022)

In one paragraph

Section titled “In one paragraph”

Radford and colleagues at OpenAI trained a plain encoder-decoder Transformer on 680 000 hours of audio paired with transcripts scraped from the internet, filtered by heuristics rather than hand-labelled. Multilingual transcription, translation to English, language identification, voice activity detection and timestamp prediction are all expressed as tokens in one output sequence, so one model does every task. Without any fine-tuning, the large model matches or beats supervised systems on many benchmarks and is far more robust to noise, accents and domain shift than models trained on LibriSpeech alone. The models and inference code were released, which is why Whisper is the default backend in almost every open STT tool, including vaak.

Key idea

Section titled “Key idea”

Weak supervision at scale beats clean supervision at small scale, and a multitask token format makes one decoder do everything. The decoder’s input starts with a prompt: <|startoftranscript|>, a language token such as <|hi|>, a task token <|transcribe|> or <|translate|>, and optionally <|notimestamps|>. Timestamp tokens, when enabled, interleave with text at 20 ms resolution and tell the decoder where to cut the next 30 s window of a long file. The encoder reads 30 s of 80-bin log-mel features through two convolutions and a Transformer; the decoder is a byte-level BPE language model with cross-attention. Robust decoding comes from heuristics: temperature fallback when the output is repetitive (compression ratio above 2.4) or low-confidence (average log-probability below -1.0), and conditioning on the previous window’s text.

Why it matters for vaak

Section titled “Why it matters for vaak”

faster-whisper: (a CTranslate2 port) and hf-whisper: (the transformers implementation, used for LoRA checkpoints) are both this model. The tutorial page on attention decoders covers its special tokens and failure modes; the fine-tuning page adapts it to Hindi. For vaak’s purposes the important facts are: Hindi is a small slice of the training set, so Hindi WER is several times English WER; the decoder is a strong language model, so it punctuates well and hallucinates on silence; every input is padded to 30 s, so RTF on short clips is poor; and the byte-level tokenizer spends many tokens per Devanagari word. The benchmark page compares Whisper sizes against Conformer models on fleurs-hi for exactly these reasons.

Numbers to remember

Section titled “Numbers to remember”
  • 680 000 hours of training audio, of which 117 000 hours cover 96 non-English languages and 125 000 hours are X-to-English translation.
  • Five sizes: tiny 39 M, base 74 M, small 244 M, medium 769 M, large 1 550 M parameters.
  • Input: 30 s windows, 80 mel bins, 25 ms window, 10 ms hop; encoder frame rate 50 Hz after a stride-2 convolution.
  • Decoding heuristics: temperature fallback over 0, 0.2, 0.4, 0.6, 0.8, 1.0; compression-ratio threshold 2.4; log-probability threshold -1.0; beam size 5.
  • Later releases (large-v3, large-v3-turbo) use 128 mel bins and are not covered by the paper.

Citation

Section titled “Citation”

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever. “Robust Speech Recognition via Large-Scale Weak Supervision.” arXiv:2212.04356, 2022; ICML 2023. https://arxiv.org/abs/2212.04356