wav2vec 2.0 (2020)
In one paragraph
Section titled “In one paragraph”Baevski, Zhou, Mohamed and Auli at Facebook AI showed that an encoder pretrained on unlabelled speech alone, then fine-tuned with CTC on a small labelled set, matches or beats fully supervised systems. The model reads the raw waveform through a convolutional feature encoder, masks spans of the resulting latent vectors, and trains a Transformer to identify the true quantised latent at each masked position among distractors. After pretraining on 53 000 hours of unlabelled LibriVox audio, fine-tuning on all 960 hours of LibriSpeech gave state-of-the-art WER, and fine-tuning on only 10 minutes of labelled audio still gave a usable recogniser.
Key idea
Section titled “Key idea”A contrastive task over learned discrete units. The feature encoder produces one latent vector z_t every 20 ms. Two things happen to it: it is quantised into q_t by picking entries from learned codebooks (Gumbel-softmax, product quantisation with 2 codebooks of 320 entries), and, for masked positions, it is hidden from the Transformer. The Transformer’s output c_t at a masked position must score the true q_t higher than K distractors sampled from other masked positions in the same utterance. A diversity loss keeps the codebook entries in use. The quantisation forces the targets to be discrete, phone-like units, so the Transformer learns phonetic structure without ever seeing a transcript. Fine-tuning adds a linear CTC layer over characters and trains the whole network on labelled data, with the feature encoder frozen.
Why it matters for vaak
Section titled “Why it matters for vaak”The wav2vec2: backend loads any Hugging Face Wav2Vec2ForCTC checkpoint. Everything about its behaviour follows from this paper: raw waveform input (no mel spectrogram; see the features page), 20 ms output frames, character-level CTC output with no casing or punctuation, and a strong dependence on the pretraining data’s languages. For Hindi, the relevant checkpoints descend from XLS-R and from AI4Bharat’s IndicWav2Vec, which apply this recipe to Indian-language audio. The paper’s 10-minute result is also the reason low-resource fine-tuning is realistic at all: it is the evidence behind the tiny model page’s advice to start from a self-supervised checkpoint rather than from scratch.
Numbers to remember
Section titled “Numbers to remember”- Two sizes: Base, 95 M parameters, 12 Transformer blocks; Large, 317 M parameters, 24 blocks.
- Feature encoder: seven convolutional layers, output stride about 20 ms with a receptive field of about 25 ms.
- Masking: each time step is chosen as a span start with probability 0.065, span length 10, so roughly half of all time steps are masked.
- Quantisation: 2 codebooks with 320 entries each.
- LibriSpeech test-clean / test-other WER with the Large model pretrained on LibriVox-60k: 1.8 / 3.3 % with all 960 hours of labels; 4.8 / 8.2 % with 10 minutes of labels.
Citation
Section titled “Citation”Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli. “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.” NeurIPS 2020. arXiv:2006.11477. https://arxiv.org/abs/2006.11477