Skip to content
vaak

XLS-R (2021)

In one paragraph

Section titled “In one paragraph”

XLS-R takes the wav2vec 2.0 recipe and scales it along every axis at once: more languages (128), more unlabelled data (436 000 hours drawn from VoxPopuli, Multilingual LibriSpeech, Common Voice, BABEL and VoxLingua107) and more parameters (300 M, 1 B and 2 B). The paper shows that a single cross-lingually pretrained model, fine-tuned per task, improves speech recognition on BABEL, MLS, Common Voice and VoxPopuli, speech translation on CoVoST-2, and language identification on VoxLingua107. It also finds that, at sufficient model size, cross-lingual pretraining beats English-only pretraining even when translating English speech into other languages.

Key idea

Section titled “Key idea”

Scale and language coverage, not a new objective. The model and loss are those of wav2vec 2.0. The contributions are the curation of a large public multilingual corpus, a sampling strategy that upweights low-resource languages during pretraining so they are not drowned out by English and other high-resource languages, and the demonstration that the 1 B and 2 B models keep improving on low-resource languages where the 300 M model saturates. Transfer works because phonetic units are shared across languages: a model that has heard Marathi and Bengali has an easier time with Hindi than one that has only heard English.

Why it matters for vaak

Section titled “Why it matters for vaak”

Most open-weight wav2vec 2.0 checkpoints for Hindi on the Hugging Face hub are fine-tuned from an XLS-R model, usually the 300 M one. When you point wav2vec2:<hf-id> at one of them, the encoder is this paper’s. Its data mix explains their behaviour: Common Voice Hindi is in the pretraining set, so these models do well on Common Voice-style read speech and worse on spontaneous phone audio. XLS-R is also the baseline that AI4Bharat’s IndicWav2Vec and later the IndicVoices models improve on by pretraining on Indian-language audio specifically. For the wav2vec2: backend, the practical consequences are documented on the multilingual tutorial page: character-level Devanagari output, no punctuation, and a strong benefit from an n-gram language model at decode time.

Numbers to remember

Section titled “Numbers to remember”
  • 128 languages, 436 000 hours of publicly available unlabelled speech.
  • Three model sizes: 0.3 B, 1 B and 2 B parameters.
  • Speech translation into English on CoVoST-2: an average improvement of 7.4 BLEU over the previous state of the art across 21 directions.
  • Speech recognition: relative error-rate reductions of 14 to 34 % on average over the best prior results on BABEL, MLS, Common Voice and VoxPopuli.

Citation

Section titled “Citation”

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli. “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale.” Interspeech 2022. arXiv:2111.09296. https://arxiv.org/abs/2111.09296