Listen, Attend and Spell (2015)

In one paragraph

Chan, Jaitly, Le and Vinyals applied the sequence-to-sequence-with-attention recipe from machine translation to speech recognition. A pyramidal bidirectional LSTM (the listener) encodes the audio into a shorter sequence of vectors; an LSTM decoder (the speller) emits characters one at a time, and at each step uses an attention mechanism to compute a weighted summary of the listener’s output. There is no CTC, no HMM, no pronunciation dictionary and no separate language model: the whole system is one network trained with cross-entropy on characters. On Google’s voice search task it came within a few points of a heavily engineered CLDNN-HMM system.

Key idea

Replace the explicit alignment of CTC and transducers with learned soft attention. For each output character, the decoder computes a distribution over encoder frames and reads a weighted sum. Because the decoder conditions on all previous characters, it is also a character-level language model, which is why it can spell words it has never seen and why it learns punctuation and casing conventions from data.

Two engineering details made it work. The pyramidal encoder halves the time resolution at each of three layers, so the attention mechanism has to attend over 8x fewer frames; without this, training did not converge well. And since the decoder is trained with teacher forcing (seeing the correct previous character), the paper samples from the model’s own predictions some of the time during training to reduce the mismatch at test time.

Why it matters for vaak

Whisper, the default backend in vaak, is an attention encoder-decoder. Nearly every strength and weakness listed on the Whisper tutorial page is already visible in this paper: the implicit language model that gives fluent output, the need for beam search and a length penalty because the decoder can stop early, the dependence on seeing the whole utterance before decoding (no streaming), and the tendency to emit plausible text when the attention loses track of the audio. The pyramidal subsampling is the direct ancestor of the convolutional subsampling in Whisper and Conformer, covered on the encoders page. When vaak reports that a Whisper backend has a higher RTF than a CTC backend on the same GPU, the reason is the sequential decoder this paper introduced.

Numbers to remember

• On a Google voice search task, LAS reached 14.1 % WER without a dictionary or language model and 10.3 % with language model rescoring, compared with 8.0 % for the state-of-the-art CLDNN-HMM system.
• The training set was about 3 million voice search utterances, roughly 2 000 hours.
• The pyramidal listener reduces the time resolution by a factor of 2 per layer, 8x over three layers.

Citation

William Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals. “Listen, Attend and Spell.” arXiv:1508.01211, 2015; also ICASSP 2016. https://arxiv.org/abs/1508.01211