Conformer (2020)

In one paragraph

Gulati and colleagues at Google asked which is the better encoder for speech, a Transformer (good at global context) or a convolutional network (good at local patterns), and answered: both, in every block. The Conformer block sandwiches a multi-head self-attention module and a convolution module between two half-step feed-forward modules. Trained as an RNN-T on LibriSpeech, the large model set a new state of the art without an external language model and the small 10 M parameter model matched much larger previous systems.

Key idea

One block, four modules, residual connections around each:

x = x + 1/2 FFN(x)
x = x + MHSA(x)          # relative positional encoding from Transformer-XL
x = x + Conv(x)          # pointwise conv, GLU, depthwise conv, batch norm, swish, pointwise conv
x = x + 1/2 FFN(x)
x = LayerNorm(x)

The “macaron” arrangement with two half-weighted feed-forward layers came from earlier work on Transformers and gave a consistent gain over one full FFN. The convolution module’s depthwise kernel (31 frames in the paper’s best configuration) models the local shape of a phone; attention supplies the sentence-level context that decides between homophones. The ablation tables show that removing the convolution module hurts more than removing anything else, and that placing convolution after attention beats placing it before or in parallel.

Why it matters for vaak

Two of vaak’s backends are Conformer encoders: indic-conformer: (AI4Bharat’s IndicConformer models, trained with NeMo) and parakeet: (NVIDIA’s FastConformer, a redesigned subsampling front end on the same block). The block is why these models reach Whisper-class accuracy with a fraction of the parameters and a non-autoregressive or lightly autoregressive output head, and therefore why they dominate the RTF column of the benchmark. The encoders tutorial page includes a PyTorch sketch of the convolution module. The 4x convolutional subsampling in this paper is also why a Conformer CTC model has one output frame per 40 ms, which bounds how many characters it can emit per second; the FastConformer paper pushes this to 8x.

Numbers to remember

• Three model sizes: Conformer S 10.3 M, M 30.7 M, L 118.8 M parameters.
• LibriSpeech test-clean / test-other WER without a language model: 2.7 / 6.3 % (S), 2.3 / 5.0 % (M), 2.1 / 4.3 % (L).
• With an external LSTM language model, the large model reaches 1.9 / 3.9 %.
• Depthwise convolution kernel size 32 in the paper’s sweep (31 and 32 are both common in later implementations); 4x time subsampling via two stride-2 convolutions before the Conformer stack.

Citation

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang. “Conformer: Convolution-augmented Transformer for Speech Recognition.” Interspeech 2020. arXiv:2005.08100. https://arxiv.org/abs/2005.08100