FastConformer (2023)
In one paragraph
Section titled “In one paragraph”NVIDIA’s NeMo team redesigned the front of the Conformer encoder. The original 4x subsampling with full 2-D convolutions was a large share of the compute and left 25 frames per second for the attention layers to process. FastConformer uses three stride-2 depthwise-separable convolutions for 8x subsampling, reducing the frame rate to 12.5 Hz and the attention cost by about 4x. The authors report 2.8x faster inference than the original Conformer at the same accuracy, scaling to billions of parameters without changing the block, and a way to transcribe hours-long audio in one pass by switching to limited-context attention plus a global token after training.
Key idea
Section titled “Key idea”Make the encoder cheaper where the cost is, not where the accuracy is. Attention cost is quadratic in the number of frames, so halving the frames from 4x to 8x subsampling quarters it, and the 80 ms output frames are still fine enough for BPE tokens (though too coarse for character-level CTC on fast speech, which is why NeMo’s FastConformer models use SentencePiece vocabularies). Replacing dense 2-D convolutions with depthwise-separable ones cuts the subsampling cost itself. For long-form audio, the trained model’s global self-attention is swapped for limited-context (local) attention with a single global token; a short fine-tune adapts the model, and the resulting encoder processes very long inputs with memory linear in length. The paper reports that this also improves accuracy on long-form test sets.
Why it matters for vaak
Section titled “Why it matters for vaak”The parakeet: backend is a FastConformer encoder with a TDT head. The 8x subsampling is why Parakeet has the lowest RTF of the GPU backends in vaak’s benchmark, and why a whole hour of audio can be transcribed in one forward pass instead of being cut into 30 s windows as Whisper requires. AI4Bharat’s more recent IndicConformer checkpoints also use NeMo’s FastConformer configuration, so this paper applies to indic-conformer: as well. The encoders tutorial page compares subsampling rates across Whisper, Conformer and FastConformer, and the streaming page explains how the limited-context attention variant relates to cache-aware streaming.
Numbers to remember
Section titled “Numbers to remember”- 8x time subsampling (80 ms per encoder frame) versus 4x in the original Conformer.
- 2.8x faster than the original Conformer, as reported in the abstract.
- Long-form transcription of audio up to 11 hours in one pass after replacing global attention with limited-context attention and a global token.
Citation
Section titled “Citation”Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, Boris Ginsburg. “Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition.” ASRU 2023. arXiv:2305.05084. https://arxiv.org/abs/2305.05084