4. Acoustic encoders

The encoder turns a log-mel matrix of shape (frames, 80) into a matrix of shape (frames / k, d) where every row is a vector that summarises what is being said around that moment, using context from both sides. Everything downstream (CTC, attention decoder, transducer) reads these vectors. Encoder quality is most of model quality.

Subsampling first

100 frames per second is more than needed. Phonemes last 50 to 100 ms, and a BPE token often covers 200 ms or more. Encoders begin with a small convolutional stack that reduces the frame rate:

• Whisper: two 1-D convolutions with GELU, the second with stride 2. Output 50 frames per second (20 ms per frame). 3 000 input frames become 1 500.
• Conformer (original): two 2-D convolutions with stride 2 each, 4x subsampling. 25 frames per second, 40 ms per frame.
• FastConformer: three stride-2 depthwise-separable 2-D convolutions, 8x subsampling. 12.5 frames per second, 80 ms per frame.
• wav2vec 2.0: seven 1-D convolutions over the raw waveform with a total stride of 320 samples, 50 frames per second.

Subsampling matters for cost because attention is quadratic in sequence length. Going from 4x to 8x halves the frames and quarters the attention cost. That is the core idea of FastConformer, which reports 2.8x faster inference than the original Conformer at the same accuracy.

The Transformer block

After subsampling, each frame is a vector of size d (384 for Whisper tiny, 768 for small, 1 280 for large). Positional information is added (sinusoidal in Whisper’s encoder, relative positions in Conformer). Then a stack of identical blocks:

x = x + MultiHeadSelfAttention(LayerNorm(x))
x = x + FeedForward(LayerNorm(x))

Self-attention lets every frame look at every other frame and weigh them. In an encoder it is bidirectional: frame 100 can see frame 900. That is why offline models are more accurate than streaming ones; they use the future.

The feed-forward layer is two linear layers with a non-linearity, expanding to 4d in the middle. Attention mixes information across time; the feed-forward layer processes each frame on its own.

Whisper’s encoder is exactly this, repeated 4 times (tiny) to 32 times (large).

The Conformer block

Plain self-attention is good at long-range context but poor at the fine local patterns that make up a consonant. Convolution is the opposite. The Conformer combines them in one block:

x = x + 0.5 * FeedForward(x)          # half-step FFN
x = x + MultiHeadSelfAttention(x)     # relative positional encoding
x = x + ConvModule(x)                 # pointwise conv -> GLU -> depthwise conv (k=31) -> BatchNorm -> Swish -> pointwise conv
x = x + 0.5 * FeedForward(x)          # half-step FFN
x = LayerNorm(x)

The two half-step feed-forward layers sandwiching the block are the “macaron” structure. The depthwise convolution with a kernel of around 31 frames gives each position a local receptive field of roughly 1.2 s at 40 ms per frame. In the original paper the large model (118.8 M parameters) reached 2.1 / 4.3 % WER on LibriSpeech test-clean / test-other without an external language model.

IndicConformer and Parakeet are Conformer or FastConformer encoders with different output heads. A sketch in PyTorch, omitting the attention details:

import torch
import torch.nn as nn

class ConvModule(nn.Module):
    def __init__(self, d, k=31):
        super().__init__()
        self.ln = nn.LayerNorm(d)
        self.pw1 = nn.Conv1d(d, 2 * d, 1)
        self.dw = nn.Conv1d(d, d, k, padding=k // 2, groups=d)
        self.bn = nn.BatchNorm1d(d)
        self.pw2 = nn.Conv1d(d, d, 1)

    def forward(self, x):                   # x: (batch, time, d)
        y = self.ln(x).transpose(1, 2)      # (batch, d, time)
        y = nn.functional.glu(self.pw1(y), dim=1)
        y = nn.functional.silu(self.bn(self.dw(y)))
        return x + self.pw2(y).transpose(1, 2)

x = torch.randn(2, 100, 256)
print(ConvModule(256)(x).shape)             # torch.Size([2, 100, 256])

wav2vec 2.0: learning the encoder without transcripts

Transcribed speech is expensive. Raw speech is cheap. wav2vec 2.0 pretrains the encoder with no transcripts at all:

1. A CNN turns the waveform into latent vectors, one per 20 ms.
2. Some time steps are masked before entering the Transformer.
3. The true latent at each masked step is quantised into a discrete code from a learned codebook.
4. The Transformer must pick the correct code for each masked step out of a set of distractors sampled from the same utterance (a contrastive loss).

The model learns phonetic structure because that is what predicts a masked segment from its neighbours. Afterwards a linear CTC layer is added and the whole network is fine-tuned on labelled data. With 960 hours of LibriSpeech the paper reports 1.8 / 3.3 % WER; with only 10 minutes of labels it still reaches 4.8 / 8.2 %. XLS-R scaled the same recipe to 128 languages and 436 000 hours, including Indian languages, which is why wav2vec2: checkpoints exist for Hindi.

Parameter and compute intuition

Rules of thumb, for a d-dimensional model with L blocks:

• Parameters per Transformer block are about 12 d^2 (4 d^2 attention, 8 d^2 feed-forward). Whisper small: d = 768, 12 encoder blocks, so about 85 M in the encoder.
• Compute per frame is about 2 x parameters FLOPs, plus the attention term 4 x d x T per frame where T is the sequence length. For a 30 s Whisper window T = 1 500, so attention is a modest share; for a 10 minute clip through a Conformer it would dominate, which is why long audio is chunked.
• Memory for activations scales with T^2 per head for attention. This is the limit when batching long utterances.

Practically: encoder cost grows linearly with audio length (plus the quadratic attention term), and is paid once. Decoder cost in attention models grows with transcript length and is paid sequentially. Page 10 shows how this appears in RTF numbers.

What the encoder output looks like

Each row of the output is a contextual embedding of a 20 to 80 ms slice. Nearby rows for the same phone are similar; rows for silence cluster together. If you train a linear classifier on these rows to predict phonemes you get surprisingly high accuracy, which is the whole reason a simple CTC head on top works at all.

Next: CTC.