5. CTC: Connectionist Temporal Classification

The encoder gives you 250 vectors for a 10 s clip. The transcript is “नमस्ते दुनिया”, 13 characters. Which vector belongs to which character? You do not know, and labelling it by hand for thousands of hours is impossible. CTC, introduced by Graves et al. in 2006, is a loss function that trains the model without ever needing that alignment.

The trick: add a blank, then define a collapse rule

Put a linear layer on each encoder frame that outputs a probability over the vocabulary V plus one extra symbol, blank, written _. Now the network emits one symbol per frame, which makes an output sequence as long as the input. Define a function B that turns such a frame sequence into a transcript:

1. Collapse runs of the same symbol into one.
2. Delete all blanks.

frames:   h h _ e e _ _ l l _ l o o _
step 1:   h _ e _ l _ l o _
step 2:   h e l l o

The blank is essential for one reason: without it, “hello” could not be told apart from “helo”, because the two ls would merge. A blank between them keeps them separate. The blank also absorbs silence and transitions.

Many frame sequences map to the same transcript. CTC says: the probability of transcript y is the sum over all frame sequences π with B(π) = y of the product of per-frame probabilities.

P(y | x) = sum over π in B^-1(y) of  prod_t  p(π_t | x_t)

The loss is -log P(y | x). The model is free to put the characters wherever it likes in time, as long as the collapse gives the right text. In practice CTC models learn to fire each character as a single spike, surrounded by blanks.

Forward-backward: summing without enumerating

The number of valid π is astronomically large, but the sum has a dynamic-programming structure. Expand the target with blanks between every character and at both ends: _ h _ e _ l _ l _ o _. Define α(t, s) as the total probability of all prefixes that have consumed t frames and are at position s of the expanded target. Each α(t, s) depends only on α(t-1, s), α(t-1, s-1) and sometimes α(t-1, s-2) (skipping a blank is allowed unless the neighbours are the same character). Fill in a (T, 2U+1) table and the loss is the sum of the last two cells. A mirror-image β pass gives gradients. This is the same algorithm as the HMM forward-backward, and it costs O(T x U).

You never implement this yourself. torch.nn.CTCLoss does it, including on GPU.

torch.nn.CTCLoss

import torch
import torch.nn as nn

vocab = ["_", " ", "a", "b", "c", "d", "e", "h", "l", "o", "w"]   # index 0 = blank
blank = 0

T, N, C = 50, 2, len(vocab)         # frames, batch, classes
log_probs = torch.randn(T, N, C).log_softmax(dim=-1)   # (T, N, C), required layout
targets = torch.tensor([7, 6, 8, 8, 9,  10, 9, 8, 4, 2])  # "hello" + "wolca" flattened
input_lengths = torch.tensor([50, 45])
target_lengths = torch.tensor([5, 5])

ctc = nn.CTCLoss(blank=blank, zero_infinity=True)
loss = ctc(log_probs, targets, input_lengths, target_lengths)
print(loss.item())

Three things that bite everyone:

• The input layout is (T, N, C) with log-probabilities, not (N, T, C) and not raw logits.
• T must be at least the expanded target length. With 4x subsampling a 1 s clip gives 25 frames; a 30-character target with repeats can exceed that. zero_infinity=True stops such samples from producing NaNs.
• Blank must be a real class in your vocabulary, by convention index 0.

Greedy decoding

Take the argmax at every frame, then apply the collapse rule. Because each frame is predicted independently (the conditional independence assumption), this is a single vectorised operation and is what indic-conformer and wav2vec2 use by default.

import torch

def ctc_greedy_collapse(log_probs: torch.Tensor, blank: int = 0) -> list[int]:
    """log_probs: (T, C) for one utterance."""
    best = log_probs.argmax(dim=-1).tolist()
    out, prev = [], None
    for idx in best:
        if idx != blank and idx != prev:
            out.append(idx)
        prev = idx
    return out

lp = torch.randn(50, len(vocab)).log_softmax(-1)
ids = ctc_greedy_collapse(lp)
print("".join(vocab[i] for i in ids))

Greedy decoding has a subtle flaw. It finds the single most probable frame sequence, but the most probable transcript is a sum over many frame sequences. Two spellings may each have several alignments whose total beats the single best path. This is usually a small effect.

Beam search and prefix merging

CTC beam search keeps the top k transcript prefixes and, at each frame, extends them with every symbol while merging paths that collapse to the same prefix. Each prefix tracks two probabilities: ending in blank and ending in a non-blank. Beam search alone gives a small gain. Its real value is that you can add a language model score when a prefix gains a new word (shallow fusion, page 9). That is where CTC models recover most of what they lose by having no decoder.

Strengths and limits

Strengths:

• Non-autoregressive. Every frame is scored in one pass, so decoding is fast and batches well.
• Streams naturally. A frame’s output depends only on the encoder’s receptive field, so with a causal or chunked encoder you get output as audio arrives.
• Gives a free alignment: the frame at which each character spikes is a usable timestamp.
• Simple to train. No teacher forcing, no exposure bias.

Limits:

• Conditional independence. The model cannot learn “after ठ comes ा not ा्” except through what the encoder sees. Spelling errors are common in CTC output on Indic scripts for this reason.
• Output cannot be longer than the (subsampled) input, so very fast speech or 8x subsampling with character vocabularies can run out of frames. BPE vocabularies (page 8) help.
• No built-in language model, so rare words and names suffer without external fusion.

Transducers fix the independence assumption while keeping streaming. Attention decoders fix it by giving up streaming. The toy model on page 12 uses CTC because it is the simplest thing that works.