CTC (Graves et al., 2006)

In one paragraph

Before 2006, training a neural network on speech required a frame-level alignment: for every 10 ms frame, which phoneme is it? That alignment came from an HMM system that had to be built first. Graves, Fernández, Gomez and Schmidhuber proposed a loss function that takes the network’s per-frame outputs and the target label sequence, and sums the probability over every way of aligning the two. The network is trained to maximise that sum. No alignment is ever given. They demonstrated it with a bidirectional LSTM on TIMIT phoneme recognition and beat an HMM baseline using the same features.

Key idea

Add one extra output symbol, blank. Let the network emit one symbol per frame. Define a many-to-one map from frame sequences to label sequences: merge repeated symbols, then delete blanks. The probability of a label sequence is the total probability of all frame sequences that map to it. That total can be computed exactly with a forward-backward recursion over a lattice of size T x (2U + 1), where T is the number of frames and U the number of labels, in the same way an HMM computes the likelihood of an observation sequence. The gradient falls out of the same recursion. Decoding is either best path (argmax per frame, then collapse) or prefix search, a beam search over label prefixes that merges alignments.

The blank symbol is what makes the collapse rule work for repeated labels and lets the network output nothing during silence or transitions. In practice, trained CTC networks emit each label as a single sharp spike and blank everywhere else.

Why it matters for vaak

CTC is the output head of wav2vec2: models and of the indic-conformer: CTC checkpoints. Its properties define their behaviour: parallel decoding, so these backends have the lowest RTF in the benchmark; conditional independence between frames, so they produce spelling errors on Devanagari that a language-model-equipped decoder would avoid; free frame-level timestamps; and no hallucination on silence, because blank is always available. CTC beam search with an n-gram language model (page 9 of the tutorial) is the standard way to add the missing language knowledge. The toy model in the tutorial is trained with torch.nn.CTCLoss, a direct implementation of this paper, and the CTC tutorial page walks through the collapse rule and the loss.

Numbers to remember

• On TIMIT phoneme recognition, the paper reports a label error rate of 30.51 % for CTC with prefix search decoding and 31.47 % with best-path decoding, against 35.21 % for the context-dependent HMM baseline it compares to.
• The lattice used by the forward-backward algorithm has 2U + 1 positions: every label with a blank between each pair and at both ends.

Citation

Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber. “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks.” Proceedings of the 23rd International Conference on Machine Learning (ICML), Pittsburgh, 2006.

This paper has no arXiv entry. Official PDF: https://www.cs.toronto.edu/~graves/icml_2006.pdf (also in the ACM Digital Library, doi:10.1145/1143844.1143891).