RNN Transducer (Graves, 2012)

In one paragraph

CTC assumes each frame’s output is independent of the others given the audio. Graves’s 2012 paper removes that assumption while keeping CTC’s alignment-free training and left-to-right decoding. It adds a second recurrent network, the prediction network, that reads the labels emitted so far, and a joint function that combines the audio encoder’s state at frame t with the prediction network’s state after u labels to give the distribution over the next symbol, including blank. The model is trained by summing over all paths through the (t, u) grid. The paper evaluates on TIMIT phoneme recognition and on a synthetic task, and shows that the transducer improves on CTC alone.

Key idea

Two networks and a lattice. The transcription network f is any sequence encoder over the input (a bidirectional LSTM in the paper). The prediction network g is a unidirectional LSTM over the output labels, starting from a null symbol. At grid cell (t, u) the joint output is h(f_t, g_u) followed by a softmax over the vocabulary plus blank. Emitting blank moves to (t + 1, u); emitting label k moves to (t, u + 1) and feeds k to the prediction network. Any path from (0, 0) to (T, U) is an alignment, and their total probability is computed by forward-backward over the grid in O(T x U). Because nothing in the model looks at future input beyond what the encoder uses, decoding can run as audio arrives.

The paper also gives a beam search for the transducer that handles the fact that several labels can be emitted at one frame.

Why it matters for vaak

The transducer is the ancestor of every on-device dictation system and of NVIDIA’s Parakeet models, which vaak exposes as parakeet:. Its two defining properties, streaming and conditioning on output history, are why a transducer is the right choice for the /v1/stream WebSocket when a streaming-trained checkpoint is available, and why transducer output has fewer character-level spelling errors than CTC output. Its cost, a joint tensor of size batch x T x U x vocab during training, is why training one is memory hungry and why the TDT paper and NeMo’s fused loss exist. The transducer tutorial page walks through the greedy decoding loop from this paper.

The modern hybrid checkpoints that AI4Bharat ship as IndicConformer carry both a CTC head and an RNN-T head on one encoder, so this paper and the CTC paper together describe indic-conformer: as well.

Numbers to remember

The paper reports phoneme error rates on the TIMIT corpus for the transducer, for CTC alone, and for ablations with the prediction network removed. We have not reproduced those exact figures here; see Table 1 of the paper. The structural number worth remembering is the training cost: the joint network is evaluated at every one of T x U grid cells per utterance.

Citation

Alex Graves. “Sequence Transduction with Recurrent Neural Networks.” ICML 2012 Workshop on Representation Learning. arXiv:1211.3711. https://arxiv.org/abs/1211.3711