Skip to content
vaak

Token-and-Duration Transducer, TDT (2023)

In one paragraph

Section titled “In one paragraph”

A standard RNN-T decoder visits every encoder frame and asks whether to emit a token or a blank. Most frames yield blank, so most of the decoder’s work is deciding to do nothing. Xu, Jia, Majumdar, Huang, Watanabe and Ginsburg extend the transducer’s joint network to predict, alongside each token, a duration: the number of frames that token occupies. At inference the decoder emits a token and then jumps forward by the predicted duration, skipping the blank decisions in between. The paper reports equal or better accuracy than RNN-T with up to 2.82x faster inference for speech recognition, and gains on speech translation and spoken language understanding as well.

Key idea

Section titled “Key idea”

Factor the joint output into two distributions. At grid cell (t, u) the joint network outputs P(token | t, u) over the vocabulary plus blank, and P(duration | t, u) over a small set such as {0, 1, 2, 3, 4}. Emitting token k with duration d moves to (t + d, u + 1); emitting blank with duration d moves to (t + d, u). Duration 0 allows several tokens on one frame, as in RNN-T. Training sums over all paths in this richer lattice with a modified forward-backward algorithm; the two distributions are assumed conditionally independent given the joint state, which keeps the loss tractable. The authors found the model learns sensible durations without supervision, and that larger maximum durations give larger speedups up to a point.

Why it matters for vaak

Section titled “Why it matters for vaak”

NVIDIA’s parakeet-tdt checkpoints, exposed in vaak as parakeet:<hf-id>, use this decoder on a FastConformer encoder. Three consequences show up in the benchmark and the API. Decoding is fast even though it is autoregressive, because most frames are skipped, so Parakeet’s RTF is dominated by the encoder. The duration output gives every token a start and an end frame, so result.segments timestamps come from the model itself rather than from an alignment post-process. And because the model is a transducer, it does not hallucinate on silence the way the Whisper decoder can. The transducer tutorial page walks through the decoding loop and shows where the frame skip happens.

Numbers to remember

Section titled “Numbers to remember”
  • Speech recognition: accuracy equal to or better than RNN-T, with inference up to 2.82x faster.
  • Speech translation on MUST-C: over 1 BLEU absolute improvement with 2.27x faster inference.
  • Spoken intent classification and slot filling: intent accuracy improved by up to over 1 % absolute, running up to 1.28x faster.
  • Implementation released in the NVIDIA NeMo toolkit; NeMo’s Parakeet TDT configurations use durations 0 to 4.

Citation

Section titled “Citation”

Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, Boris Ginsburg. “Efficient Sequence Transduction by Jointly Predicting Tokens and Durations.” ICML 2023. arXiv:2304.06795. https://arxiv.org/abs/2304.06795