1. The pipeline end to end

This series explains how modern speech-to-text (STT) works, for a developer who writes Python but has not worked with audio or sequence models. Each page is short, uses plain terms, and includes code you can run. By the end you will understand what every vaak backend does under the hood, why Hindi is harder than English, and how to read a benchmark table without being fooled.

The whole pipeline in one picture

Every STT system, from a 2006 research model to Whisper, follows the same shape:

microphone / file
   |  1. capture and resample        -> 16 kHz mono float32 waveform
   v
feature extraction
   |  2. log-mel spectrogram         -> (frames, 80 or 128) matrix, one frame per 10 ms
   v
acoustic encoder
   |  3. CNN + Transformer/Conformer -> (frames / 4 or / 8, hidden) representation
   v
output head + decoder
   |  4. CTC, attention decoder, or transducer -> token ids
   v
tokenizer + post-processing
   |  5. BPE/SentencePiece detokenise, punctuation, timestamps
   v
text

Stages 1 and 2 are classical signal processing. Stages 3 and 4 are the neural network. Stage 5 is bookkeeping that matters more than it looks, especially for Indic scripts.

The hard part is stage 4. Audio has thousands of frames; the transcript has tens of tokens. Nobody tells the model which frame belongs to which letter. The three families of STT models (CTC, attention encoder-decoder, transducer) are three different answers to that alignment problem. Pages 5, 6 and 7 cover one each.

Where each vaak backend sits

vaak does not train production models. It wraps open-weight models behind one Python and HTTP interface. Every backend is a point in the design space above:

Backend spec	Input	Encoder	Output head	Decoding	Languages
faster-whisper:<size>	80 or 128 log-mel, 30 s windows	Transformer	Attention decoder, byte-level BPE	Beam search, temperature fallback	~100 incl. hi, en
hf-whisper:<hf-id>	same as above	Transformer	Attention decoder	Greedy or beam (HF generate)	same; used for LoRA checkpoints
indic-conformer:<lang>	80 log-mel	Conformer	CTC (some checkpoints also RNN-T)	Greedy	Indian languages
parakeet:<hf-id>	80 log-mel	FastConformer, 8x subsampling	TDT transducer	Greedy with frame skipping	en
wav2vec2:<hf-id>	raw waveform	CNN + Transformer (self-supervised pretrained)	CTC over characters	Greedy, optional n-gram LM	depends on checkpoint
vosk:<model-name>	MFCC / filterbank	Kaldi-style neural acoustic model	HMM states	WFST graph with n-gram LM	per model, CPU only

The spec string is <backend>:<model>. In Python:

from vaak import get_backend, load_audio

audio = load_audio("clip.wav")                 # float32 numpy, 16 kHz mono
backend = get_backend("faster-whisper:small")  # weights load on first use
result = backend.transcribe(audio, language="hi")
print(result.text)
print(result.rtf)        # processing_time / audio_duration
for seg in result.segments:
    print(f"{seg.start:6.2f} {seg.end:6.2f} {seg.text}")

load_audio is stage 1. Everything after that happens inside transcribe, and the backend decides the details of stages 2 to 5.

The three model families in one paragraph each

CTC (page 5). The encoder emits one probability distribution per frame over the vocabulary plus a special blank symbol. Decoding collapses repeats and drops blanks. Simple, fast, streams naturally, but each frame is predicted independently, so the model has no built-in language model. indic-conformer and wav2vec2 use it.

Attention encoder-decoder (page 6). An autoregressive decoder reads the whole encoded audio through cross-attention and writes tokens one at a time, like a translation model. Accurate and has a strong implicit language model, which is also why it can hallucinate text that is not in the audio. Whisper uses it.

Transducer (page 7). A prediction network over previous tokens and the encoder are combined by a joint network at every (frame, token) position. It streams like CTC but conditions on history like a decoder. Parakeet uses the TDT variant.

What “model size” means here

A rough mental model: Whisper tiny has 39 M parameters, small 244 M, large 1.55 B. Parameter count predicts memory and quality. Compute per second of audio depends on the frame rate after subsampling (page 4) and on whether decoding is parallel (CTC) or sequential (attention). This is why a 600 M Conformer CTC model can be faster than a 244 M Whisper on the same GPU. Page 10 explains the real-time factor (RTF) that vaak reports.

How to read this series

Pages 2 to 4 cover the signal path and the encoder. Pages 5 to 7 cover the three output heads. Pages 8 to 11 cover tokens, decoding, metrics and streaming. Pages 12 and 13 are hands-on: a toy CTC model trained from scratch, and LoRA fine-tuning of Whisper for Hindi. Page 14 links into the research section where each paper is summarised.

Next: Audio and sampling.