12. Build a tiny STT from scratch

Everything so far describes models trained by others. This page builds one, end to end, so that each piece from the earlier pages becomes concrete. The script is train/tiny_ctc_from_scratch.py in the vaak repository. It is the only model vaak trains from scratch, and it is a toy. Read the “What to expect” section before you run it so you are not disappointed.

The plan

FLEURS English clips (a few hundred)
  -> load_audio, 16 kHz mono
  -> log-mel, 80 bins, 10 ms hop                        (page 3)
  -> Conv2d x2, stride 2 each: 4x time subsampling      (page 4)
  -> 2-layer bidirectional GRU, hidden 256              (a small encoder)
  -> Linear to |vocab| + 1                              (CTC head)
  -> torch.nn.CTCLoss                                   (page 5)
  -> greedy collapse at eval                            (page 5)
  -> CER / WER with vaak.metrics                        (page 10)

Character vocabulary: the 26 lowercase letters, apostrophe and space, plus blank at index 0. No tokenizer, no language model, no pretraining.

Data

FLEURS (google/fleurs, config en_us) has about 10 s read-speech clips with clean transcripts. The script streams the dataset from the Hugging Face hub, takes the first few hundred training clips (the --n-train flag, default 500) and fifty held-out test clips (--n-test), and normalises the text with normalize_text(text, lang="en") so the references contain only characters in the vocabulary. A few hundred clips is roughly an hour of audio.

from datasets import load_dataset
from vaak.metrics import normalize_text

ds = load_dataset("google/fleurs", "en_us", split="train", streaming=True)
for ex in ds.take(3):
    print(ex["audio"]["sampling_rate"], len(ex["audio"]["array"]) / 16000, normalize_text(ex["transcription"], "en"))

The model

import torch
import torch.nn as nn

class TinyCTC(nn.Module):
    def __init__(self, n_mels=80, vocab_size=29, hidden=256):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 32, 3, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(32, 32, 3, stride=2, padding=1), nn.ReLU(),
        )
        conv_out = 32 * ((n_mels + 3) // 4)           # channels x reduced mel bins
        self.proj = nn.Linear(conv_out, hidden)
        self.gru = nn.GRU(hidden, hidden, num_layers=2, bidirectional=True, batch_first=True)
        self.head = nn.Linear(2 * hidden, vocab_size)  # vocab includes blank at 0

    def forward(self, feats, lengths):
        # feats: (batch, T, n_mels) log-mel; lengths: frames per item
        x = self.conv(feats.unsqueeze(1))              # (batch, 32, T/4, n_mels/4)
        b, c, t, f = x.shape
        x = x.permute(0, 2, 1, 3).reshape(b, t, c * f)
        x = torch.relu(self.proj(x))
        lengths = (lengths + 3) // 4
        packed = nn.utils.rnn.pack_padded_sequence(x, lengths.cpu(), batch_first=True, enforce_sorted=False)
        x, _ = self.gru(packed)
        x, _ = nn.utils.rnn.pad_packed_sequence(x, batch_first=True)
        return self.head(x).log_softmax(-1), lengths   # (batch, T/4, vocab)

model = TinyCTC()
print(sum(p.numel() for p in model.parameters()) / 1e6, "M parameters")   # about 3 M

The two convolutions do what Whisper’s and Conformer’s subsampling stacks do: reduce 100 frames per second to 25 and learn local time-frequency patterns. The bidirectional GRU stands in for the Transformer. It sees both directions, so this is an offline model. The linear head gives one distribution per 40 ms frame.

Training loop

ctc = nn.CTCLoss(blank=0, zero_infinity=True)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

for epoch in range(epochs):
    for feats, feat_lens, targets, target_lens in loader:     # targets flattened, page 5
        log_probs, out_lens = model(feats, feat_lens)
        loss = ctc(log_probs.transpose(0, 1), targets, out_lens, target_lens)   # (T, N, C)
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 5.0)
        opt.step()

Details the script adds: a character vocabulary built from the training transcripts; length-bucketed batches to reduce padding; gradient clipping at 5.0; greedy collapse and vaak.metrics.wer on the held-out clips after every epoch, with a few decoded examples printed so you can watch the output turn from blanks into letters into words. It deliberately leaves out SpecAugment, learning-rate schedules and checkpointing so the file stays under 300 lines. On a laptop GPU an epoch over 500 clips takes about a minute; on CPU about ten.

Run it:

python train/tiny_ctc_from_scratch.py --n-train 500 --n-test 50 --epochs 20   # streams FLEURS English
python train/tiny_ctc_from_scratch.py --synthetic --max-steps 3                 # smoke test, no download

What to expect

Be honest with yourself about what an hour of audio and about 3 M parameters can do.

• Epochs 1 to 5. Loss falls fast, output is all blank. CER is 100 %. This is normal: the model first learns that blank is almost always right.
• Epochs 5 to 15. Spaces and the most common letters appear, roughly in the right places. Output looks like t e s a o th .... CER around 60 to 70 %.
• Epochs 15 to 40. Partial words: the presdent sad tht the for “the president said that the”. CER around 30 to 45 % on validation. WER stays above 80 % because almost every word has one wrong letter.
• Beyond that. Validation CER stops improving while training CER keeps falling. That is overfitting; you have seen all the data the model will ever see.

This is a model that reaches partial words, not a product. It demonstrates the alignment-free training that CTC makes possible, and nothing else. It will not work on Hindi without changing the vocabulary, and with a few hundred Hindi clips it will do worse than on English because Devanagari has more symbols to learn per hour of audio.

What a real model would need

The distance from this toy to indic-conformer:hi or Whisper is not cleverness, it is data and compute.

	This toy	wav2vec 2.0 base fine-tune	Conformer-CTC medium	Whisper large-v3
Parameters	about 3 M	95 M	about 120 M	1 550 M
Labelled audio	about 1 h	10 min to 960 h (plus 960 h unlabelled)	hundreds to thousands of hours	680 000 h weakly labelled
Vocabulary	28 characters	characters	128 to 1 024 SentencePiece	51 865 byte-level BPE
Encoder	2-layer BiGRU	12-layer Transformer	16 to 18 Conformer blocks	32-layer Transformer
Training compute	minutes on one GPU	days on 8 GPUs (pretraining)	days on 8 GPUs	thousands of GPU-days

Concretely, to get a usable Hindi CTC model from scratch you would need on the order of a thousand hours of transcribed speech (IndicVoices provides 1 639 transcribed hours across 22 languages; Kathbath and Common Voice add more for Hindi), a Conformer encoder of 100 M or more parameters, SentencePiece tokens, SpecAugment and speed perturbation, and roughly a week on a node of eight data-centre GPUs. Starting from a self-supervised checkpoint such as XLS-R or IndicWav2Vec cuts the labelled data requirement by an order of magnitude. That is why every production backend in vaak is a downloaded checkpoint, and why the next page fine-tunes rather than trains.

Next: Fine-tuning Whisper with LoRA.