13. Fine-tuning Whisper with LoRA

Whisper knows Hindi, but not your Hindi: not your accents, your domain words, your preferred script for English loanwords or your numeral convention. Fine-tuning adapts it. Low-Rank Adaptation (LoRA) makes that affordable on one consumer GPU. The script is train/finetune_whisper_lora.py; the resulting checkpoint loads through the hf-whisper:<path-or-hf-id> backend.

What LoRA changes

Full fine-tuning updates all 244 M (small) or 1.55 B (large) weights, needs optimiser state for each, and risks erasing what the model knew. LoRA freezes the original weights and adds, to chosen linear layers, a low-rank update:

W' = W + (alpha / r) * B A        with  A: (r x d_in),  B: (d_out x r),  r << d

A and B are the only trainable parameters. With r = 32 on a 1280 x 1280 projection, that is 2 x 32 x 1280 = 82 k parameters instead of 1.6 M. Across the model, LoRA adds about 1 to 3 % of the parameter count. Training memory drops to roughly the model’s inference footprint plus activations, and the adapter file is a few tens of megabytes. At inference the product BA can be merged into W, so there is no speed cost.

Which modules

The attention projections are the standard target: q_proj, k_proj, v_proj, out_proj in both encoder and decoder. Adding the feed-forward layers (fc1, fc2) gives a little more capacity for new vocabulary at the cost of more trainable parameters. For Hindi, decoder modules matter most because most of the adaptation is about which tokens to write; the encoder already hears Hindi well. The script defaults to attention projections in both, r = 32, alpha = 64, dropout 0.05.

from transformers import WhisperForConditionalGeneration
from peft import LoraConfig, get_peft_model

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
config = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "out_proj"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()   # about 1 to 2 % of total

Whisper’s language and task tokens must be fixed during training so the model does not learn to drift:

model.generation_config.language = "hi"
model.generation_config.task = "transcribe"
model.generation_config.forced_decoder_ids = None

Learning rate and schedule

LoRA tolerates a higher learning rate than full fine-tuning because the update is constrained. Working ranges for Whisper:

• 1e-4 to 3e-4 for r = 16 to 32, AdamW, weight decay 0.01.
• Linear warm-up over the first 5 to 10 % of steps, then linear or cosine decay.
• Effective batch of 16 to 32 utterances (use gradient accumulation on small GPUs).
• 2 to 5 epochs over a few thousand utterances. More epochs rarely help and often hurt.
• fp16 or bf16 autocast. Gradient checkpointing if memory is tight.

A learning rate of 1e-5, which is right for full fine-tuning, will barely move a LoRA adapter. A rate of 1e-3 will diverge within a few hundred steps.

Data preparation for Hindi

The transcripts decide what you teach the model. Before training, choose and enforce:

1. Script convention. Devanagari for everything, or Latin for English loanwords? Pick one, convert the references, and keep the same rule at evaluation.
2. Numerals. Words, Latin digits or Devanagari digits. Whisper’s prior is Latin digits; fighting it costs data.
3. Punctuation. Keep it if your references have it consistently (Whisper benefits), remove it otherwise.
4. Unicode NFC normalisation. Always. Mixed forms of nukta characters make the model learn two spellings for one sound.
5. Length. Clips must be at most 30 s. Drop or split longer ones. Drop clips whose transcript tokenises to more than 448 tokens (the decoder’s limit).
6. Audio. 16 kHz mono via vaak.load_audio; compute Whisper’s log-mel with WhisperFeatureExtractor, which applies the clamp and scaling from page 3.
7. Augmentation. Mild SpecAugment (apply_spec_augment=True in the Whisper config) and speed perturbation at 0.9 / 1.1 help on small sets.

Sources: fleurs-hi, cv-hi, kathbath-hi from bench/datasets.py, each already yielding 16 kHz arrays and text. A few thousand utterances (5 to 10 hours) is enough to see a clear change; a few hundred is enough to see a change but not a reliable one.

from transformers import WhisperProcessor
proc = WhisperProcessor.from_pretrained("openai/whisper-small", language="hindi", task="transcribe")

def prepare(ex):
    feats = proc.feature_extractor(ex["audio"], sampling_rate=16000).input_features[0]
    labels = proc.tokenizer(normalize_text(ex["text"], "hi")).input_ids
    return {"input_features": feats, "labels": labels}

Evaluate before you train

Run the base model on your held-out set and record WER and CER with vaak.metrics. Then look at twenty worst-scoring examples by hand and categorise the errors: script choice, numerals, hallucination, genuinely misheard words. Fine-tuning fixes the first two cheaply, helps the fourth with enough data, and does not fix hallucination unless your data contains silence and noise with empty transcripts.

vaak bench --models faster-whisper:small --datasets fleurs-hi --limit 200
python train/finetune_whisper_lora.py --base-model openai/whisper-small --data fleurs --max-minutes 30 --merge --out train/out
vaak bench --models hf-whisper:train/out/whisper-small-hi-merged --datasets fleurs-hi --limit 200

Evaluate on a held-out split that the adapter never saw, and on at least one dataset from a different source than the training data. An adapter trained on FLEURS that improves FLEURS by 10 points and worsens Kathbath by 5 has learned the FLEURS reading style, not Hindi.

Signs of overfitting and other failure

• Validation loss rises while training loss falls. Stop at the minimum validation loss, which the script checkpoints as best.
• WER improves while CER worsens, or the reverse: the model has changed convention (script, numerals) rather than accuracy. Check outputs.
• Outputs become shorter than references, especially on long clips: the model learned to emit end-of-text early, a common effect of training on short utterances only. Mix in longer clips.
• Repetition loops appear that the base model did not have: learning rate too high or too many epochs.
• Good results on the training speakers, no change on new speakers: too little speaker diversity. Common Voice has many speakers per hour; FLEURS has few.
• The adapter helps Hindi and destroys English: expected for a Hindi-only adapter. Keep the base model for English, or train on a mixed set.

Merging and shipping

merged = model.merge_and_unload()
merged.save_pretrained("runs/whisper-small-hi-merged")
proc.save_pretrained("runs/whisper-small-hi-merged")

The merged model loads with hf-whisper:runs/whisper-small-hi-merged and can be converted with ct2-transformers-converter for the faster-whisper: backend. Document the training data, the normalisation rules and the before/after numbers in the model card, labelled “reproduced by vaak” on the benchmark page.

Next: Further reading.