Fine-tuning

vaak does not train models from scratch. The fine-tuning script adapts an existing Whisper checkpoint with LoRA, which trains a small number of extra weights and leaves the original model untouched. This fits in the RunPod budget and the adapter is a few megabytes.

train/tiny_ctc_from_scratch.py exists for the tutorial only. It trains a toy CTC model to show the mechanics and is not meant to produce a usable recogniser.

What you need

• A GPU with at least 8 GB of VRAM for whisper-small in fp16 with LoRA. A 4090 is comfortable.
• pip install "vaak[whisper]" peft accelerate datasets
• A Hindi dataset. The script defaults to google/fleurs hi_in, which streams without a token. Common Voice Hindi needs HF_TOKEN.

Run

python train/finetune_whisper_lora.py   --base-model openai/whisper-small   --data fleurs   --max-train-samples 2000   --eval-samples 100   --max-steps 1000   --max-minutes 30   --lora-r 16   --merge   --out train/out

Flag	Default	Meaning
--base-model	openai/whisper-small	Hugging Face id of the starting checkpoint
--data	fleurs	fleurs (google/fleurs hi_in, no token) or cv (Common Voice Hindi, needs HF_TOKEN)
--max-train-samples	2000	Cap on training utterances. Use it to fit the time budget
--eval-samples	100	Test utterances for the WER eval before and after training
--max-steps	1000	Optimiser steps
--max-minutes	30	Hard wall-clock cap. Training stops cleanly and still saves
--batch-size	16	Per-device batch; lower it on cards under 12 GB
--lr	1e-3	LoRA tolerates a higher rate than full fine-tuning
--lora-r	16	LoRA rank. 8 to 32 is the usual range (--lora-alpha 32)
--merge	off	Also write a merged full model you can load without peft
--skip-eval-before	off	Skip the baseline WER pass to save a few minutes
--out	train/out	Output directory

The language is fixed to Hindi inside the script (decoder language token hi); copy the file and change two lines for another language.

The script:

1. Loads the base model in fp16 and wraps attention projections (q_proj, v_proj) with LoRA adapters from peft.
2. Builds mel features with the base model’s feature extractor and tokenises references with the forced <|hi|><|transcribe|> prefix.
3. Trains with the Hugging Face Seq2SeqTrainer.
4. After each epoch, decodes the eval split and reports WER and CER using vaak.metrics with the same normaliser as the benchmarks.
5. Saves the adapter to <out>/whisper-small-hi-lora, plus <out>/whisper-small-hi-merged when --merge is set.

Use the result

The merged checkpoint is a normal Whisper model, so the hf-whisper backend loads it directly:

vaak transcribe clip.mp3 --model hf-whisper:train/out/whisper-small-hi-merged --language hi

or after publishing it:

vaak transcribe clip.mp3 --model hf-whisper:<your-user>/whisper-small-hi-lora

To benchmark it against the base model:

vaak bench --models hf-whisper:openai/whisper-small,hf-whisper:<your-user>/whisper-small-hi-lora --datasets fleurs-hi --limit 200

Both rows will carry the “reproduced by vaak” badge on the benchmarks page.

Expectations

• A couple of thousand FLEURS utterances and two epochs is a small amount of data. It usually nudges WER down on FLEURS itself; whether it helps on your own audio depends on how similar that audio is.
• Do not evaluate on the data you trained on. The script uses the test split for eval and the train split for training; keep it that way.
• Over-training on read speech can hurt conversational audio. Compare on a held-out sample of your real data before switching.
• There are no fine-tuning results on the website until a run has been done and merged through bench/report.py.

Converting for faster-whisper

faster-whisper needs a CTranslate2 model:

pip install ctranslate2
ct2-transformers-converter --model train/out/whisper-small-hi-merged \
  --output_dir train/out/whisper-small-hi-ct2 --quantization float16
vaak transcribe clip.mp3 --model faster-whisper:train/out/whisper-small-hi-ct2