Privacy
Audio stays on your machine. No upload, no retention policy to read, no third-party terms.
vaak
Self-hosted speech-to-text for English, Hindi and Indic languages. Open weights, your hardware, no cloud APIs.
vaak (वाक्, “speech”) is a Python toolkit that wraps open speech-to-text models behind one API, one CLI and one HTTP server. You pick a backend with a spec string such as faster-whisper:small or indic-conformer:hi, and every backend runs on your own CPU or GPU.
How it works
Section titled “How it works”audio→mel spectrogram→encoder→decoder / CTC→text
Audio. load_audio() decodes any file ffmpeg can read, mixes it to mono and resamples to 16 kHz float32. Every backend sees the same array.
Mel spectrogram. The waveform is cut into overlapping 25 ms windows, each is transformed to a frequency spectrum, and the spectrum is squashed onto a mel scale that follows human hearing. The result is a 2-D image: time along one axis, 80 or 128 mel bands along the other.
Encoder. A neural network (Transformer in Whisper, Conformer in IndicConformer and Parakeet, convolution plus Transformer in wav2vec2) turns each spectrogram frame into a vector that summarises what was said around that moment.
Decoder or CTC. Whisper uses an autoregressive decoder that writes tokens one at a time while attending to the encoder output. CTC models (IndicConformer, wav2vec2, Vosk) emit one label per frame and collapse repeats and blanks; Parakeet uses a transducer, a close relative. Decoders tend to be more accurate; CTC and transducers are faster and stream naturally.
Text. Tokens become a string. vaak returns it with segment timestamps, the detected language and the real-time factor.
The tutorial walks through each stage with code.
Backends
Section titled “Backends”| Spec | Model family | Languages | Typical hardware |
|---|---|---|---|
faster-whisper:<size> |
OpenAI Whisper via CTranslate2 | 99 languages incl. en, hi | CPU for tiny/small; GPU for large-v3, large-v3-turbo |
indic-conformer:<lang> |
AI4Bharat IndicConformer via NeMo | 22 Indian languages, one model per lang | GPU recommended, CPU works |
parakeet:<hf-id> |
NVIDIA Parakeet TDT via NeMo | English | GPU |
wav2vec2:<hf-id> |
Hugging Face transformers CTC | Depends on checkpoint | CPU or GPU |
vosk:<model-name> |
Vosk (Kaldi) | en, hi and others, small models | CPU only, runs on a laptop |
hf-whisper:<hf-id> |
Hugging Face transformers Whisper | 99 languages; used for LoRA checkpoints | CPU or GPU |
Pick with --model <spec> on the CLI, get_backend(spec) in Python, or VAAK_MODEL for the server. See model specs.
Quickstart
Section titled “Quickstart”pip install "vaak[whisper]"vaak transcribe clip.mp3 --language hivaak serveThe last command starts an HTTP API on port 8000 with POST /v1/transcribe, an OpenAI-compatible POST /v1/audio/transcriptions, and a WebSocket stream. Details in the quickstart.
Live demoRun whisper-tiny in your browser, or point the page at your own vaak server.
HTTP APIEvery route, the WebSocket protocol, and curl, Python and Node examples.
BenchmarksWER, CER and real-time factor per backend, every row labelled with its provenance.
Fine-tuningLoRA on whisper-small for Hindi, on a budget-capped RunPod GPU.
Why self-host
Section titled “Why self-host”Cost
Cloud STT bills per minute of audio. A GPU you already have, or a rented one for an hour, has no per-minute meter.
Offline
Once weights are downloaded, vaak runs with the network cable unplugged. Vosk and small Whisper models run on a CPU.
Indic coverage
Hindi and other Indian languages through AI4Bharat IndicConformer and multilingual Whisper, with Devanagari-aware WER normalisation.