11. Streaming

Offline transcription sees the whole file. Streaming has to produce text while the person is still talking. Every design choice is a trade between latency (how soon text appears), accuracy (how much right context the model had), and compute (how often you re-run the model).

Chunking

Audio arrives in small packets. The server accumulates them into chunks and runs the model on each chunk. Two things define a chunk:

• Chunk size. How much new audio is processed per step. 0.5 to 2 s is typical. Smaller gives lower latency but more model calls per second of audio.
• Lookahead (right context). How much audio after the chunk the model is allowed to see before committing. A bidirectional encoder needs some future to recognise the end of a word. More lookahead means better accuracy and more latency: with 1 s of lookahead, the earliest any word can appear is 1 s after it was spoken.

Streaming-trained encoders (chunked attention, causal convolutions) accept a fixed small lookahead. Offline encoders like Whisper’s were trained on full 30 s windows and degrade badly on short chunks. The common workaround for Whisper is to re-run it on a growing buffer and keep only the stable prefix (the “local agreement” method used by whisper-streaming). It works but costs several times more compute than the audio duration.

Voice activity detection

A VAD classifies short frames (10 to 30 ms) as speech or non-speech. Silero VAD is the usual choice: a small neural network, runs on CPU in well under real time, works on Hindi and English alike because it models speech-ness rather than words. Uses:

• Drop silence before feeding Whisper, which removes the main cause of hallucination (page 6).
• Find natural boundaries to end a segment, so a final hypothesis is emitted at a pause instead of mid-word.
• Keep the buffer short: when the VAD says the speaker has paused, flush and start fresh.

VAD is not free of errors. Breathy consonants at the start of a word and quiet endings get trimmed. Keep 100 to 300 ms of padding on both sides of each detected region.

Partial and final hypotheses

A streaming system emits two kinds of result:

• Partial. The current best guess for the audio so far. It may change: “मुझे कल” can become “मुझे कल बाज़ार” or be revised to “मुझे कल्पना”. Show it in a lighter colour, do not store it.
• Final. Text the server will not revise, emitted when a segment ends (a VAD pause, a maximum duration, or the client signalling end of stream). Store this one.

Revisions happen because the model with more right context changes its mind. A well-designed streamer marks text final only after it has been stable across a few chunks or after a pause.

How each model family streams

• CTC with a chunked encoder. Natural. Each chunk’s frames are decoded with greedy collapse; the only state is the last emitted character for the collapse rule. Partials are just the running concatenation. IndicConformer models can be run in this mode with NeMo’s cache-aware streaming, if the checkpoint was trained for it; otherwise use buffered re-decoding.
• Transducer. Also natural; the prediction network state carries across chunks (page 7). Parakeet TDT streams with a cache-aware FastConformer.
• Whisper. Not natural. Buffer, re-decode, keep stable prefix.

vaak’s /v1/stream

The server exposes one WebSocket endpoint:

WS  /v1/stream?language=hi
client -> server : binary frames of PCM16LE, 16 kHz, mono (any frame size; 20 to 100 ms is typical)
server -> client : {"type": "partial", "text": "...", "start": 0.0, "end": 1.6}
                   {"type": "final",   "text": "...", "start": 0.0, "end": 2.4}
client -> server : text frame "EOS"    (flush remaining audio, server sends final results then closes)

Internally the server keeps a per-connection buffer, runs VAD on incoming audio, and calls the loaded backend’s transcribe on the buffered segment. For CTC and transducer backends it decodes the growing buffer and emits partials; for Whisper it re-decodes the buffer and emits a partial with the current text, then a final when the VAD detects a pause or the buffer reaches the window limit. start and end are seconds since the start of the stream.

A minimal client:

# pip install websockets soundfile numpy
import asyncio, json
import numpy as np
import soundfile as sf
import websockets

async def stream_file(path, url="ws://localhost:8000/v1/stream?language=hi", chunk_ms=100):
    wav, sr = sf.read(path, dtype="float32")
    assert sr == 16000, "resample first, see page 2"
    if wav.ndim > 1:
        wav = wav.mean(axis=1)
    pcm = (np.clip(wav, -1, 1) * 32767).astype("<i2").tobytes()
    step = 16000 * 2 * chunk_ms // 1000            # bytes per chunk

    async with websockets.connect(url) as ws:
        async def sender():
            for i in range(0, len(pcm), step):
                await ws.send(pcm[i:i + step])
                await asyncio.sleep(chunk_ms / 1000)  # simulate real time
            await ws.send("EOS")

        async def receiver():
            async for msg in ws:
                m = json.loads(msg)
                tag = "FINAL  " if m["type"] == "final" else "partial"
                print(f"{tag} [{m['start']:6.2f}-{m['end']:6.2f}] {m['text']}")

        await asyncio.gather(sender(), receiver())

asyncio.run(stream_file("clip.wav"))

From a browser, capture with AudioWorklet at 16 kHz, convert float32 to int16, and send ArrayBuffer frames over the same socket. The live demo on this site does exactly that.

Measuring streaming latency

Three numbers matter, and they differ:

• First-partial latency. Time from the first speech sample to the first partial. Governed by chunk size plus model time.
• Word latency. Time from a word being spoken to it appearing in a partial. Governed by chunk size plus lookahead.
• Final latency. Time from the end of speech to the final result. Governed by the VAD’s silence threshold (often 300 to 800 ms) plus the last decode.

A streaming system with a 1 s chunk, 0.5 s lookahead and 0.5 s VAD silence will show words roughly 1.5 to 2 s after they are spoken and finalise 1 s after a pause. If that is too slow for your product, shrink the chunk first, then the VAD threshold, and only then the lookahead.

Offline accuracy is the ceiling

A streaming model with limited right context is always a little worse than the same model offline. The gap is typically a few percent relative for transducers and larger for improvised Whisper streaming. If the use case can wait for the end of an utterance, send the whole utterance to /v1/transcribe and get the offline number.

Next: Build a tiny STT from scratch.