2. Audio and sampling

A sound is a pressure wave. A microphone turns it into a voltage that changes over time. An analogue-to-digital converter measures that voltage at a fixed rate and stores each measurement as a number. The result is a one-dimensional array: the waveform.

Two choices define the array: how often you measure (sample rate) and how many bits you store per measurement (bit depth).

Sample rate

The sample rate is samples per second, in hertz. CD audio is 44 100 Hz. Phones and video calls use 8 000 or 16 000 Hz. The Nyquist theorem says a sample rate of fs can represent frequencies up to fs / 2. Speech intelligibility lives mostly below 4 kHz, and almost all useful content is below 8 kHz. So 16 kHz, with a Nyquist limit of 8 kHz, keeps everything that matters for recognition and halves the data compared to 32 kHz.

Nearly every open STT model is trained on 16 kHz audio: Whisper, Conformer, wav2vec 2.0, Parakeet, Vosk. Feeding them 8 kHz telephone audio upsampled to 16 kHz works, but the upper band is empty and accuracy drops. Feeding them 44.1 kHz without resampling is simply wrong; the model sees speech slowed down by 2.75x.

vaak.load_audio always returns 16 kHz mono float32, so you do not need to think about this at the API boundary. You do need to think about it when you build your own pipeline or a streaming client.

Mono

Stereo adds nothing for recognition and doubles the data. Convert to mono by averaging the channels. Averaging is safe for ordinary recordings. If one channel is a different speaker (a two-channel call recording), transcribe the channels separately instead.

Quantisation and PCM16

Each sample is stored as an integer or a float. Common formats:

Format	Range	Bytes per sample	Used by
PCM16 (int16)	-32768 to 32767	2	WAV files, WebSocket streams, telephony
PCM24 / PCM32	wider	3 or 4	studio recordings
float32	-1.0 to 1.0	4	model inputs in numpy/torch

16 bits give a dynamic range of about 96 dB, far more than any microphone in a room needs. PCM16 is the universal interchange format because it is compact, exact, and every tool reads it. Models consume float32 in [-1, 1]; the conversion is a division by 32768. vaak’s /v1/stream WebSocket accepts raw PCM16LE (little-endian) 16 kHz mono bytes for exactly this reason: 32 000 bytes per second, no container, no codec.

Lossy codecs (MP3, Opus, AAC) are fine as storage. Decode them to PCM before feature extraction. Codec artefacts do affect accuracy; one of the 2026 Indic ASR studies measured the effect of telephone codecs and bit depth on Hindi WER.

Resampling correctly

Resampling is not “drop every nth sample”. Dropping samples lets frequencies above the new Nyquist limit fold back into the audible band (aliasing). Correct resampling low-pass filters first. scipy.signal.resample_poly does this with a polyphase filter and is fast. Use the ratio of integers up / down, reduced by their greatest common divisor.

import math
import numpy as np
import soundfile as sf
from scipy.signal import resample_poly

def load_16k_mono(path: str) -> np.ndarray:
    audio, sr = sf.read(path, dtype="float32", always_2d=True)  # (samples, channels)
    audio = audio.mean(axis=1)                                   # mono
    if sr != 16000:
        g = math.gcd(sr, 16000)
        audio = resample_poly(audio, 16000 // g, sr // g).astype(np.float32)
    return audio

wav = load_16k_mono("clip.wav")
print(wav.shape, wav.dtype, wav.min(), wav.max())
print(f"{len(wav) / 16000:.2f} seconds")

# float32 [-1, 1]  ->  PCM16 bytes for the WebSocket API
pcm16 = (np.clip(wav, -1.0, 1.0) * 32767).astype("<i2").tobytes()
print(len(pcm16), "bytes")

For 44 100 to 16 000 Hz the gcd is 300, so up=160, down=441. For 48 000 to 16 000 Hz it is simply up=1, down=3.

soundfile reads WAV, FLAC and OGG. For MP3 and video containers, vaak.load_audio falls back to ffmpeg; if you build your own loader, shell out to ffmpeg -i in.mp4 -ac 1 -ar 16000 -f s16le - and reinterpret the bytes with np.frombuffer(..., dtype="<i2").

Loudness and clipping

Models are fairly tolerant of level, but two things hurt. Clipping (samples pinned at the maximum) destroys information and cannot be undone. Very quiet audio, below about -40 dBFS, pushes the log-mel features toward the floor. A simple peak normalisation to about -3 dBFS before inference is cheap insurance:

peak = np.abs(wav).max()
if peak > 0:
    wav = wav * (0.7 / peak)

Do not apply aggressive compression or noise suppression intended for human listening. These often remove the high-frequency consonant cues that models rely on. If the recording is noisy, let the model handle it, or use a VAD to cut silence (page 11).

Duration and framing

At 16 kHz, one second is 16 000 samples. The feature extractor in the next page slices this into 25 ms windows every 10 ms, giving 100 frames per second. Whisper pads or trims every input to 30 s (480 000 samples, 3 000 frames). Conformer and transducer models accept any length. Keep this in mind when you see RTF numbers: a Whisper model always pays for 30 s of encoder compute even on a 3 s clip.

Next: Features and the mel spectrogram.