3. Features: the log-mel spectrogram

A raw waveform is 16 000 numbers per second with no obvious structure. Speech is better described by which frequencies are present at each moment. The log-mel spectrogram is that description, and it is the input of every model in vaak except wav2vec 2.0 (which learns its own front end from the waveform) and Vosk (which uses the closely related MFCC).

Step 1: short-time Fourier transform (STFT)

Take a short window of audio, typically 25 ms (400 samples at 16 kHz). Multiply it by a Hann window so the edges taper to zero. Take the FFT. The squared magnitude of each FFT bin is the energy at that frequency in that window. Slide the window forward by a hop of 10 ms (160 samples) and repeat.

The result is a matrix of shape (frames, n_fft/2 + 1). With n_fft = 400 that is 201 linear-frequency bins from 0 to 8 000 Hz, 100 frames per second.

Step 2: mel filterbank

Humans hear pitch roughly logarithmically: the difference between 100 and 200 Hz is as noticeable as between 1 000 and 2 000 Hz. The mel scale encodes this. A mel filterbank is a set of triangular filters, equally spaced in mel, that map the 201 linear bins down to 80 (or 128) mel bins. Low frequencies get narrow filters; high frequencies get wide ones. This compresses the input and matches where speech information is dense.

The mel formula used by most toolkits:

mel(f) = 2595 * log10(1 + f / 700)

Step 3: log compression

Energy varies over many orders of magnitude. Taking the log makes the values roughly Gaussian and makes loudness additive rather than multiplicative, which neural networks prefer. Toolkits differ in the base (natural log vs log10), the floor (a small epsilon added before the log), and per-feature normalisation. These details must match the training recipe exactly, which is why vaak lets each backend compute its own features instead of sharing one extractor.

Full example in numpy

import numpy as np

def hz_to_mel(f):  return 2595.0 * np.log10(1.0 + f / 700.0)
def mel_to_hz(m):  return 700.0 * (10.0 ** (m / 2595.0) - 1.0)

def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0):
    n_bins = n_fft // 2 + 1
    fft_freqs = np.linspace(0, sr / 2, n_bins)
    mel_pts = np.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2)
    hz_pts = mel_to_hz(mel_pts)
    fb = np.zeros((n_mels, n_bins), dtype=np.float32)
    for m in range(n_mels):
        lo, mid, hi = hz_pts[m], hz_pts[m + 1], hz_pts[m + 2]
        up = (fft_freqs - lo) / (mid - lo)
        down = (hi - fft_freqs) / (hi - mid)
        fb[m] = np.maximum(0.0, np.minimum(up, down))
    return fb

def log_mel(wav, sr=16000, n_fft=400, hop=160, n_mels=80):
    window = np.hanning(n_fft + 1)[:-1].astype(np.float32)   # periodic Hann
    pad = n_fft // 2
    wav = np.pad(wav, (pad, pad), mode="reflect")
    n_frames = 1 + (len(wav) - n_fft) // hop
    idx = np.arange(n_fft)[None, :] + hop * np.arange(n_frames)[:, None]
    frames = wav[idx] * window                                 # (frames, n_fft)
    spec = np.abs(np.fft.rfft(frames, axis=1)) ** 2            # power, (frames, 201)
    mel = spec @ mel_filterbank(sr, n_fft, n_mels).T           # (frames, n_mels)
    return np.log(mel + 1e-10)

sr = 16000
t = np.arange(sr * 2) / sr
wav = (0.5 * np.sin(2 * np.pi * 220 * t) + 0.2 * np.sin(2 * np.pi * 1760 * t)).astype(np.float32)
feats = log_mel(wav)
print(feats.shape)                 # (201, 80): 2 seconds -> about 200 frames of 10 ms
print(feats.argmax(axis=1)[:5])    # which mel bin is loudest per frame

Run this on a real clip from vaak.load_audio and plot feats.T with imshow and you will see the horizontal bands of vowels and the broadband bursts of consonants.

What Whisper expects

Whisper’s recipe (from the released audio.py):

• n_fft = 400, hop = 160, Hann window, 16 kHz.
• 80 mel bins for tiny through large-v2; 128 mel bins for large-v3 and large-v3-turbo.
• Power spectrogram, mel filterbank from librosa (Slaney-style), then log10(max(x, 1e-10)).
• Dynamic range clamp: max(log_spec, log_spec.max() - 8.0), then (log_spec + 4.0) / 4.0.
• Input padded or trimmed to exactly 30 s = 3 000 frames.

The clamp and scaling put features in roughly [-1, 1]. If you compute features yourself and skip these two lines, Whisper outputs garbage.

What Conformer models expect

NeMo-based models (IndicConformer, Parakeet) use FilterbankFeatures:

• 25 ms window, 10 ms hop, 80 mel bins, natural log with a small guard value.
• Optional pre-emphasis filter (y[t] = x[t] - 0.97 x[t-1]) that boosts high frequencies.
• Per-feature normalisation: subtract the mean and divide by the standard deviation of each mel bin over the utterance.
• No fixed length; any number of frames.

Per-utterance normalisation is why Conformer models are less sensitive to recording level than Whisper, and also why streaming them needs care (page 11): the statistics of a 1 s chunk are not those of the full utterance.

wav2vec 2.0 does not use mel features

wav2vec 2.0 feeds the raw waveform, normalised to zero mean and unit variance, into a stack of seven 1-D convolutions that produce one vector every 20 ms. The convolutions are learned, and after pretraining they resemble a filterbank. The practical difference is minor; the conceptual one is that the model chooses its own frequency resolution.

Data augmentation on features

SpecAugment masks random blocks of time frames and mel bins during training. It is cheap and consistently improves robustness. You will see it in the Conformer, wav2vec 2.0 and Whisper fine-tuning recipes. Page 13 turns it on for LoRA training.

Next: Acoustic encoders.