đź“ť Originally published (in Japanese) at forge.workstyle.tech.
While building a "Voice Design" app, I added a slider to adjust speech rate (speaking speed). The goal was simple: speed up or slow down the tempo without changing the pitch. This is a very common requirement.
librosa has a function called librosa.effects.time_stretch designed exactly for this. It’s a one-liner. That’s what I used at first. However, as soon as I moved the slider even slightly, a faint metallic ringing would appear in the output. The voice sounded slightly "robotic" and "echoey," becoming muffled. The original voice was natural, but the moment the tempo changed, the quality dropped.
This article is a record of how I discovered that the cause was phase blurring in the phase vocoder and how I resolved it by implementing WSOLA (Waveform Similarity Overlap-Add) from scratch using numpy.
Premise: Changing tempo while preserving pitch
If you simply drop or duplicate audio samples, the pitch will shift along with the playback speed (the "chipmunk effect" you get when fast-forwarding). Time stretching is the process of changing only the tempo while avoiding this pitch shift.
There are two main approaches to this:
- Phase Vocoder: Converts the signal into the frequency domain using STFT, then stretches/compresses it by adjusting the phase advancement of each frequency bin. It operates in the frequency domain.
- WSOLA: Cuts the waveform (time domain) into short frames and finds the best positions to overlap and add them so they connect smoothly. It operates in the time domain.
librosa's time_stretch uses the former: the phase vocoder.
The Symptom: Metallic ringing (Robotic voice)
A phase vocoder treats each frequency bin independently and updates the phase to the "intended" advancement amount. While mathematically sound, in real speech, the phase relationships between harmonic components (overtones) gradually fall apart. This is known as a loss of phase coherence.
To the human ear, this manifests as:
- A metallic or electronic sound (often called "phasiness").
- A faint reverberation or echo effect.
- Blurry vowel cores, making the voice sound "synthetic."
While this is often unnoticeable in music or percussive material, human speech relies heavily on formants and harmonic structures. Because of this, phase blurring is heard very clearly as a "robotic voice." Furthermore, the effect worsens as the stretch ratio increases. The conclusion was that it was a poor match for a speech-rate slider.
The Solution: WSOLA in the time domain
WSOLA does not enter the frequency domain; it simply cuts and pastes the waveform. Instead of "recalculating" the phase, it uses cross-correlation to find the position where adjacent frames connect most naturally. Therefore, phase blurring does not occur by design.
The logic works like this:
- On the output side, frames are arranged at fixed intervals (
syn_hop). - From the input side, we take frames at an interval corresponding to the stretch rate (
ana_hop = syn_hop × rate) at the "ideal" positions. - However, instead of the exact ideal position, we search within a range of ±
tolto find the frame that most closely matches the continuation of the previously placed frame (creating a natural waveform continuity). - The found frame is then combined using a Hann window (overlap-add).
The heart of the implementation (_apply_speed) is this "coarse search for the similar position."
ana_hop = int(round(syn_hop * float(rate)))
win = np.hanning(frame).astype(np.float32)
...
# The "natural continuation" of the previous frame = nat
nat = xp[off + prev_ana + syn_hop: off + prev_ana + syn_hop + frame]
best_d, best = 0, -1e18
for d in range(-tol, tol + 1, 8): # Coarse search within ±tol to align phase
cand = xp[off + ideal + d: off + ideal + frame + d]
sc = float(np.dot(cand, nat)) / (float(np.linalg.norm(cand)) + 1e-6