How it works
Cover Me
Six held notes are enough to sing a whole song, if you can move each one to any pitch and any length without changing what the voice sounds like. Below are the six ideas that make that work, from the signal processing to the recording screen, each with the lines of code that carry it.
- Signal processing
- Interaction design
- Web Audio
- Privacy
From six notes to a cover
- Recorda reference tone plays, you sing it back while a tuner watches
- AnalyseYIN finds the pitch, then a mark goes on every glottal pulse
- Find the tunePianoify extracts the melody; it moves by octaves into your range
- Re-singeach note takes the closest take, retuned and stretched by PSOLA
- Mixadd piano chords or the original, play, download a WAV
Find the pitch of a wobbly voice
The YIN pitch tracker
A sung note repeats its waveform once per vibration of the vocal folds, so its pitch is one over the length of that repeat. YIN finds the repeat by sliding each 1024-sample frame against a delayed copy of itself and measuring how different the two are. The same function runs offline on every take and live, on the microphone, in the tuner.
d′(τ) = d(τ) · τ / Σj=1..τ d(j), d(τ) = Σi (xi − xi+τ)²
d[0] = 1;
let run = 0;
for (let tau = 1; tau <= tauMax; tau++) {
let acc = 0;
for (let i = 0; i < W; i++) { const v = x[s + i] - x[s + i + tau]; acc += v * v; }
run += acc;
d[tau] = acc * tau / (run || 1);
}
let tau = -1;
for (let t = tauMin; t <= tauMax; t++) {
if (d[t] < o.thresh) { while (t + 1 <= tauMax && d[t + 1] < d[t]) t++; tau = t; break; }
}
if (tau < 0) { // no dip under threshold: take the global minimum
let best = tauMin;
for (let t = tauMin; t <= tauMax; t++) if (d[t] < d[best]) best = t;
if (d[best] < 0.35) tau = best;
}
if (tau > 0) {
const a = d[tau - 1], b = d[tau], c = tau + 1 <= tauMax ? d[tau + 1] : b;
const den = a + c - 2 * b;
const shift = den ? 0.5 * (a - c) / den : 0;
f0[k] = SR / (tau + Math.max(-1, Math.min(1, shift)));
conf[k] = 1 - b;
}- The difference function: for each candidate period
tau, the squared distance between the frame and itself shifted bytausamples. A true period gives a deep dip. - YIN's key step: divide by the running mean of all earlier values. A raw difference is always zero at
tau= 0 and tends to be small at short lags; after normalising, “small” means small relative to its neighbours. - Take the first lag that dips under 0.15, then slide down to the bottom of that dip. Picking the first good dip rather than the deepest one avoids jumping an octave down to twice the period.
- If nothing crosses the threshold, fall back to the global minimum, but only if it is reasonably clear (under 0.35). Otherwise the frame is marked unvoiced.
- Fit a parabola through the dip and its two neighbours to get a fractional period. At 22,050 Hz, whole-sample periods are tens of cents apart for higher notes (a 440 Hz period is only 50 samples); this is what lets a take's pitch be found to a fraction of a cent.
On the synthetic test voice, with vibrato, jitter and breath noise, every take's pitch was found within 0.2 cents. A median filter afterwards (cleanTrack) replaces any lone frame that still lands an octave off.
Put a mark on every glottal pulse
Pitch marks, and the steadiest stretch
Knowing the pitch per frame is not enough to retune a voice. The next step needs to know where each individual period starts in the waveform. These “pitch marks” are placed by walking through the voiced part of the take one period at a time, using the YIN pitch to predict where the next pulse should be and the waveform itself to say where it actually is.
// pitch marks: walk period by period through voiced frames, snapping to the waveform peak
const f0At = s => { const k = Math.max(0, Math.min(n - 1, Math.round((s - half) / hop))); return f0[k]; };
const marks = [];
let s = tOf(va);
const end = Math.min(x.length - 1, tOf(vb));
// start on the largest peak within one period of the voiced start
let p0 = SR / (f0At(s) || 200);
let best = s; for (let i = s; i < Math.min(end, s + p0); i++) if (x[i] > x[best]) best = i;
s = best;
while (s < end) {
marks.push(s);
const f = f0At(s) || midiToHz(midi);
const P = SR / f;
let c = Math.round(s + P), bi = c;
const w = Math.round(P * 0.15);
for (let i = c - w; i <= c + w; i++) if (i > s && i < x.length && x[i] > x[bi]) bi = i;
s = bi > s ? bi : c;
}- Anchor the first mark on the highest sample within one period of where singing starts, so marks sit on the same point of every cycle.
- Predict the next mark one period ahead, using the local pitch, or the take's median pitch where YIN had nothing.
- Search ±15% of a period around the prediction and move to the highest peak. This lets the marks follow vibrato and drift rather than accumulating error.
- A guard: if the search did not land past the current mark, step by the predicted period, so the walk always moves forward.
The same function also finds the take's steadiest stretch: the longest run of frames within ±0.6 semitone of its median pitch. Long notes are later looped inside it, and the fraction of the take it covers becomes a steadiness score that the take's card reports as “a bit wobbly” when it is under 0.35.
Change the pitch, keep the vowel
TD-PSOLA: pitch-synchronous overlap-add
Playing a recording faster raises its pitch but also slides every resonance of the mouth upward, which is the chipmunk sound. TD-PSOLA avoids that. It cuts a short windowed grain around each pitch mark, each grain holding one pulse as shaped by the singer's vocal tract, then lays the grains back down spaced at the target period. The pulses come faster or slower; the shape of each one, and so the vowel, does not change.
// 3) overlap-add two-period Hann grains at the target period
const baseHz = midiToHz(targetMidi);
let t = lead;
const crossLead = lead > 0 ? Math.round(0.01 * SR) : 0; // tiny blend from consonant into vowel
while (t < N) {
const sec = (t - lead) / SR;
let hz = baseHz;
if (o.vibrato && dur > 0.35 && sec > 0.25) { // gentle delayed vibrato on long notes
const depth = Math.min(1, (sec - 0.25) / 0.3) * 0.22; // up to ±22 cents
hz *= Math.pow(2, depth * Math.sin(2 * Math.PI * 5.4 * sec) / 12);
}
const P = SR / hz;
const k = nearestMark(srcPos(t - lead));
const c = marks[k], Ps = clip.periods[k];
const L = Math.round(Math.min(Ps, P * 1.5)); // grain half-length
const g = Ps / P > 1.8 ? 0.8 : 1;
for (let i = -L; i <= L; i++) {
const si = c + i, oi = Math.round(t) + i;
if (si < 0 || si >= clip.x.length || oi < 0 || oi >= y.length) continue;
const w = 0.5 + 0.5 * Math.cos(Math.PI * i / (L + 1));
let v = clip.x[si] * w * g;
if (oi < lead + crossLead) v *= (oi - lead + crossLead) / (2 * crossLead + 1);
y[oi] += v;
}
t += P;
}- Notes longer than 0.35 s get a delayed vibrato: it fades in after a quarter of a second up to ±22 cents at 5.4 Hz, the way singers tend to settle into a note.
- The output period
Pcomes from the target note. The grain comes from whichever source mark is nearest to the current position in the take (section 04 decides that position). - A grain is up to one source period on each side of its mark, capped at 1.5 target periods so grains don't pile up when shifting far upward. When they still overlap heavily (source period over 1.8× the target), the gain drops to 0.8.
- Overlap-add: a Hann window tapers each grain to zero at its edges, so neighbouring grains blend without clicks.
- The whole pitch change is this one line: the next grain is placed one target period later, not one source period.
The first formant (the main resonance of “ah”) of the test take was 861 Hz. Moved up 5 semitones with PSOLA it measured 807 Hz; moved up 5 semitones by speeding up, 996 Hz. Across Pianoify's demo melody, 52 of 52 re-sung notes landed within 30 cents of the score, with a median error of 0.3 cents.
Make a two-second take fit any note
Time mapping, clip choice and legato
PSOLA asks, for every output grain, “which part of the take should this come from?” That mapping from output time to source time is what lets a 2.6-second recording sing a quaver or a four-bar whole note. A consonant at the start of a word plays at natural speed first; only the vowel is stretched.
// 2) voiced part: map output time → source time
const vStart = marks[0], vEnd = Math.min(clip.voicedEnd, marks[M - 1]);
const [sA, sB] = clip.stable[1] - clip.stable[0] > 0.08 * SR ? clip.stable : [vStart, vEnd];
const outLen = N - lead;
const natural = vEnd - vStart;
const sMid = (sA + sB) / 2;
const loA = sA + (sB - sA) * 0.25, loB = sA + (sB - sA) * 0.75, span = Math.max(1, loB - loA);
function srcPos(t) { // t: samples into the voiced output
if (outLen <= natural) return vStart + t * natural / Math.max(outLen, 1); // short note: compress
if (vStart + t < sMid) return vStart + t; // play into the steady part
const q = (sMid - loA) + (vStart + t - sMid); // then bounce inside it
const ph = q % (2 * span);
return loA + (ph < span ? ph : 2 * span - ph);
}- Use the steadiest stretch from section 02 if it is longer than 80 ms, otherwise the whole voiced part.
- Loop only the middle half of the steady stretch, away from where the note settles in or fades.
- A note shorter than the take squeezes the whole voiced part into it. Grains are still placed at the target period, so the pitch is unaffected.
- A long note plays forward into the steady part, then bounces back and forth inside it. Playing backward and forward (rather than jumping back to the start) means there is never a seam where two distant periods meet.
Every melody note then needs a take. PSOLA sounds best over small shifts (it turns buzzy past about five semitones; see Limits), so the choice weighs pitch distance against steadiness.
} else {
// closest recorded pitch, preferring steady takes
clip = usable.reduce((a, c) => (Math.abs(c.midi - n.midi) - 2 * c.steadiness < Math.abs(a.midi - n.midi) - 2 * a.steadiness ? c : a));
}
assign.push(usable.indexOf(clip));
const next = notes[i + 1];
const legato = next && next.t0 - n.t1 < 0.03;
const dur = Math.max(0.08, (n.t1 - n.t0) + (legato ? 0.03 : 0));
const y = psola(clip, n.midi, dur, { release: legato ? 0.03 : 0.07 });- Score each take as semitones away minus twice its steadiness, and keep the lowest. A very steady take can win over a slightly closer, shaky one.
- If the next note starts within 30 ms, this one is extended by 30 ms so the two overlap. Without that, a legato line would have tiny gaps between notes.
- Legato notes also get a shorter 30 ms release instead of 70 ms, so the overlap is a quick crossfade rather than a fade-out.
A recording screen that coaches you
Guided takes and a live tuner
The engine is only as good as the takes, and most people have never recorded a held note. So the page doesn't hand over a record button and hope. You pick a range (low, middle or high), and it walks you through six target notes spread evenly across it: it plays each note, then records while showing how close you are.
async function recordTake(i) {
const m = targets()[i];
renderTakes(i);
$('meter').hidden = false;
$('mMsg').textContent = `Listen: ${noteName(m)}`; $('mNote').textContent = '–';
playTone(m, 1.0);
await sleep(1150);
$('mMsg').textContent = 'Now sing "ah" and hold it';
const raw = await record(2.6, m);
const clip = V.analyzeClip(raw, { kind: 'note', label: noteName(m) });
takes[i] = { clip, raw };
if (clip.ok) DB.put({ id: 'note-' + i, kind: 'note', slot: i, target: m, data: raw });
renderTakes(-1);
}- Play a one-second reference tone and wait for it to finish, so the singer hears the target and the recording doesn't pick it up.
- Then record 2.6 seconds with the tuner running, aimed at this target note.
- The take is analysed straight away. Only a take where singing was actually found is saved; a failed one shows its reason in the card and a Redo button.
The tuner runs once per animation frame on the microphone's analyser buffer, resampled to 22,050 Hz and passed to the same yin function as in section 01. The microphone is opened with echoCancellation, noiseSuppression and autoGainControl all off, because those voice-call filters change exactly the pitch and loudness the engine needs to measure.
if (rms > 0.01 && n >= 1100) { const t = V.yin(d, { frame: 1024, hop: 1024 }); hz = t.f0[0]; }
if (hz > 0) {
const m = V.hzToMidi(hz);
let ref = target != null ? target : Math.round(m);
if (Math.abs(m - ref) > 1) { // far off: tune against the nearest note, say where to go
$('mMsg').textContent = target != null ? `You're on ${noteName(m)}. Aim for ${noteName(target)}` : $('mMsg').textContent;
ref = Math.round(m);
} else if (target != null) $('mMsg').textContent = 'Now sing "ah" and hold it';
let cents = (m - ref) * 100;
const show = Math.max(-50, Math.min(50, cents));
$('mNote').textContent = noteName(m);
$('mNeedle').style.left = `calc(${50 + show}% - 2px)`;
$('mNeedle').style.background = Math.abs(cents) < 20 ? css('--teal') : css('--ink');- Skip near-silent buffers, then run YIN on a single frame for one pitch reading per frame.
- More than a semitone off is not shown as “very flat”. The needle switches to the nearest note and the message says in words where you are and where to go: “You're on E3. Aim for G3”.
- Cents off are clamped to ±50 so the needle stays on the scale.
- Within 20 cents, the needle turns teal: close enough, keep holding.
Each take's card says what you actually sang (“Sang G3 (−12¢)”), so a bad take is obvious and Redo fixes one note without redoing all six. The meter is an aria-live region, and “Use a sample voice” or “Upload recordings” let anyone without a microphone try the whole page.
Your voice stays on your device
IndexedDB, and no server at all
A recording of someone's voice is personal data, and the easiest way to keep it safe is never to send it anywhere. Everything here runs in the browser: the song's melody is found by Pianoify's analysis.js, in a Web Worker where the browser allows it, and the voice engine is plain JavaScript with no libraries. Takes are saved in the browser's own database so they survive a reload.
const DB = (() => {
let dbp = null;
const open = () => dbp || (dbp = new Promise((res, rej) => {
const r = indexedDB.open('cover-me', 1);
r.onupgradeneeded = () => r.result.createObjectStore('takes', { keyPath: 'id' });
r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error);
}));
const tx = async (mode, fn) => {
try {
const db = await open();
return await new Promise((res, rej) => { const t = db.transaction('takes', mode); const q = fn(t.objectStore('takes')); t.oncomplete = () => res(q && q.result); t.onerror = () => rej(t.error); });
} catch (e) { return null; }
};
return { put: rec => tx('readwrite', s => s.put(rec)), del: id => tx('readwrite', s => s.delete(id)), all: () => tx('readonly', s => s.getAll()) };
})();- One IndexedDB database,
cover-me, with one object store keyed by take id (note-0…note-5, orword-…). It lives in this browser on this device only. - Every operation goes through one wrapper that swallows errors and returns
null. In a private window or with storage blocked, the page still works; takes just aren't remembered. - The whole interface is three calls: save a take, delete one, read them all.
(async () => {
renderTakes(-1); renderWords(); drawRoll();
const saved = await DB.all();
if (!saved || !saved.length) return;
for (const r of saved) {
const raw = r.data instanceof Float32Array ? r.data : new Float32Array(r.data);
if (r.kind === 'note' && r.slot >= 0 && r.slot < 6) takes[r.slot] = { raw, clip: V.analyzeClip(raw, { kind: 'note', label: noteName(r.target) }) };
if (r.kind === 'word') words.push({ id: r.id, label: r.label, raw, clip: V.analyzeClip(raw, { kind: 'word', label: r.label }) });
}
renderTakes(-1); renderWords(); rerender();
})();- On load, read back whatever was saved earlier; a first visit finds nothing and stops here.
- Only the raw samples are stored. Pitch tracks and marks are recomputed from them, so an improvement to the analysis applies to old takes too.
Browsers only grant microphone access on secure (https) pages. GitHub Pages serves the site over https, so there is still no server of its own.
The backing track
Piano chords, or the original with the vocal turned down
Before re-singing, Pianoify finds the song's beat, bar lines, chords and melody, and the melody is moved by whole octaves into the middle of the range you sang. Under your voice you can then choose one of four backings: piano chords, the original song with its vocal reduced, the original, or nothing.
“Piano chords” plays Pianoify's arrangement. “Vocal reduced” is the karaoke trick: lead vocals are usually panned dead centre, so subtracting the left channel from the right removes them. The centre's low end is then added back to keep the bass and kick.
How well it works
Checked without a microphone
The test sings six synthetic held notes and three words (a formant-synthesised voice with vibrato, jitter and breath noise), re-sings Pianoify's demo melody with them, then pitch-tracks the result.
| Pitch of each take found | within 0.2 cents |
|---|---|
| Re-sung notes within 30 cents of the score | 52 of 52 |
| Median pitch error of re-sung notes | 0.3 cents |
| First formant of “ah”, original | 861 Hz |
| … moved up 5 semitones with PSOLA | 807 Hz |
| … moved up 5 semitones by speeding up | 996 Hz |
| Time to render 46 s of vocal | 0.2 s |
The formant rows are the point of PSOLA (section 03). Speeding a recording up drags its vowel up with it, which is the chipmunk sound. PSOLA keeps the vowel where your mouth put it.
Limits
What it can't do
- It sings the tune on your sounds, not the song's lyrics. Putting the real words in your voice needs a neural singing-voice-conversion model (such as Seed-VC or RVC) running on a GPU. Such a model re-voices the original singer's isolated vocal, which can't run on a static page.
- Moving a recording more than about five semitones starts to sound buzzy. Recording all six notes keeps every melody note close to one of them. Words are sung on one note, so they stretch further.
- Vocal reduction only removes a centre-panned voice. Reverb, doubled vocals and anything else in the centre (snare, bass) partly go with it.
- The melody is whatever Pianoify hears as the tune, so a wrong note there is a wrong note here.
Reproduce
Run the checks yourself
Plain JavaScript with no audio libraries. This runs the checks in How well it works; it needs ffmpeg to decode the song.
node tests/run_voice.js
Read further
Where everything lives
- web/voice.js YIN, pitch marks, TD-PSOLA, melody rendering, chord backing, vocal reduction, WAV export
- web/app.js the page: microphone, guided takes, live tuner, IndexedDB, mixing, playback
- tests/run_voice.js pitch and formant checks with a synthetic voice, no microphone needed
- ../pianoify/ finds the song's melody and chords