William Chang /Projects/Pianoify Back to project

How it works

Pianoify

A recording is one waveform with every instrument mixed together. Pianoify pulls the beat, the bar lines, the chords and the tune back out of it, then writes the simplest piano part that keeps all four. Below are the six ideas that make it work, each with the few lines of code that carry it.

  • Signal processing
  • Hidden Markov models
  • Dynamic programming
  • Python to JavaScript
  • Evaluation

From audio to sheet music

  1. Spectrathree FFTs, resampled onto musical pitches
  2. Beattempo from autocorrelation, beats by dynamic programming
  3. Chords & barstwo HMMs, one for the chord and one for beat 1
  4. Melodyharmonic summation, then a smooth Viterbi path
  5. Arrangeright hand melody, voice-led left-hand triads, MusicXML

The ideas

  1. 01Hear in semitones, not hertz
  2. 02Find the pulse, then place the beats
  3. 03Chords that change only when the evidence says so
  4. 04A second HMM for where beat 1 falls
  5. 05Follow the voice through the band
  6. 06A browser port, scored against known answers
  7. →Writing the piano part
  8. →How well it works
  9. →Limits
  10. →Reproduce
  11. →Read further
01

Hear in semitones, not hertz

Short-time FFTs on a musical scale, with the drums filtered out

An FFT gives energy at evenly spaced frequencies, but notes are spaced evenly on a log scale: each octave doubles the frequency. So every spectrum frame is resampled onto pitch bins, one per semitone for harmony and one per third of a semitone for the melody. Each job gets its own window length, because a long window resolves pitch well and a short one resolves time well.

web/analysis.jslines 174–189
  // A) harmony: 8192-point STFT → semitone spectrum E1..E7
  progress('Listening for harmony', 0.05);
  const mapA = makeLogMap(8192, CH_FMIN, CH_BINS, 12, 0.25);
  const A = new Float32Array(nFramesTotal * CH_BINS);
  const tmpA = new Float64Array(CH_BINS);
  stft(y, 8192, HOP, (f, mag) => { mapA(mag, tmpA); A.set(tmpA, f * CH_BINS); });
  progress('Separating drums from harmony', 0.25);
  harmonicMask(A, nFramesTotal, CH_BINS, 17, 9);

  // B) melody: 4096-point STFT → 1/3-semitone spectrum C2..C9
  progress('Listening for the melody', 0.35);
  const mapB = makeLogMap(4096, MEL_FMIN, MEL_BINS, MEL_BPO);
  const B = new Float32Array(nFramesTotal * MEL_BINS);
  const tmpB = new Float64Array(MEL_BINS);
  stft(y, 4096, HOP, (f, mag) => { mapB(mag, tmpB); B.set(tmpB, f * MEL_BINS); });
  harmonicMask(B, nFramesTotal, MEL_BINS, 17, 13);
  1. Harmony uses a long 8192-sample window (about 0.37 s at 22,050 Hz) and 72 semitone bins from E1 upward. Long windows separate neighbouring low notes.
  2. The STFT calls back once per frame; each magnitude spectrum is mapped to pitch bins and copied into one flat Float32Array, so a whole song's spectrogram is a single typed array.
  3. Drums are removed before any chord is read (next snippet).
  4. The melody gets its own 4096-sample STFT and finer bins: 36 per octave, so a slightly sharp singer still lands near the right note.

Harmonic/percussive separation (HPSS) rests on one observation about a spectrogram: a held chord is a horizontal line (steady across time), while a drum hit is a vertical line (spread across frequency). Median filters along each direction estimate the two parts.

web/analysis.jslines 151–163
/** Soft harmonic mask (HPSS) on a frames × bins spectrogram stored frame-major. */
function harmonicMask(S, nFrames, nBins, wTime, wFreq) {
  // transpose to bins × frames for the time median
  const T = new Float32Array(nBins * nFrames);
  for (let f = 0; f < nFrames; f++) for (let b = 0; b < nBins; b++) T[b * nFrames + f] = S[f * nBins + b];
  const H = medianFilterRows(T, nBins, nFrames, wTime, 1);
  const P = medianFilterRows(S, nFrames, nBins, wFreq, 1);
  for (let f = 0; f < nFrames; f++) for (let b = 0; b < nBins; b++) {
    const h = H[b * nFrames + f], p = P[f * nBins + b];
    const m = (h * h) / (h * h + p * p + 1e-12);
    S[f * nBins + b] *= m;
  }
}
  1. Transpose so each pitch bin is one row, then take a 17-frame running median along time. Sustained notes survive this; short hits are voted out.
  2. A median across neighbouring pitch bins in each frame: high where energy is spread over many frequencies at once, which is what drums look like.
  3. A soft mask, h² / (h² + p²), keeps each cell in proportion to how “harmonic” it looks, instead of a hard yes or no.
02

Find the pulse, then place the beats

Autocorrelation for the tempo, Ellis's dynamic program for the beats

A third, short STFT (2048 samples) measures spectral flux: how much louder each frequency band got since the last frame. Peaks in that curve are onsets. If the song has a steady tempo, the onset curve looks like itself shifted by one beat, so the tempo is the shift (lag) with the strongest autocorrelation.

web/analysis.jslines 235–246
  // tempo: autocorrelation weighted by a log-normal prior around 120 BPM (or the hint)
  const fps = SR / HOP, maxLag = Math.round(4 * fps);
  let best = 0, bestLag = Math.round(fps / 2);
  const center = bpmHint || 120, spread = bpmHint ? 0.15 : 1.0;
  for (let lag = Math.round(fps * 60 / 240); lag <= maxLag; lag++) {
    let ac = 0;
    for (let i = lag; i < n; i++) ac += o[i] * o[i - lag];
    ac /= (n - lag);
    const bpm = 60 * fps / lag;
    const w = Math.exp(-0.5 * Math.pow(Math.log2(bpm / center) / spread, 2));
    if (ac * w > best) { best = ac * w; bestLag = lag; }
  }
  1. The prior: centred on 120 BPM with a spread of one octave, or tightly around a tempo the user supplied.
  2. Try every lag from one beat at 240 BPM up to 4 seconds, and correlate the onset curve with a shifted copy of itself.
  3. A log-normal weight on BPM. Without it, half and double tempos score almost as well as the true one, since a song that repeats every beat also repeats every two beats.

Knowing the period is not the same as knowing where the beats are. Dan Ellis's dynamic program picks beat times that land on strong onsets while keeping the gaps close to one period. Each frame stores the best score of a beat sequence ending there and a pointer to the previous beat; following the pointers back from the end gives the answer.

web/analysis.jslines 258–269
  const cum = new Float64Array(n), back = new Int32Array(n).fill(-1);
  const tight = 100;
  for (let i = 0; i < n; i++) {
    let bv = -Infinity, bi = -1;
    for (let j = i - 2 * period; j <= i - Math.round(period / 2); j++) {
      if (j < 0) continue;
      const v = cum[j] - tight * Math.pow(Math.log((i - j) / period), 2);
      if (v > bv) { bv = v; bi = j; }
    }
    cum[i] = local[i] + (bi >= 0 ? bv : 0);
    back[i] = bi;
  }
  1. The previous beat can only be between half a period and two periods back.
  2. The penalty grows with the squared log of how far the gap is from one period, so stretching a gap by 10% costs the same whether the tempo is fast or slow.
  3. Best score here = onset strength here + best score of a compatible earlier beat. back remembers which one, for the backtrack.
03

Chords that change only when the evidence says so

Template matching, a bass bonus and Viterbi smoothing

For each beat, the spectrum is folded into a 12-bin chroma vector: how much C, C♯, D and so on is sounding, ignoring octave. Each candidate chord is a 12-bin template (G major is G, B and D), and the score is a cosine similarity. Scoring every beat on its own gives jittery output, so a hidden Markov model (HMM) decides the sequence: a chord prefers to stay, and changes only when several beats agree.

The key is found first, because the chord scores use it: the song's overall pitch-class profile is correlated against the Krumhansl–Kessler profiles for all 24 major and minor keys, and the best match wins.

pianoify.pylines 218–246
def detect_chords(chroma_b, bass_b, energy_b, key, sevenths=False, p_stay=0.85):
    vocab = chord_vocab(sevenths)
    T = chord_templates(vocab)
    C = chroma_b / (np.linalg.norm(chroma_b, axis=0, keepdims=True) + 1e-9)
    scores = T @ C                                          # (n_chords, n_beats)

    # bass evidence: does the lowest register agree with the chord root?
    B = bass_b / (bass_b.sum(axis=0, keepdims=True) + 1e-9)
    diat = diatonic_chords(*key)
    for i, (r, q) in enumerate(vocab):
        if r is None:
            continue
        scores[i] += 0.25 * B[r]
        if (r, q) in diat:
            scores[i] += 0.04                               # gentle key prior
        if q not in ("", "m"):
            scores[i] -= 0.03                               # prefer plain triads

    # silence / non-harmonic beats → N.C.
    quiet = energy_b < 0.08 * np.percentile(energy_b, 90)
    raw = T[1:] @ C
    atonal = (raw.max(axis=0) - np.median(raw, axis=0)) < 0.22         # flat chroma = drums/noise
    scores[0] = np.where(quiet | atonal, 2.0, 0.35)

    probs = np.exp(25 * (scores - scores.max(axis=0, keepdims=True)))
    probs /= probs.sum(axis=0, keepdims=True)
    trans = librosa.sequence.transition_loop(len(vocab), p_stay)
    path = librosa.sequence.viterbi(probs, trans)
    return [vocab[i] for i in path]
  1. One matrix product scores all 24 major and minor chords (or 60 with sevenths) against every beat at once.
  2. The bass register gets a vote: a chord whose root is the loudest bass note gets up to 0.25 extra. This is what tells C major from A minor, which share two notes.
  3. Two small nudges: chords that belong to the detected key get 0.04, and seventh chords pay 0.03 so plain triads win close calls.
  4. If no chord stands out from the rest (flat chroma, typical of drums or noise), or the beat is nearly silent, “no chord” wins outright.
  5. Viterbi decoding with a self-loop probability of 0.85 finds the single most likely chord sequence, which is what removes one-beat flickers.

The browser version can't call librosa, so it carries its own Viterbi decoder. It works in log probabilities (sums instead of products, so long songs don't underflow to zero) and takes the transition rule as a function, so the chord, bar-line and melody models all share it.

web/analysis.jslines 338–355
function viterbi(logEmit, nStates, T, logTrans, logInit) {
  // logEmit(t, s); logTrans(prevState, s) or a specialised maxPrev(delta, s) function
  let delta = new Float64Array(nStates), next = new Float64Array(nStates);
  const ptr = new Int32Array(T * nStates);
  for (let s = 0; s < nStates; s++) delta[s] = (logInit ? logInit[s] : -Math.log(nStates)) + logEmit(0, s);
  for (let t = 1; t < T; t++) {
    for (let s = 0; s < nStates; s++) {
      const [v, k] = logTrans(delta, s);
      next[s] = v + logEmit(t, s);
      ptr[t * nStates + s] = k;
    }
    const tmp = delta; delta = next; next = tmp;
  }
  const path = new Int32Array(T);
  path[T - 1] = argmax(delta);
  for (let t = T - 1; t > 0; t--) path[t - 1] = ptr[t * nStates + path[t]];
  return path;
}
  1. For each state, logTrans returns the best predecessor and its score. The chord model passes a simple “stay or switch” rule; the melody model passes a smarter one that runs in less than quadratic time.
  2. Pick the best final state, then follow the stored back-pointers to recover the whole path.
04

A second HMM for where beat 1 falls

Position-in-bar tracking that can re-sync

Beats alone don't say where bars start. Pianoify uses a musical habit: chords tend to change, and the bass tends to hit, on beat 1. A second HMM has one hidden state per position in the bar (four in 4/4). It normally steps 1, 2, 3, 4, 1, but it is allowed a rare jump, so one beat the tracker dropped does not shift every bar line after it.

pianoify.pylines 249–270
def track_downbeats(labels, chroma_b, low_onset_b, beats_per_bar, p_resync=0.004):
    """HMM over position-in-bar. Beat 1 is where harmony changes and the bass hits;
    the model normally advances one position per beat but may re-sync, so a dropped
    beat or an odd-length bar doesn't shift every bar line after it."""
    n, B = len(labels), beats_per_bar
    C = chroma_b / (np.linalg.norm(chroma_b, axis=0, keepdims=True) + 1e-9)
    flux = np.r_[0, 1 - np.sum(C[:, 1:] * C[:, :-1], axis=0)]
    flux = flux / (np.percentile(flux, 95) + 1e-9)
    change = np.array([0.0] + [float(labels[i] != labels[i - 1]) for i in range(1, n)])
    lo = low_onset_b / (np.percentile(low_onset_b, 95) + 1e-9)
    ev = 2.0 * change + np.clip(flux, 0, 2) + 0.7 * np.clip(lo, 0, 2)
    logp = np.zeros((B, n))
    logp[0] = 1.2 * ev
    if B % 2 == 0:
        logp[B // 2] = 0.6 * ev          # half-bar changes are common but less likely
    P = np.exp(logp - logp.max(axis=0, keepdims=True))
    P /= P.sum(axis=0, keepdims=True)
    trans = np.full((B, B), p_resync / (B - 1))
    for s in range(B):
        trans[s, (s + 1) % B] = 1 - p_resync
    pos = librosa.sequence.viterbi(P, trans, p_init=np.full(B, 1 / B))
    return [i for i in range(n) if pos[i] == 0]
  1. Evidence that a beat is a downbeat: the chord label changed (weight 2), the chroma changed (cosine distance), and there was a low-frequency onset (weight 0.7).
  2. Position 0 (beat 1) is rewarded for that evidence. In 4/4, beat 3 gets half the reward, because songs often change chord mid-bar.
  3. The transition matrix: step to the next position with probability 0.996, jump anywhere else with 0.004. That small leak is what lets the model re-sync.

The downbeats then rebuild the beat grid, so that every bar has exactly four evenly spaced beats, and the chords are recomputed on the new grid.

pianoify.pylines 273–288
def regrid(beat_times, downbeats, B):
    """New beat grid with exactly B evenly spaced beats between consecutive downbeats.
    Returns (grid, index of the first downbeat in the grid)."""
    if len(downbeats) < 2:
        return np.asarray(beat_times), 0
    period = float(np.median(np.diff(beat_times)))
    d0 = beat_times[downbeats[0]]
    pre = list(np.arange(d0 - period, -1e-6, -period))[::-1]
    out = pre[:]
    for a, b in zip(downbeats[:-1], downbeats[1:]):
        out.extend(np.linspace(beat_times[a], beat_times[b], B, endpoint=False))
    t = beat_times[downbeats[-1]]
    while t <= beat_times[-1] + 1e-6:
        out.append(t)
        t += period
    return np.array(out), len(pre)
  1. Between two downbeats, place exactly B beats evenly. An odd bar from the tracker is stretched to fit, which is the stated assumption of a steady meter.
05

Follow the voice through the band

Harmonic summation, median subtraction and a smooth Viterbi path

A sung note is not one frequency but a stack of harmonics at 1, 2, 3 … times the fundamental. So the salience of a candidate pitch is the weighted sum of the spectrum at its first eight harmonics. Then anything that stays put for seconds (held chords, pads) is treated as accompaniment and subtracted.

web/analysis.jslines 494–510
  const lo = (MEL_LO - MEL_FMIN) * 3, hi = (MEL_HI - MEL_FMIN) * 3, nc = hi - lo + 1;
  const offs = [], wts = [];
  for (let h = 1; h <= 8; h++) { offs.push(Math.round(MEL_BPO * Math.log2(h))); wts.push(Math.pow(0.8, h - 1)); }
  // S stored candidate-major for the time-median filter
  const S = new Float32Array(nc * nFrames);
  for (let f = 0; f < nFrames; f++) {
    const base = f * MEL_BINS;
    for (let c = 0; c < nc; c++) {
      let s = 0;
      for (let h = 0; h < 8; h++) { const k = lo + c + offs[h]; if (k < MEL_BINS) s += wts[h] * C[base + k]; }
      S[c * nFrames + f] = s;
    }
  }
  // suppress sustained accompaniment: subtract 0.6 × running median over ~4 s
  const w = (Math.round(4 * SR / HOP)) | 1;
  const med = medianFilterRows(S, nc, nFrames, w, 8);
  for (let i = 0; i < S.length; i++) S[i] = Math.max(S[i] - 0.6 * med[i], 0);
  1. Harmonic h sits log₂(h) octaves above the fundamental, which on a log-frequency axis is a fixed offset of round(36·log2 h) bins. Higher harmonics count for less (0.8h−1).
  2. Salience of every candidate from C3 to C6 in every frame: one shifted, weighted sum.
  3. A running median over about 4 seconds estimates the sustained background for each pitch; 0.6 of it is subtracted. The median is only recomputed every 8 frames, which keeps this step fast in JavaScript.

The loudest pitch in each frame jumps around. A third Viterbi pass follows a smooth line instead: moving a step or two is cheap, and a big leap is allowed but costs extra. Afterwards a check moves the pitch down an octave when the octave below shows its own evidence, since a voice's even harmonics can make the wrong octave look salient.

web/analysis.jslines 532–543
  const leap = Math.log(0.02 / nc);
  const trans = (delta, s) => {
    let best = -Infinity, k = 0, gm = -Infinity, gk = 0;
    for (let p = 0; p < nc; p++) if (delta[p] > gm) { gm = delta[p]; gk = p; }
    for (let d = -W; d <= W; d++) {
      const p = s + d; if (p < 0 || p >= nc) continue;
      const v = delta[p] + Math.log(0.98 * tri[d + W] / rowNorm[p] + 0.02 / nc);
      if (v > best) { best = v; k = p; }
    }
    if (gm + leap > best) { best = gm + leap; k = gk; }
    return [best, k];
  };
  1. The best state anywhere in the previous frame, found once per frame and shared by every target state.
  2. Nearby pitches (up to six bins, two semitones, either way) are scored with a triangular kernel: the closer, the more likely.
  3. Any other leap competes only against that one global best. This makes each step O(states) instead of O(states²), about 109 candidates instead of 109² comparisons per frame.

The path is then cut into notes, snapped to an eighth-note grid (quarter notes in Easy) and moved into a comfortable right-hand range.

06

A browser port, scored against known answers

Same pipeline in plain JavaScript, in a Web Worker, checked by the same evaluator

The Python tool is built on librosa and music21, which can't run on a static web page. web/analysis.js is a from-scratch port: its own FFT, STFT, median filters, Viterbi and MusicXML writer, with no audio libraries. The same file runs in three places: a Web Worker in the browser, the main thread as a fallback, and Node for the tests.

web/analysis.jslines 1016–1033
// Web Worker protocol: {cmd:'features', pcm} → progress…, {type:'features', feats}; {cmd:'arrange', opts} → {type:'song'}
if (typeof WorkerGlobalScope !== 'undefined' && root instanceof WorkerGlobalScope) {
  let F = null;
  root.onmessage = e => {
    const m = e.data;
    try {
      if (m.cmd === 'features') {
        F = features(m.pcm, (msg, p) => root.postMessage({ type: 'progress', msg, p }));
        root.postMessage({ type: 'ready' });
      } else if (m.cmd === 'arrange') {
        root.postMessage({ type: 'song', song: arrange(F, m.opts) });
      }
    } catch (err) { root.postMessage({ type: 'error', msg: String(err && err.stack || err) }); }
  };
}
if (typeof module !== 'undefined' && module.exports) module.exports = api;
else root.Pianoify = api;
})(typeof self !== 'undefined' ? self : this);
  1. Only inside a worker does the file install a message handler, so the same code can be loaded as a plain script too.
  2. The slow part (spectra, onset curve and melody contour) runs once per song while the page stays responsive and shows progress messages.
  3. Changing level, key or tempo re-runs only arrange on the stored features: beats, key, chords, bars, melody notes and MusicXML, with no new FFTs.
  4. Under Node it exports a module, which is how tests/run_js.js runs the browser pipeline on the test song.

Accuracy on real songs is hard to measure, so tests/make_test_song.py synthesises a song from a score (drums, bass, chords and a lead voice) and writes the answers to a JSON file. One evaluator scores both versions. Chords are scored by time, every 20 ms:

tests/evaluate.pylines 21–32
    # chords: fraction of time with the correct label (root + maj/min)
    det = [(beat_to_time(bt, org, c["start_beat"]), beat_to_time(bt, org, c["start_beat"] + c["beats"]),
            c["name"]) for c in an["chords"]]
    ok = tot = 0
    for t in np.arange(0, end, step):
        truth = next((c["chord"] for c in tr["chords"] if c["start"] <= t < c["end"]), "N")
        guess = next((n for a, b, n in det if a <= t < b), "N.C.")
        if truth == "N":
            continue
        tot += 1
        ok += guess == truth
    chord_acc = ok / tot
  1. Detected chords are in beats; they are converted back to seconds through the detected beat grid, so a wrong tempo or bar line also costs accuracy.
  2. Sample every 20 ms, skip moments the truth marks as no chord, and count how often the label matches.
Result

On the test song (G major, 96 BPM, 17 chord changes including one in the middle of a bar), both versions find G major and 95.7 BPM. Chords are right 98.5% of the time in Python and 99.6% in the browser. The full comparison is in How well it works.

→

Writing the piano part

Melody in the right hand, voice-led chords in the left

The right hand plays the melody from section 05. The left hand plays the chord as a triad, choosing the inversion that moves the hand least from the previous chord, struck again on every bar. Easy mode uses root and fifth instead. The result is written as MusicXML and engraved by Verovio.

→

How well it works

Scored against a song with known answers

The demo song is synthesised from a score (drums, bass, chords and a sung-style lead), so the right answers are known exactly. It's in G major at 96 BPM, with 17 chord changes including one in the middle of a bar.

PythonBrowser
Tempo (true 96)95.795.7
Key (true G major)G majorG major
Chords, share of time correct98.5%99.6%
Melody, exact pitch92.7%95.0%
Note starts found (recall)92.3%96.2%
Note starts that are real (precision)92.3%89.3%

On real recordings the chords and key are usually right. The melody is the hardest part, because the singer competes with everything else in the mix. It comes out as a good sketch of the tune rather than a finished transcription. The Python version can first split the vocal out with Demucs, which helps a lot.

→

Limits

What it gets wrong, and what it doesn't try

  • Transposing moves the melody by whole octaves when needed to keep it on the treble staff.
  • Tempo can come out doubled or halved, which is a judgement call even for people. Use Count it on the project page to fix it.
  • It assumes four beats to a bar and a steady meter. A genuinely odd bar is stretched to fit.
  • Chords are major and minor triads, plus 7ths if you ask. Sus, diminished, augmented and slash chords aren't recognised yet.
  • The melody is one line: the loudest tune at each moment. Harmonies, backing vocals and lyrics aren't transcribed.
→

Reproduce

The Python tool and the tests

The original is a Python command-line tool built on librosa, music21 and Verovio. It writes a PDF, MusicXML, MIDI and a chord chart for any audio file:

pip install -r requirements.txt
python pianoify.py song.mp3 -o out/
python pianoify.py song.mp3 --level easy --to-key C
python pianoify.py song.mp3 --separate      # Demucs vocal split (pip install demucs)

The project page runs a JavaScript port of the same pipeline, written from scratch with no audio libraries. Both are scored by the same evaluator:

python tests/make_test_song.py     # builds the test song and its answer key
node tests/run_js.js               # runs the browser version under Node
python tests/evaluate.py           # compares either one with the truth
→

Read further

Where everything lives