When you drop a song into the free transcription tool, a few seconds later you see notes falling onto a keyboard. Here’s what happens in between, in plain language.
The problem: hearing notes in a waveform
A recording is just a long list of numbers: the position of a speaker cone, measured tens of thousands of times a second. Nothing in that list says “C4 starts here”. A single piano note isn’t one frequency either — it’s a fundamental plus a stack of overtones at two, three, four times that frequency, all fading at different rates. Play a chord and those stacks overlap. Untangling them is the core challenge of automatic music transcription (AMT).
Step 1: From sound to a picture
The first step is to turn audio into a time–frequency picture, a spectrogram: time runs left to right, pitch runs bottom to top, and brightness shows how much energy is at each pitch at each moment. Music transcription models usually use a constant-Q transform, whose frequency bins are spaced like the keys of a piano — every semitone gets the same number of bins — so a melody looks like a shape you could almost read.
Basic Pitch goes one step further with a harmonic constant-Q transform: it stacks shifted copies of the spectrogram so that a note’s fundamental and its overtones line up. That makes it much easier for the network to tell “a C with strong overtones” from “a C plus a G an octave and a fifth above”.
Step 2: A neural network listens
A small convolutional neural network looks at that picture and produces three outputs for every frame (about 86 frames per second):
- Onsets — how likely it is that a note starts at each pitch right now.
- Notes — how likely it is that a note is sounding at each pitch.
- Contours — a finer pitch track, a third of a semitone wide, that captures vibrato and bends.
Basic Pitch was published by Spotify’s Audio Intelligence Lab in 2022. Its standout feature is size: it’s light enough to run in a web browser on a phone, yet it handles polyphony and works across many instruments, not just piano.
Step 3: From probabilities to notes
The network gives probabilities, not notes. A decoding step turns them into events: wherever the onset probability peaks above a threshold, a note begins; it lasts while the note probability stays above a second threshold; very short blips are thrown away. That’s exactly what the Notes setting in our tool adjusts — Fewer raises the thresholds, More lowers them — which is why it updates instantly without running the network again.
Step 4: Notes become MIDI
Each note now has a pitch, start, end and loudness — the contents of a MIDI file. From there we draw it as a piano roll, play it on a sampled grand, and let you export it.
Why it runs in your browser
The model runs with TensorFlow.js, which uses your device’s graphics chip. That brings three benefits: privacy (audio never leaves your device), speed (no upload or queue) and cost (it’s free for everyone, because there is no server doing the work).
What AI transcription still finds hard
- Dense mixes. Drums and distorted guitars spray energy across every frequency.
- Pedal-heavy piano. With the sustain pedal down, notes overlap and their ends are ambiguous.
- Octave errors. A strong overtone can be mistaken for a note an octave up.
- Expressive timing. Real players push and pull the beat. The model records exactly when notes happened, which is great for playback but needs quantising for sheet music.
Knowing these limits helps you get better results — see 10 tips for cleaner piano transcriptions.
Frequently asked questions
Is AI transcription better than transcribing by ear?
It’s faster, and good at catching every note in a chord. A trained ear is still better at judgment calls — which notes matter, which hand plays what. The best workflow is AI first, ear second. Train yours with learning piano by ear.
Does Basic Pitch only work for piano?
No — it was trained on many instruments and voices. Results are best for a single instrument at a time.
Is my audio used to train the model?
No. The model runs on your device and nothing is sent back to us or to Spotify.
Try it now: everything in this guide works in the free a song for free. For lessons, backing tracks and transcriptions, subscribe to GrooveForte on YouTube.