Skip to content

A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription

Summary

Spotify's lightweight neural network for converting audio to MIDI. Uses a harmonic CQT representation fed through a small CNN to predict multi-pitch activations and note events with pitch bends. Designed for consumer use: ~15MB model, runs on CPU in near real-time, works on any pitched instrument (voice, violin, guitar, etc.). Instrument-agnostic by design — no instrument-specific training needed.

Key Claims

  • A small harmonic CQT + CNN architecture can achieve competitive polyphonic transcription
  • Instrument-agnostic training works: the model generalizes across instrument timbres
  • Pitch bend detection is critical for realistic transcription of non-keyboard instruments
  • 15MB model size enables on-device consumer deployment