Onsets and Frames: Dual-Objective Piano Transcription

Summary

Foundational neural piano transcription architecture combining a CNN acoustic model with a bidirectional LSTM temporal model. Uses dual objectives: one head predicts note onsets, the other predicts frame-level note activations. Onset predictions are used to post-process frame activations into clean note events. Trained on the MAESTRO dataset (200+ hours of aligned piano audio + MIDI). 457+ citations, making it the most influential piano transcription paper.

Key Claims

• Joint onset + frame prediction significantly improves transcription accuracy over frame-only approaches
• The onset detector acts as an attention mechanism — the model learns where to "look" for note boundaries
• CNN + BiLSTM architecture captures both local timbral features and longer temporal dependencies
• The MAESTRO dataset (created alongside this work) enables training high-quality transcription models

Related

• ../entities/onsets-and-frames — the tool (part of Magenta, now archived)
• ../entities/basic-pitch — more recent, instrument-agnostic approach
• The onset+frame architecture is well-suited to plucked string instruments (sharp attacks)
• ../entities/mt3 — successor using T5 transformer instead of CNN+BiLSTM