MT3

About

Google's T5-based multi-instrument transcription model. Treats music transcription as a sequence-to-sequence task, tokenizing MIDI into a vocabulary and using a T5 transformer to decode instrument-specific note tokens from spectrogram inputs. Transcribes multiple instruments simultaneously from a single audio mixture. Trained on Slakh2100.

Relevance

Limited for banjo: the instrument vocabulary does not include "banjo." Could be extended by adding a banjo token and fine-tuning, but the T5 architecture is heavy (needs GPU). More useful as a reference for the token-based transcription approach than as a practical tool for bluegrass.

Mentions

• ../entities/basic-pitch — lighter alternative
• ../entities/onsets-and-frames — predecessor CNN+BiLSTM approach
• Paper: ICLR 2022 (oral)

Links

• GitHub: https://github.com/magenta/mt3 (1.7k stars)
• PyTorch port: https://github.com/rlax59us/MT3-pytorch