Skip to content

Hybrid Transformers for Music Source Separation

Summary

Extends Hybrid Demucs by replacing the innermost U-Net layers with a cross-domain Transformer encoder. The Transformer applies self-attention within each domain (time, frequency) and cross-attention between them. Achieved 9.20 dB SDR on MUSDB18 with extra training data, establishing a new state of the art. The hybrid architecture lets the model learn whether waveform or spectrogram processing is better for each source.

Key Claims

  • Cross-domain Transformer attention significantly outperforms LSTM bottlenecks
  • Self-attention within time/frequency domains + cross-attention between them captures long-range dependencies
  • The model learns to weight waveform vs. spectrogram processing per source (vocals benefit from spectrogram, drums from waveform)
  • 4-stem and 6-stem variants available; both outperform all prior work