The Science Behind AI Audio Separation: How Demucs Works

When you upload a track to Stemify and receive four clean stems minutes later, it can feel like magic. It isn't — it's a deep neural network trained on thousands of hours of music. Here's what's actually happening, in plain terms.

The Core Problem

Audio is a single time-domain waveform. A full mix — vocals, drums, bass, guitars, keys — collapses all of those independent signals into one signal. Separating them is called the cocktail party problem: how do you isolate one voice in a room full of people all talking at once?

For decades, the answer was: you can't, reliably. You could use spectral subtraction tricks for simple cases (like midfield vocal cancellation for mono recordings), but true separation was considered nearly impossible without the original session files.

How Demucs Approaches It

Demucs (Deep Extractor for Music Sources) uses a U-Net architecture — the same class of neural network used in medical image segmentation. It processes audio in two representations simultaneously:

• Time domain — the raw waveform, with its precise temporal information
• Frequency domain — a spectrogram, which shows how frequency content evolves over time

The model learns to associate patterns in these representations with specific instruments. A snare drum has a characteristic transient shape in the time domain and a specific frequency spread in the spectrogram. Vocals have formant patterns, vibrato, and harmonic structures that differ from string instruments. The network learned all of this by being trained on music where the ground truth stems were known — i.e., music where the original multitrack sessions were available for supervised training.

Why Some Tracks Separate Better Than Others

The model is only as good as the patterns it learned. It struggles with:

• Heavily distorted guitars — their harmonics overlap with many other instruments
• Synthesizers that mimic acoustic instruments — the model may misclassify them
• Extreme reverb — the reverb tail blends sources together in frequency
• Lo-fi recordings — compression artifacts obscure the instrument signatures

HTDemucs: The Current State of the Art

Stemify uses HTDemucs (Hybrid Transformer Demucs), the most recent version. It adds a Transformer attention mechanism to the encoder, allowing the model to attend to long-range dependencies in the audio — important for understanding how a guitar phrase in bar 2 relates to a vocal phrase in bar 10. This significantly improved separation quality for complex arrangements.

What This Means for You

AI separation is a tool, not a replacement for the original recording. Think of it as a very sophisticated starting point. The cleaner your source material and the more distinct the instruments are in the mix, the better your stems will be. For the vast majority of practical uses — remixing, practice, DJ performance — the results are more than sufficient.