The developer went all out for 6 months to "twist" regular stereo into spatial audio.

One person, six months, in spare time, built a diffusion model that can “twist” ordinary stereo music into 7.1.4-channel spatial binaural audio.

This happened in Reddit’s r/MachineLearning community. The project is called Stereo2Spatial, and the creator’s motivation could not be simpler—he enjoys listening to spatial audio versions of songs, but there are too few high-quality spatial mixes on the market. So, he decided to train a model to do the conversion himself. The open-source code, model weights, and inference pipeline have all been released.

This is not one of those “demo-runs-then-paper-published” academic projects, but a work that genuinely solves a real need and smooths out all the engineering details.

Why This Matters

First, some background. Spatial audio has been heavily pushed in recent years by Apple, Dolby Atmos, and Sony 360 Reality Audio. Apple’s head-tracked spatial audio experience on AirPods has indeed made some users “never go back.” But the industry faces an awkward reality: a severe lack of content.

The vast majority of music on streaming platforms is still in stereo. Even tracks labeled “Dolby Atmos” on Apple Music or Tidal are often mechanically upmixed from stereo masters—the sound field is widened, but the sense of direction and immersion remains weak. True Atmos tracks that were remixed from multitrack masters make up only a small fraction of the catalog, because that process requires human labor, studios, engineers—and costs far beyond what streaming royalties can cover.

This is the gap Stereo2Spatial aims to fill: using a generative model to “reimagine” stereo as spatial audio, instead of simply delaying, adding reverb, or positioning the two channels in a virtual speaker layout.

Traditional stereo-to-spatial solutions—Windows Sonic, Dolby Atmos for Headphones, or various plugins—are fundamentally rendering approaches: they use HRTFs (Head-Related Transfer Functions) to place existing sound channels at fixed points in virtual space. These do not create new spatial information; they just spatialize the existing two channels psychoacoustically.

Generative models take a completely different path—they attempt to infer where each sound element in the original recording should be in space: the drums behind, vocals front-center, backing vocals upper sides—and then reassign everything to the 7.1.4’s 11 output channels. That’s true “upmixing.”

Technical Breakdown: More Than Just Plugging in a Diffusion Model

The author made restrained and clear design choices—no over-engineering, just solid reasoning.

Step one: train a VAE. They trained a variational autoencoder called EAR-VAE to compress audio into latent space. This is the modern standard for audio generation models—Stable Audio, AudioLDM all do this. Running diffusion directly on raw waveforms is simply too heavy.

Step two: a flow-matching diffusion model. Not DDPM, but flow matching—a more recent direction in diffusion models, popular in image and now audio generation for its stability and faster sampling. Meta’s Voicebox and Stable Audio 2 both use similar approaches.

The key architectural choice: the VAE encodes the stereo input into one latent representation, while each of the 7.1.4 output channels is encoded separately. That means the model learns a mapping from “one stereo latent” to “11 mono latents.”

This design was inspired by a paper called ImmersiveFlow. The author said all the code links in that paper were 404, so he rewrote everything from scratch—something anyone who’s tried to reproduce academic work will understand.

The Real Innovation: Memory Tokens

If it only reproduced ImmersiveFlow, this wouldn’t be that interesting. The author’s key improvement was adding a cross-window memory token mechanism.

Here’s the background: diffusion models usually process long audio in short, independent windows. The problem is that each window doesn’t know what happened previously, leading to weird discontinuities when the outputs are stitched together—drums jump from left to right mid-beat; vocals drift in space.

The author’s solution was to have the model carry a set of memory tokens from one window to the next, passing along the prior “spatial state.” It’s reminiscent of how Transformer-XL handles text or how temporal attention works in video diffusion models—but applying such cross-window spatial consistency to audio spatialization is still rare in open research.

And the effect? Stable spatial coherence across long tracks. Listen to a 4-minute song; the drums won’t drift around every few seconds. Without solving this, even the most powerful single-window model wouldn’t work for real listening.

The Lingering Issue: Out-of-Distribution VAE

The author candidly acknowledged one remaining issue: the VAE is out of distribution.

EAR-VAE was trained on finished stereo mixes—two channels of final music. But in Stereo2Spatial, it’s now used to encode individual mono channels of a 7.1.4 output. Mono tracks differ significantly in spectrum, energy distribution, and dynamics from stereo masters. Using an encoder trained on “finished dishes” to encode “single ingredients” inevitably hurts reconstruction quality.

An ideal fix would be retraining a VAE specifically for mono signals or one unified multi-channel VAE. But as a one-person side project, retraining VAEs is too heavy in compute and data terms.

This highlights the typical gap between open-source and corporate projects: the idea is solid, but the resources aren’t there to optimize every component. Still, by open-sourcing the design and pipeline, the author provides a baseline for others to build upon.

How It Compares to Industrial Solutions

Let’s be realistic—Stereo2Spatial isn’t going to beat Dolby’s own Atmos masters or Sony’s 360 Reality Audio studio remixes. Those are mixed from multitrack stems, which inherently contain far more information than stereo. Even the best generative model can’t magically recreate all the spatial cues lost in downmixing—it can only make educated guesses.

But compared to other approaches, it’s quite compelling:

• Versus Apple’s algorithmic “Spatial Audio” upmix: Apple’s processing for non-Atmos tracks is essentially HRTF rendering plus room simulation—it widens the stage but doesn’t actually separate and relocate sources. Stereo2Spatial theoretically provides more structured spatial placement.
• Versus LCC or Windows Sonic real-time upmixers: These focus on low latency and low computation, using fixed DSP chains. Generative models like this aim for offline, high-quality transformation—a different category.
• Versus commercial upmix plugins (Penteo, Nugen Halo Upmix): Traditional plugins use frequency-domain decomposition and energy redistribution, which are controllable but not “intelligent.” Generative models may capture subtler spatial patterns beyond handcrafted rules.

In short, Stereo2Spatial is more of an experimental tool for enthusiasts and researchers than a production-ready pipeline for streaming platforms. But it proves the concept works.

A One-Person Project—And the Start of a Direction

This project recalls the early days of AI music: RVC, So-VITS-SVC, Diff-SVC—all started as individual hobby projects that the community later expanded, eventually influencing industry technology. Speech conversion went that way; singing synthesis went that way; AI spatial upmixing could well follow.

A notable trend: diffusion and flow matching are quickly penetrating every audio subfield—speech (Voicebox), music generation (Stable Audio, Suno’s backend), singing voice conversion (Diffusion-SVC)—and now spatial reconstruction, a niche area. As the base infrastructure matures, every audio domain becomes ripe for reinvention through generative models.

Two takeaways for developers:

1. The VAE + Diffusion + Flow Matching combo has become the de facto standard stack for audio generation. Stop overthinking architectures; focus on designing appropriate input-output mappings for your task.
2. Long-sequence stability is a universal challenge for audio diffusion models. Cross-window state transfer mechanisms (memory tokens, KV cache, temporal attention) will be key breakthroughs in the near future.

For creators and listeners, a fun next step is cloning Stereo2Spatial’s repo and testing your favorite songs that lack official spatial versions. Hearing is believing. The author admits the model is still evolving and welcomes community PRs.

And back to that simple motivation—“I want to listen to spatial versions of songs, but they don’t exist, so I’ll train one myself.” The most interesting AI projects often start exactly like that.

References

• Stereo2Spatial discussion thread (Reddit r/MachineLearning) — The original post by the author, detailing the architecture, motivation, and known issues.