Google Magenta RealTime 2: Local Music AI with Latency Reduced to 200ms

On June 4, the Google Magenta team released Magenta RealTime 2 (MRT2), a music generation model that can run in real time on local devices. The biggest breakthrough is latency: from the previous generation's 3-second control latency compressed to 200 milliseconds, reduced to 1/15 of the original. What does this number mean? For musicians, 200ms is already close to the feedback speed of real instruments—you play a note, and the AI can almost respond synchronously.

This is not just a paper parameter optimization. The core question of real-time music generation has always been: “Can it keep up with a human's playing tempo?” The previous generation Magenta RealTime required a TPU or high-performance GPU and generated audio in 2-second blocks, making its latency more like that of a “half-beat-late accompanist” rather than an improvisational collaborator. MRT2 switches to frame-by-frame generation—40ms per frame—fundamentally changing the interaction experience.

Two Versions, Optimized for Apple Silicon

MRT2 offers two model sizes: the high-quality mrt2_base with 2.4 billion parameters, and the high-speed mrt2_small with only 230 million parameters. Google explicitly stated that both versions are optimized for Apple Silicon chips, and mrt2_small can run in real time on Macs with M1 and later chips.

This is an interesting choice. Macs have an extremely high market share in the music production community. Mainstream DAWs like Logic Pro, Ableton Live, and FL Studio prioritize macOS as their primary platform. By focusing optimization on Apple Silicon instead of CUDA GPUs, Google is showing they know exactly where their target users are.

The difference becomes clear when compared to the previous generation. Magenta RealTime formerly required a TPU or at least a mid-to-high-end NVIDIA GPU to run, which was too high a barrier for independent musicians. Now, an M1 MacBook Air can run mrt2_small—a qualitative shift. Of course, if you want better sound quality, the 2.4B parameter mrt2_base is superior, but the 230M parameter version is already adequate for real-time performance and rapid prototyping.

Evolution of Control Methods

MRT2 is much more complete in control than its predecessor. It supports text prompts, audio style references, MIDI note input, and drum track toggling. These input methods can be combined: for example, you can describe “80s synth pop” in text, then play melodies on a MIDI keyboard, and the model will adjust its content generation in real time according to your notes.

A finer detail is the Auto-Strum mode. When enabled, the model automatically decides the strumming or note onset timing, making the generated music sound more natural. When disabled, you can specify the exact start time of each note—ideal for composition scenarios needing strict rhythm control. For drums, the system supports switching output with or without drum tracks, which is practical in multitrack arrangements—you might only want melody and harmony, leaving the drum section for other tracks.

This finer control is backed by changes in model architecture. MRT2 is essentially still a Codec Language Model, processing 48kHz stereo audio via the SpectroStream audio codec and generating audio tokens at a 25Hz frame rate. Each frame is 40ms, meaning the model must make 25 decisions per second—requiring extremely high inference speed. That Google can run this on Apple Silicon shows they invested significant work in model quantization and operator optimization.

Jam App and DAW Plugin

Google simultaneously released two supporting tools. The first is Jam, a free standalone app. You can type the desired music style into the text box, e.g., “jazz with piano and drums.” Once generation begins, you can adjust pitch via the virtual keyboard below. Input devices aren’t limited to mouse clicks—MacBook’s physical keyboard and MIDI keyboards are also supported.

Jam is positioned as a rapid experimentation tool. If you want to test a music idea or need inspiration for background music, open Jam, input a description, and you're ready to start. It’s not a professional-grade production tool, but it's sufficient for brainstorming and concept validation.

The second tool is the MRT2 plugin—more important for many users. It allows you to call Magenta RealTime 2 directly within your usual DAW. For musicians with established workflows, this plugin offers a straightforward integration path: no need to switch to a new app or alter existing project structures—just insert MRT2 as an instrument or effect into a track.

This is a pragmatic design. The music production tool ecosystem is mature, and professional users won’t abandon DAWs they've used for years just because an AI model exists. By choosing to integrate as a plugin rather than replace DAWs with a standalone app, Google demonstrates an understanding of actual market needs.

Open Source Strategy and Competitive Landscape

MRT2 is released as open source under the Apache 2.0 license. Model weights are hosted on Hugging Face, and code is on GitHub. In the model card, Google explicitly states: “Google claims no rights over outputs generated by your use of MRT2; users bear full responsibility for the outputs and their subsequent use.”

This open source strategy is notable. MRT2 is the open version of Lyria RealTime, which powers the closed-source Music FX DJ and Google AI Studio’s real-time music APIs. Google maintains both open and closed lines: open for research and community experimentation, closed for commercial products. This dual-track approach is increasingly common in AI—OpenAI uses GPT for commercial models, Meta open-sources Llama; Google DeepMind uses Gemini for products, and the Magenta team releases open-source models.

Compared to other music generation tools, MRT2’s advantages are real-time capability and local execution. Services like Suno and Udio may take tens of seconds to generate a complete song and must be online to call APIs. Stable Audio and AudioCraft, while open-source and capable of local execution, target one-off generation and aren’t designed for real-time interaction. MRT2 fills the niche of “local + real-time + controllable.”

Of course, MRT2 has clear limitations: its sound quality and musicality lag behind models dedicated to high-quality generation. With parameter counts from 230M to 2.4B, it’s modest in music generation scope. It’s better suited as part of a creative toolkit rather than fully replacing human composers. Google’s documentation notes it inherits some limitations from its predecessor, such as limited handling of complex harmonic structures and potential lack of long-term structural coherence.

Technical Details: From Block Generation to Frame-by-Frame Generation

The latency optimization core is a change in generation strategy. The previous generation generated 2-second audio blocks, each based not only on user prompts but also on the preceding 10 seconds of audio context to maintain musical continuity. However, generating a 2-second block itself took time, plus processing between blocks, making overall latency hard to reduce.

MRT2 switched to frame-by-frame generation: 40ms per frame, at 25Hz. This means the model must complete an inference within 40ms, including encoding, Transformer computation, and decoding. To achieve this speed on Apple Silicon, quantization and operator optimization were necessary. GitHub code shows Google used Core ML, Apple’s machine learning framework optimized for Apple Silicon, leveraging the Neural Engine and GPU.

The SpectroStream audio codec is also key—it compresses 48kHz stereo audio into discrete tokens on which the Transformer generates. This codec-based architecture is mainstream in current audio generation: Meta’s AudioCraft and Microsoft’s VALL-E adopt similar designs. Differences lie in codec quality and compression ratio, directly affecting fidelity and inference speed.

Impact on Music Production Workflow

Real-time music generation models like MRT2 may not “replace composers” but could “change the starting point of creation.” Traditional workflows are: idea → melody/harmony writing → arrangement → mixing. With AI tools involved, it might become: describe idea → AI generates initial version → human refines → mixing.

The impact varies by creator type. For professional musicians, MRT2 may serve as a rapid prototyping tool to quickly validate harmonic progressions or arrangement ideas. For game developers or video producers, MRT2 offers a low-cost scoring solution—perhaps with lower sound quality than hiring a composer, but sufficient for budget-limited indie projects.

More intriguing is live performance potential. Real-time generation means every show can be unique, with musicians adjusting AI output according to audience reactions. This human-AI collaborative performance could find a market in electronic and experimental music circles. However, performers would need strong understanding and control of AI behavior to avoid mishaps.

The Next Step for the Open Source Ecosystem

Magenta has been a key open-source force in music AI since starting in 2016. Early tools like MusicVAE, Music Transformer, and Magenta Studio plugins provided usable tools and research baselines. MRT2’s release marks Magenta’s return after a quiet period; Google noted in their announcement: “We are excited that Magenta RT marks the return of Magenta’s open-source releases.”

From a technical perspective, real-time generation is a clear direction. AI music generation has moved beyond “can it generate?” to “how can it generate more controllably and usefully?” Real-time interaction is an important dimension of controllability—it turns generation from a black box into a tunable loop: you input → AI outputs → you adjust → AI outputs again. This rapid iteration could impact creative workflows more than merely improving sound quality.

Local execution is another trend to watch. Cloud APIs face latency, cost, and privacy issues. Music production often involves unreleased works, and many creators are unwilling to upload raw audio to third-party servers. Local models solve this concern and lower costs—no per-call API fee, with hardware investment being one-off. Apple Silicon’s performance gains make such local deployment feasible, and more AI tools may choose this route in future.

Limitations and Room for Improvement

Despite MRT2’s breakthrough in latency and local execution, it’s still a research preview with clear limitations. First is sound quality: the 230M parameter model’s audio lacks detail and expressiveness compared to high-quality models like Suno or Udio. Second is musicality: AI-generated music often lacks long-term structure and emotional variation, sounding somewhat flat.

Control precision is another issue. While MRT2 supports MIDI input and text prompts, control is relatively coarse. You can specify “jazz style” or “fast tempo,” but precise control over chord inversions or nuanced timbre is difficult—insufficient for professional production needs.

Instrument type support is also limited. From Jam's interface and documentation, MRT2 mainly targets common pop music instruments (piano, guitar, bass, drums, synthesizer). Support for orchestral, traditional, or experimental sounds may be lacking, related to training data coverage and model capacity.

Obvious improvement areas include: larger models, better audio codecs, finer control interfaces, and more diverse instrument support. But these require trade-offs: larger models slow inference, possibly preventing real-time on M1; complex controls may raise learning costs and deter casual users. Finding balance among these dimensions is the next challenge for the Magenta team and community.

Conclusion

MRT2’s release timing is interesting. It’s June 2026, more than three years since ChatGPT triggered generative AI’s boom. AI music tools have moved from “novel toys” to “usable tools.” Suno and Udio prove AI can generate full, decent-sounding songs; Stable Audio and AudioCraft prove open-source models can run locally; now MRT2 proves real-time interaction on consumer-grade hardware is feasible.

These three directions—high-quality generation, local execution, real-time interaction—may converge in coming years. Ideally: a locally-running, near-professional-sound-quality, low-latency music AI model capable of real-time performance. We are not there yet, but MRT2 shows at least one piece of that puzzle is achievable.

For developers, MRT2’s open-source release offers a starting point to fork, modify, and experiment. You can build your own instrument app, refine control interfaces, or integrate it into other creative tools. For musicians, it may not yet be time to fully depend on AI, but it’s certainly another worthwhile tool to try.

Finally, if you’re an Apple Silicon Mac user, you can head to GitHub to download the code and run mrt2_small to experience 200ms latency real-time generation. Such technology needs to be tried firsthand to truly understand its potential and limitations.