Noiz AI teams up with HKUST and Tsinghua to release an open-source audio large model: Single GPU, four steps to generate sound in 0.24 seconds
Noiz AI, together with teams from HKUST and Tsinghua University, has open-sourced an audio generation large model. It only requires 4 sampling steps and just 0.24 seconds on a single GPU to generate a piece of high-quality audio. It can also understand fine-grained control instructions at the timestamp level, pushing TTS and sound effect generation to a new benchmark.
Another Field Getting “Pressed Down by Step Count”: Audio
On June 14, Noiz AI, together with teams from HKUST and Tsinghua University, released an audio-generation large model on GitHub focusing on two main features: four-step sampling and single GPU generates 0.24s of audio segment. Plus another phrase that may sound like icing on the cake—but is actually crucial for engineers: it understands timestamps.
With this combination, insiders can immediately get the idea: The familiar “dozens of sampling steps for quality” approach of diffusion models is being squeezed in the audio domain just as in image diffusion acceleration (LCM, SDXL Turbo, etc.), now down to a comparable scale. More importantly, they’ve also fixed controllability—something that TTS / sound-effect generation has been struggling with over the past year.
Let’s Put the Numbers Out First
According to official benchmark data (single A100 GPU, FP16):
- Sampling steps: 4 steps
- End-to-end latency: About 0.24s to generate ~10s of audio
- RTF (Real-Time Factor): Well below 0.05, meaning generation speed is over 20× faster than real-time playback
- Supported tasks: Text-to-Speech (TTS), Text-to-Audio (TTA), mixed generation of voices + environmental sounds
- Controllable dimensions: Timestamp-level event alignment, emotion tags, speaker timbre, speech rate, pauses
Some reference comparisons:
- A year ago, mainstream works like AudioLDM2, Stable Audio generally needed 50–200 DDIM steps to converge stably, with single GPU latency in seconds.
- Even ElevenLabs Turbo, though popular in the latter half of last year, struggled to keep first packet latency under 200ms when generating long emotional sentences.
- Domestic open-source projects like CosyVoice, F5-TTS have already achieved near-commercial quality, but four-step acceleration + timestamp control has not yet been seen.
So the highlight here isn’t “just another open-source TTS,” it’s changing the inference paradigm for diffusion audio models.
How Four Steps Were Achieved: Consistency Distillation + Flow Matching
Looking at the technical report, Noiz’s solution isn’t a single-point optimization—it’s multiple lines combined:
1. Replace Diffusion with Flow Matching
Traditional DDPM/DDIM aims to predict noise at each step, requiring sequential traversal along the SDE/ODE during inference. Flow Matching learns a continuous vector field from noise to data directly, allowing ODE solving with drastically fewer steps. This route has already been validated in image generation (Stable Diffusion 3, Flux) and video generation (Sora lineage), now adapted to audio.
2. Consistency Model Distillation
Flow Matching alone isn’t enough—they stack consistency distillation: letting the model predict the ODE endpoint directly from any starting point. This is why they can compress from dozens of steps to 4—essentially “what the teacher model learned in 50 steps, the student model approximates in one.”
3. Latent-Space Diffusion, Not Waveform Diffusion
The model operates in the latent space after VAE encoding, with frame rate compressed dozens of times, plus a lightweight neural vocoder decoding latents back to 24kHz waveform. This architecture borrows from Stable Audio but retrains both VAE and vocoder, adapting for mixed voice and sound-effect scenes.
4. DiT Backbone + Timestamp Cross-Attention
The backbone is Diffusion Transformer (DiT), more stable than U-Net on long sequences. Timestamp injection encodes each event’s (start, end, label) into conditional tokens fed into cross-attention—this step is key for “understanding timestamps.”
What “Understands Timestamps” Really Means
This is worth elaborating, as it’s the most underestimated but most practical capability for developers.
Previously, if you asked TTS to read lines, you could at most control emotion or speed. If you asked TTA to generate “rain, thunder, distant dog barking,” the model would likely layer all sounds together—when each sound happens is random.
Noiz’s model lets you write prompts like:
[0.0s-2.0s] Rain, medium intensity
[1.5s-2.2s] Distant thunder, low frequency
[3.0s-3.5s] Dog bark, two times
[4.0s-7.0s] Rain gradually decreases
The generated 10-second audio matches event start and end positions closely—official report says alignment error is around 100ms.
Where is this useful?
- Video dubbing: Editors want effects precisely at frame changes, previously requiring manual sync
- Game audio: Triggered sound effects need millisecond-level response, traditional generative models unusable
- Audiobooks / podcasts: Timing transitions between narration, dialogue, and background music
- Multimodal agents: If an LLM wants a virtual character to “wave while saying ‘hello,’” timestamps are the only way audio and actions sync
In short, timestamps are the essential path to making audio models production tools, not toys. Previously, only a few closed-source commercial models (e.g., ElevenLabs’ Sound Effects API, Suno’s stems mode) managed parts of this—this is the first in the open-source world.
Training Data and Model Scale
Disclosed details from the report:
- Training data: ~200,000 hours of audio, covering speech in Chinese, English, Japanese, Korean, plus open-source sound libraries like BBC Sound Effects, Freesound, and proprietary licensed data
- Model parameters: DiT backbone ~1.3B, VAE + vocoder ~200M combined
- Training compute: Not specified, but scale suggests hundreds of H100-class GPUs
Compared to purely academic projects, 200k hours is “quasi-industrial” scale—explaining why quality reaches commercial usability.
Where It Stands Compared to Peers
Lining up mainstream open-source / semi-open audio-generation models:
| Model | Task | Sampling Steps | Timestamp Control | Open Source Level | |---|---|---|---|---| | Stable Audio Open | Music / SFX | ~100 | No | Fully open source | | AudioLDM2 | SFX / Speech | ~200 | No | Fully open source | | CosyVoice 2 | TTS | ~25 | No | Fully open source | | F5-TTS | TTS | ~32 | No | Fully open source | | ElevenLabs Turbo | TTS | Not disclosed | Partial | Closed API | | Suno v4 | Music | Not disclosed | Partial | Closed | | Noiz Audio | TTS + SFX | 4 | Yes | Fully open source |
Noiz’s position is clear: the only fully open-source model combining TTS and SFX, with single-digit sampling steps and timestamp control. Of course, the table can’t show everything—dimensions like audio MOS, emotional expressiveness, special languages need broader community blind tests. But from architecture and initial demos, it’s at least competitive.
A Bit of Cold Water
We’ve covered the strengths; here are a few caution points:
- Weak singing synthesis (SVS). This version focuses on speech and sound effects—not singing. For music, look at Suno, Udio, or DiffSinger lineage.
- Generalization to minor languages. The 200k hours mainly cover Chinese, English, Japanese, Korean. Spanish, Arabic can be generated, but quality and naturalness drop.
- Zero-shot voice cloning is stable with 3s reference audio; shorter samples cause timbre drift—less polished than ElevenLabs.
- License. Code under Apache 2.0, but some pretrained weights are non-commercial—commercial deployment requires checking. Stable Diffusion and Llama have tripped on this before—developers must be aware.
What This Means for Developers
The “step-count revolution” in audio models has two direct impacts:
First, real-time audio generation enters consumer-grade hardware range for the first time. 0.24s on A100 means 0.5–1s on 4090 GPU—local deployment for real-time conversational agents or game NPCs is no longer just a PPT concept.
Second, audio is no longer the bottleneck for multimodal agents. Previously, TTS’s first packet latency (TTFB) often matched or exceeded LLM first token times. Four-step sampling brings TTS first packet into tens of milliseconds, shifting the bottleneck entirely back to LLM—making “interruptible conversation” and “real-time low-latency voice interaction” viable for new entrants.
For integrators, mainstream large-model aggregation platforms (including OpenAI Hub) will gradually adopt such open-source audio models into unified APIs—developers can call GPT-5, Claude 4, Gemini 3, and these new audio models in the same project with one key, avoiding GPU deployment headaches. Truthfully, deployment barriers are low—you could run local inference entirely with a vLLM-style service.
In Closing
From late 2025 to early 2026, the open-source world is racing hard in video and audio. Video has Wan, Open-Sora, CogVideoX pushing prices down; audio has CosyVoice, F5-TTS, and now Noiz Audio—each pushing “commercially usable, controllable, low latency” closer together.
Noiz’s work isn’t based on radically new concepts—they use Flow Matching, Consistency Distillation, DiT, all proven in the past year. But putting these components together in the right order, and engineering specifically for timestamp conditioning, produces something more practical than peers.
This kind of “combinatorial innovation” is exactly what open-source excels at—closed-source firms may run faster, but open source spreads tools to every developer. In this sense, today’s developer who can clone from GitHub and run a four-step generation demo is closer to real production environments than last year’s user who clicked “trial” on some commercial platform.
