NVIDIA Audex Is Here: Audio and Text Unified in One Model, Only 3B of 30B Parameters Activated

NVIDIA’s paper posted on arXiv last month — “Unified Audio Intelligence Without Regressing on Text Intelligence” — now has an official product release: Nemotron-Labs-Audex-30B-A3B. It’s a unified audio-text large model with 30B total parameters and 3B active parameters. NVIDIA also quietly released a smaller 2B version on Hugging Face so the community can start experimenting immediately.

If you’ve followed the multimodal field over the past two years, you already know the problem Audex is trying to solve isn’t new: once you attach audio/speech modalities to a pure text model, the model’s text capabilities often degrade significantly. The industry’s standard approaches are either to bolt an audio adapter onto a text LLM through aggressive fine-tuning, or to retrain an omni model from scratch. The former tends to produce weak audio performance, while the latter usually sacrifices text quality. Audex takes a different path: keep the text backbone untouched, encode audio and map it into the text embedding space, then let quantized audio output tokens and text tokens share the same inference pipeline.

Architecture: Mamba-Transformer Hybrid + MoE

Audex is built on NVIDIA’s in-house Nemotron-Cascade-2-30B-A3B, a 52-layer Mamba-Transformer hybrid architecture containing 128 routable experts, with 6 activated per forward pass. This follows the same lineage as NVIDIA’s recently released Nemotron 3 Nano Omni and represents the company’s current technical direction: Mamba has natural advantages for long sequences, making it especially well-suited for audio modalities that generate dozens or hundreds of tokens per second.

On the audio input side, Audex uses the AF-Whisper encoder to process 16kHz audio. Two MLP adapter layers map the audio features into the text embedding space. The vocabulary was also expanded from the original 131,072 tokens to 205,312 tokens, with the additional 70k+ tokens reserved for discrete audio representations.

On the output side, NVIDIA separates speech and non-speech audio into two different paths:

• Speech output uses X-Codec2 at 50 tokens/second, with single-layer finite scalar quantization (FSQ) and a codebook size of 65,536
• Non-speech audio (sound effects, music, ambient audio, etc.) uses X-Codec at 200 tokens/second with 4-layer flattened residual vector quantization

This split is carefully designed. Speech signals have lower information density and more regular structure, so 50 tokens/second plus single-layer FSQ is sufficient. General audio signals are much more complex and require residual vector quantization to preserve detail. Many models that lump speech and audio together use a one-size-fits-all approach here and end up underperforming on both.

The Core Question: Did Text Capabilities Actually Survive?

This is the central claim Audex tries to prove — and the point most likely to draw skepticism. Here are the benchmark results:

| Benchmark | Audex-30B-A3B | Backbone Model | |---|---|---| | MMLU-Redux | 86.4 | 86.3 | | IMO AnswerBench | 81.1 | 79.3 | | MMLU-Pro | Slight drop | - | | GPQA-Diamond | Slight drop | - |

The conclusion is essentially “performance remained flat, with slight gains on some tasks.” Audex even outperformed the pure text backbone on MMLU-Redux and IMO AnswerBench, which isn’t surprising — expanded vocabularies and multimodal training can themselves act as a form of regularization and produce positive side effects. However, the slight declines on MMLU-Pro and GPQA-Diamond suggest that audio training still introduces minor interference on difficult reasoning-heavy tasks.

That tradeoff is within acceptable bounds. Compared with omni-style models like Qwen3-Omni, Audex takes a more conservative “preserve text, add audio” approach, while Qwen3-Omni focuses on pushing ASR and voice interaction toward Gemini 2.5 Pro-level performance. The two are aiming at different goals: one wants a universal foundation model, while the other wants to specialize in audio interaction.

The Practical Value of 3B Active Parameters

A 30B model with only 3B active parameters means inference costs are close to those of a dense 3B model, while retaining the knowledge capacity of a 30B system. That ratio is critical for audio workloads, where real-time responsiveness and long context windows are standard requirements — you can’t have users finish speaking and then wait three seconds before the model starts responding.

The open-source Audex-2B release follows the same recipe and mainly targets edge and on-device scenarios. NVIDIA also published full usage examples for Transformers, vLLM, SGLang, and Docker Model Runner on Hugging Face. Deploying with vLLM is literally a one-line command:

python3 -m pip install \"vllm[audio]\"
vllm serve nvidia/Nemotron-Labs-Audex-2B

The 2B version supports a broad set of tasks: audio understanding, speech recognition, speech translation, TTS, audio generation, and S2S (speech-to-speech) generation. That coverage is effectively enough to support the entire backend stack of a voice assistant product.

How Does This Relate to NVIDIA’s Nemotron 3 Nano Omni?

A few months ago, NVIDIA released Nemotron 3 Nano Omni 30B-A3B, a unified four-modal model covering video, audio, image, and text. Audex looks like NVIDIA pulled the audio component out of the Omni family and optimized it more deeply — because for pure voice interaction scenarios, video and image modalities become unnecessary overhead, and dedicated audio optimization can achieve better results.

Omni follows the “one model replaces the entire agent pipeline end-to-end” philosophy, while Audex follows the “audio specialization while preserving text” route. Both use MoE architectures and share the same Nemotron-Cascade backbone, making NVIDIA’s broader strategy clear: build a strong text foundation first, then stack modalities on top one by one.

Some Technical Details Worth Discussing

Audio tokenization granularity: speech runs at 50 tokens/second while general audio runs at 200 tokens/second. That 4x difference has a huge impact on training efficiency. If everything used 200 tokens/second, speech training costs would increase fourfold unnecessarily. If everything used 50 tokens/second, general audio quality would collapse. Audex’s dual-codebook design increases engineering complexity, but from a cost-benefit perspective it makes sense.

Vocabulary expansion strategy: the vocabulary grew from 131,072 to 205,312 tokens, with the extra 74,240 tokens allocated to the two audio codecs. There’s a hidden cost here — both the embedding layer and output head grow proportionally. At the 30B scale this overhead is manageable, but at 100B+ scale, vocabulary inflation will create noticeable VRAM pressure.

The natural affinity between Mamba-Transformer architectures and audio: audio sequences are long and exhibit strong local correlations. Mamba’s state-space modeling uses far less memory than pure attention during long-audio inference. NVIDIA’s backbone choice was highly intentional. Combined with MoE sparse activation, this architecture should deliver substantially higher throughput than dense Transformers at the same parameter scale for audio inference workloads.

Who Should Care About This Model

• Teams building voice assistants or customer service systems: Audex’s S2S capabilities plus preserved text intelligence are a strong combination. You need a model that can understand questions, generate accurate responses, and speak naturally.
• Audio content and podcast generation teams: the X-Codec pathway specifically handles non-speech audio, theoretically enabling generation of sound effects and simple music segments.
• Teams working on multilingual speech applications: the paper emphasizes speech translation capabilities. While the language coverage isn’t as extreme as Qwen3-Omni’s 119 languages, NVIDIA’s usual strength in English and major languages should provide reliable quality.

The 30B weights have not yet been fully released on Hugging Face, but the 2B version is already available for download (with 62.2k followers and strong community interest). OpenAI Hub is already tracking integration progress for the Audex series and plans immediate support once the 30B weights are released, enabling direct access through a unified API without manually setting up vLLM and audio codec environments.

A Final Take

The most valuable contribution of the Audex paper is not simply adding audio to an LLM — everyone is doing that already — but rather demonstrating with data that preserving text capabilities is actually achievable. Over the past year, the multimodal space has been full of claims like “we added audio/vision/video and benchmark scores only dropped by a few points.” Audex is one of the few systems that genuinely maintained — and occasionally improved — scores on major text benchmarks.

For developers, this means you may no longer need to choose between “a strong text model” and “a model that can listen and speak.” While there’s still some degradation on hard reasoning tasks like GPQA-Diamond, Audex’s tradeoff is acceptable for the overwhelming majority of product scenarios.

NVIDIA has clearly accelerated its pace in open-source models recently. From Nemotron-Cascade to Nano Omni to Audex, the product line has evolved into a complete modality matrix. At a time when many closed-source AI giants are tightening restrictions around open models, NVIDIA is moving in the opposite direction and openly releasing near-SOTA systems — likely following the same logic as always: open-sourcing the model is fine, because inference will still run on NVIDIA GPUs anyway.

References

• NVIDIA releases the unified Audex model: 30B total parameters, avoiding text capability degradation after multimodal expansion - ITHome: Original reporting on the Audex architecture and benchmark data
• NVIDIA launches Nemotron 3 Nano Omni: 30B-A3B hybrid MoE architecture - ITHome: Background on NVIDIA’s related Omni multimodal model
• nvidia/Nemotron-Labs-Audex-2B - Hugging Face: Model card and usage examples for the open-source Audex-2B release