NVIDIA Open-Sources TwoTower Language Model: Diffusion Architecture Accelerates LLM Token Generation

Yesterday (July 2), NVIDIA officially open-sourced Nemotron-Labs-TwoTower, a discrete diffusion language model built on a pre-trained autoregressive backbone network. At a time when every company is competing on model capabilities, NVIDIA has chosen a different entry point: instead of chasing higher benchmark scores, it directly targets token generation speed, the core bottleneck of large model inference.

Empirical data shows the effect is indeed significant. In a 2×H100 GPU environment, TwoTower retains 98.7% of the baseline model’s quality, while achieving 2.42× higher throughput. For teams that need to mass-produce synthetic data or perform large-scale text generation, this level of improvement is very compelling — doubling the output with the same hardware cost.

Dual-Tower Architecture: Separating Context Understanding and Denoising Generation

The key innovation of TwoTower lies in its architecture. Traditional autoregressive models must decode text token by token in sequence, re-computing the full context at each step. When dealing with long texts or large-scale batch generation, this serial decoding becomes a clear performance bottleneck.

NVIDIA’s solution is to split the task into two independent neural network “towers”:

• Context Tower: kept frozen, focused on processing the input prompt and maintaining the autoregressive context. This tower is essentially the original 30B autoregressive model with fixed parameters, preserving its language understanding capability.

• Denoiser Tower: specifically trained for parallel denoising of noisy token blocks. This tower is also 30B in scale but uses a Mixture of Experts (MoE) structure—only 3B parameters are activated at a time out of 128 routeable experts.

The two towers collaborate through layer-by-layer cross-attention. The context tower provides stable semantic representations, and the denoiser tower generates multiple token blocks in parallel instead of one by one. This design greatly improves parallelism while maintaining quality.

60B Total Parameters, but Only 3B Activated at a Time

TwoTower has a total of 60B parameters—two 30B towers combined. However, because of the MoE architecture, only 3B parameters per tower are activated during inference. This means that while the model’s capacity is large, the actual computation cost is not excessive.

This design strikes a balance between memory and compute efficiency. On one hand, the total parameter size ensures expressive power; on the other, the 3B active parameters keep inference speed within a reasonable range. According to NVIDIA, this model can even run on consumer GPUs like the RTX 4090—though demanding on VRAM, it doesn’t strictly require data-center hardware.

The model supports three decoding modes for high flexibility:

1. Diffusion Mode: fully parallel blockwise denoising—best performance, slightly lower quality.
2. Simulated AR Mode: dual-tower collaboration while maintaining autoregressive generation—nearly baseline quality.
3. Standard AR Mode: purely autoregressive decoding—used as the baseline reference.

Developers can choose the balance between quality and speed according to their specific tasks. The default configuration, “confidence-based unmasking,” uses threshold γ = 0.8 and block size S = 16, delivering 98.7% quality retention and 2.42× throughput improvement. Lowering the confidence threshold further boosts speed at a cost to quality.

Performance: Close to Baseline for Most Tasks, Slight Drop in Code and Math

Benchmark results show that TwoTower’s performance is close to its autoregressive baseline across most general tasks. Accuracy on benchmarks such as MMLU, ARC-Challenge, WinoGrande, and RACE remains nearly identical or slightly improved.

However, in code generation (HumanEval) and mathematical reasoning (MATH-500) tasks, there is a small drop in performance. HumanEval falls from 79.27% to 75.58%, and MATH-500 from 84.40% to 80.60%. While modest, this decline may matter for tasks requiring highly precise reasoning, where traditional autoregressive models remain preferable.

| Task | AR Baseline | TwoTower Diffusion | |------|-------------|-------------------| | MMLU (5-shot) | 78.56 | 78.24 | | MMLU-Pro (5-shot CoT) | 62.59 | 60.93 | | ARC-Challenge (25-shot) | 91.72 | 92.66 | | WinoGrande (5-shot) | 76.09 | 76.09 | | RACE (0-shot) | 88.90 | 88.90 | | HumanEval (0-shot) | 79.27 | 75.58 | | MBPP-Sanitized (3-shot) | 74.71 | 74.28 | | GSM8K (8-shot) | 92.49 | 90.14 | | MATH-500 (4-shot) | 84.40 | 80.60 | | Quality Retention | 100% | 98.7% | | Generation Throughput | 1.0× | 2.42× |

This outcome aligns with expectations. Diffusion models sample from probability distributions, making them less stable for tasks requiring strict logical reasoning. But for most text-generation use cases—content creation, data synthesis, dialogue generation—the slight quality trade-off is worth the substantial speed gains.

Training Strategy: Only Train Denoiser Tower, Keep Context Tower Frozen

The training approach for TwoTower is also noteworthy. It is built on NVIDIA’s Nemotron-3-Nano-30B-A3B backbone, which was itself trained on around 2.1 trillion tokens.

During TwoTower’s training, the context tower (the original autoregressive model) remains frozen—only the denoiser tower is trained. This design brings several advantages:

1. Lower training cost: only the new denoiser tower is trained, not all 60B parameters.
2. Preserved capability: the context tower retains the baseline model’s language understanding, avoiding catastrophic forgetting.
3. High flexibility: theoretically, any autoregressive model can serve as the context tower, enabling quick adaptation to the dual-tower architecture.

NVIDIA notes that the context tower is “optionally trainable,” but is kept frozen in the open-source release—likely to simplify training and facilitate community experimentation with different base models.

The denoiser tower is trained on data analogous to the baseline—subsets extracted from Nemotron’s mixed pre-training data. Details of data ratios and filtering were not disclosed, but the results suggest effective coordination between the towers.

License and Hardware Requirements

TwoTower’s model weights are released on Hugging Face under the NVIDIA Nemotron Open Model License, which allows commercial use but includes some restrictions—such as prohibiting competitive model training and requiring attribution in derivative works. It is stricter than Apache 2.0 but acceptable for most practical use cases.

In terms of hardware, while only 3B parameters are active per tower during inference, the entire 60B weights still need to load into memory. According to community feedback, at least 80GB VRAM (A100 or H100) is needed for BF16 precision. With FP8 or INT4 quantization, the memory requirement can drop to around 40GB, making RTX 4090 (24GB) feasible with CPU offloading, albeit slower.

NVIDIA provides a simple usage example:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "nvidia/Nemotron-TwoTower-30B-A3B-Base-BF16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Assign the two towers to different GPUs
model.place_towers_on_devices("cuda:0", "cuda:1")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The API remains largely compatible with standard Hugging Face Transformers, with one addition: place_towers_on_devices() for manual GPU assignments. If VRAM permits, both towers can run on a single GPU.

Diffusion Models: A New Frontier for LLMs

TwoTower isn’t the first to use diffusion for language generation, but it’s among the most production-ready open-source implementations.

Diffusion models have proven their power in image generation—Stable Diffusion, DALL·E, etc.—thanks to their ability to generate outputs in parallel without adversarial training (like GANs) or serialized generation (like autoregressive models).

However, diffusion models for text generation have not yet been widely commercialized, primarily because denoising discrete tokens is much harder than denoising continuous pixel data, and text demands logical coherence. Early diffusion LMs (e.g., Diffusion-LM) lagged in both quality and efficiency compared with autoregressive models.

NVIDIA’s dual-tower setup offers a promising middle ground: it retains contextual coherence using a frozen autoregressive model while accelerating generation through the trained diffusion head—combining the best of both worlds.

Google also recently released a diffusion-based language model—a 26B MoE architecture—claiming 4× faster token generation on GPU. But since that model was trained entirely from scratch without an autoregressive backbone, it suffered more noticeable quality drops in some tasks. Compared with that, TwoTower’s design achieves a better quality–speed balance.

Use Cases: Data Synthesis and Batch Generation

TwoTower is best suited for large-scale, high-throughput text generation tasks such as:

• Synthetic training data: generate labeled data for downstream models—faster output means lower data generation cost.
• Content creation assistance: mass-produce outlines, product descriptions, or marketing copy for later human curation.
• Dialogue systems: generate multiple candidate replies in parallel, then select the best one using ranking models.
• Code annotation generation: automatically add docstrings to large codebases where perfect accuracy isn’t required.

For high-precision reasoning—competition-grade math, complex code synthesis, or legal document analysis—traditional autoregressive models are still preferable. But for most applications, the 98.7% quality retention with 2.42× speed delivers a far more attractive cost–performance ratio.

Another highlight is its low training cost. If you already have a trained autoregressive model, you can simply add a denoiser tower on top and train only that part — a cost-effective optimization approach for teams aiming to accelerate inference.

OpenAI Hub Now Supports Mainstream Models

Deploying or fine-tuning a 60B-parameter model remains challenging for many developers. If you just want to test how diffusion architectures perform for your use case, API aggregation platforms like OpenAI Hub offer a unified interface. With a single API key, you can access major models—GPT, Claude, Gemini, DeepSeek, etc.—through an OpenAI-compatible API, with direct domestic connectivity. Though TwoTower isn’t available as a hosted API yet, that’s likely to change as diffusion-based language models mature.

Final Thoughts

By open-sourcing TwoTower, NVIDIA introduces a fresh perspective on LLM inference optimization—not merely scaling hardware, but innovating at the architectural level. The dual-tower design redefines the balance between quality and speed.

With 98.7% quality retention and a 2.42× throughput improvement, the model offers exceptional efficiency for cost-sensitive applications. While code and math reasoning still have room for improvement, for general text generation, it’s a trade-off well worth making.

More importantly, open-sourcing unlocks community innovation. We can expect future work to scale this design up, or refine the denoiser tower even further. The exploration of diffusion models in language generation has just begun—TwoTower is merely the first step.

---

References

• NVIDIA Open-Sources TwoTower AI Model: 98.7% Quality Retention, 2.42× Token Speedup – ITHome (Chinese) — Chinese coverage of NVIDIA’s official release, including performance benchmarks and architectural details.