Orthrus: Frozen Backbone + Diffusion Attention, 7.8× Faster Inference

A Vietnamese research team recently published a paper proposing a new architecture called Orthrus. The core idea is to insert a trainable diffusion attention module into a frozen autoregressive Transformer, enabling the model to generate 32 tokens in parallel—boosting inference speed by 7.8×—while maintaining identical accuracy to the base model. The key highlight: extremely low training cost—only 16% of parameters are trained, using less than 1 billion tokens, and can be completed in 24 hours on eight H100 GPUs.

How Diffusion Attention Fits into a Transformer

Orthrus’s design is straightforward: insert a Diffusion Head into every Transformer layer, sharing the same KV cache with the original Autoregressive (AR) Head. The diffusion head predicts the next K = 32 tokens in parallel, while the AR head performs a second pass to verify and accept the longest matching prefix. Mathematically, the resulting output distribution can be proven identical to that of the base model.

The key lies in the Dual‑View design: the diffusion head follows the denoising diffusion probabilistic model (DDPM) idea, treating token generation as a progressive denoising process from noise to clean text; the AR head remains a classic autoregressive generator, producing one token at a time. Since both heads share the same KV cache, memory overhead is O(1)—measured to add only about 4.5 MiB—far less than speculative decoding, which needs two separate caches.

Advantages Over Existing Methods

Compared with Diffusion Language Models

There have been several attempts at diffusion language models—such as Dream, Fast‑dLLM‑v2, SDAR, Mercury, and Gemini Diffusion. Their common problem: modifying base‑model weights, which degrades accuracy—Fast‑dLLM‑v2, for example, drops 11 points on MATH‑500.

Orthrus, instead, keeps all base‑model weights frozen and trains only the newly added diffusion‑attention module. Thus, accuracy remains completely unaffected—the performance on MATH‑500 is identical to Qwen3‑8B baseline. And since just 16% of parameters are trained with under 1 billion tokens, the training cost is extremely low—converging in 24 hours on eight H200 GPUs.

Compared with Speculative Decoding

Speculative decoding—represented by EAGLE‑3 and DFlash—is another major acceleration approach that uses a small drafter model to propose candidate tokens for the larger verifier model to check. Its issues:

1. Requires an extra drafter model: must either train one from scratch or reuse an existing small model—maintaining two models in total.
2. KV cache cost doubles: drafter and verifier each need their own KV cache.
3. TTFT (Time‑to‑First‑Token) penalty: initialization and synchronization of the drafter model delay the first‑token generation.

Orthrus avoids all of this. It needs no external drafter; diffusion and AR heads share the same KV cache (constant O(1) overhead). And since the diffusion head is embedded directly in the main model, there’s no TTFT penalty—the first‑token latency matches the base model exactly.

Experimental Results: Speed and Accuracy

On the MATH‑500 dataset using Qwen3‑8B as the base model, results are impressive:

• TPF (Tokens Per Forward) up 7.8×: Traditional autoregressive decoding generates only one token per forward pass; Orthrus generates 32 in parallel, achieving TPF = 7.8 after accounting for acceptance rate.
• End‑to‑end speed up ≈ 6×: Wall‑clock speed gains remain about 6× after verification overhead.
• Identical accuracy: Accuracy on MATH‑500 matches the base Qwen3‑8B exactly, with zero loss.

The acceptance rate—fraction of parallel tokens accepted—is a key metric. Orthrus averages 24% (about 7–8 accepted out of 32). This may sound modest, but considering it trains only 16% of parameters with under 1 billion tokens, it’s a strong result.

Training Cost: 24 Hours on 8 × H200

Training cost is one of Orthrus’s biggest advantages:

• Trained parameters: only 16% (the diffusion‑attention module), base weights frozen.
• Data: under 1 billion tokens—far less than training a new LM.
• Compute: 8× H200 GPUs for 24 hours.

This is feasible for most research teams or companies. By contrast, training an 8 B‑parameter LM from scratch typically requires thousands of GPUs for weeks or months and trillions of tokens.

Training mainly teaches the diffusion head to predict the next K tokens, minimizing cross‑entropy loss between its output and the true sequence. Since the base model is frozen, the diffusion head learns to adapt to the base model’s representation space rather than forcing the base to adapt to it.

Technical Detail: Dual‑View Diffusion Guarantee

Orthrus’s theoretical foundation is Dual‑View Diffusion, proving that the joint output distribution from diffusion + AR heads is mathematically equivalent to the base model’s distribution.

Specifically, the diffusion head generates candidate tokens sampled from ( p(x_{t+1:t+K} \mid x_{1:t}) ). The AR head then evaluates each token’s conditional probability ( p(x_{t+i} \mid x_{1:t+i-1}) ) in the base model and accepts the longest prefix.

The paper proves that when the diffusion head minimizes KL divergence with the base distribution, ( p(x_{t+1:t+K} \mid x_{1:t}) ) converges to the true base‑model distribution. Thus, Orthrus preserves generative behavior with zero accuracy loss.

This mathematical guarantee is Orthrus’s main edge over other diffusion LMs, which modify base weights and shift output distributions. Freezing the base model fully avoids this issue.

KV Cache Overhead: Constant O(1)

The KV cache is the main memory bottleneck in Transformer inference. Standard autoregressive generation stores key/value pairs per token, with memory scaling linearly with sequence length. Speculative decoding doubles cost by maintaining separate caches for drafter and verifier.

Orthrus cleverly avoids this: diffusion and AR heads share one KV cache, so cache size never increases with the number of parallel tokens. Empirically, Orthrus adds only ≈ 4.5 MiB of fixed overhead—an O(1) constant independent of sequence length or K.

This is achieved because while the diffusion head generates candidates using the current KV cache, the AR head verifies and incrementally updates that same cache—so only one copy is ever needed.

Use Cases: Long‑Form Generation and Reasoning

Orthrus’s speed‑up varies by task. Evaluations on MATH‑500—a reasoning benchmark requiring long multi‑step outputs—show the largest gains, with TPF up 7.8×.

Other tasks differ:

• Long‑form generation (e.g., writing, code generation): large token counts maximize parallel benefit—expected gains close to MATH‑500.
• Short‑text generation (e.g., chat, QA): fewer tokens mean less benefit.
• Highly stochastic generation (e.g., creative writing): lower acceptance rates reduce gains.

Overall, Orthrus suits tasks producing long, relatively deterministic sequences; it speeds inference dramatically without accuracy loss.

Open‑Source Code and Reproduction

The authors have open‑sourced Orthrus on GitHub: https://github.com/chiennv2000/orthrus.
It’s implemented in PyTorch, compatible with Hugging Face Transformers, allowing any pretrained Transformer to load the diffusion‑attention module.

The repository includes full training/inference scripts and MATH‑500 results. To try Orthrus on your own model:

1. Load a pretrained Transformer (e.g., Qwen3‑8B).
2. Insert diffusion‑attention modules in every layer.
3. Freeze base‑model weights—train only the diffusion modules.
4. Fine‑tune on target‑task data.
5. During inference, use dual‑view generation: diffusion head generates in parallel, AR head verifies and accepts the longest prefix.

The README notes that Orthrus’s training is not sensitive to hyperparameters—default settings work well for most tasks. For tasks differing greatly from MATH‑500, you may tweak learning rate or step count.

Future Directions: Larger K and Higher Acceptance

The paper uses K = 32—a balance between speed and acceptance. Larger K means more parallel tokens but potentially lower accuracy. Future work may explore:

1. Dynamic K: adjust K based on generation certainty—use higher K for deterministic tasks like code completion, lower for creative tasks.
2. Higher acceptance: improve training or enlarge diffusion head to raise acceptance above 24%.
3. Multi‑layer parallelization: different K per layer—small in early layers, large in deeper ones—for better trade‑off.
4. Scaling to larger models: test Orthrus on 70B or 405B models.

Industry Impact

Orthrus is a significant breakthrough for accelerating LLM inference. It demonstrates that you can achieve major speed‑ups without modifying base weights or adding an external drafter model.

For companies already deploying large LLMs, this is appealing: the base weights remain untouched, so Orthrus can be layered onto existing models with minimal retraining cost—just 24 hours on 8× H200 GPUs.

Another standout is Orthrus’s memory efficiency. Since the KV cache overhead is O(1), it’s ideal for long‑context scenarios where memory is critical.

Of course, Orthrus isn’t a silver bullet—the speed‑up depends on acceptance rate, which in turn depends on task determinism. For highly random tasks, gains may shrink. But for reasoning, code, or long‑form content generation, Orthrus already shows strong competitiveness.

---

References

• Orthrus paper – arXiv preprint detailing dual‑view diffusion theory and experiments
• Orthrus GitHub Repository – open‑source code and reproduction scripts
• Reddit Discussion – author’s post on r/MachineLearning