Tencent Has Turned Speculative Decoding into a Complete Framework

Tencent Hunyuan Fills In the Hardest Missing Link in Productionizing Speculative Decoding

On July 29, the Tencent Hunyuan team officially open-sourced AngelSpec, a speculative decoding framework covering everything from drafter training and architecture design to production deployment. Also released were the MTP and DFly drafter weights and training code for Hy3-A21B.

What makes this release noteworthy is not that Tencent has published yet another inference acceleration algorithm, but that it attempts to turn speculative decoding from a standalone technique described in research papers into a complete engineering pipeline that can actually be integrated into production systems.

Over the past two years, speculative decoding has become nearly indispensable for optimizing large-model inference. The principle is straightforward, but implementation is difficult: how to select and train the drafter, how the target model should validate its outputs, how the serving framework should schedule execution, and whether the gains can be sustained under batching and concurrency. If any one of these elements is mishandled, the theoretical speedup can be negated in production.

AngelSpec is aimed squarely at this engineering gap.

Speculative Decoding Essentially Lets a “Small Model Guess First”

A conventional autoregressive model generates one token at a time. At each generation step, the target model must perform a complete forward pass before proceeding to the next step. As models grow larger, this serial process directly increases time to first token and per-token latency, while also limiting the number of requests that a single GPU or server can handle.

The idea behind speculative decoding is to use a less computationally expensive drafter, or draft model, to predict several candidate tokens in succession. The larger target model then checks those candidates in a single pass. If the candidates are accepted, multiple tokens can be advanced during one verification step, eliminating the need to invoke the full target model at every step.

It can be understood as follows:

• The drafter rapidly produces a draft;
• The target model reviews the draft in batches;
• The more accurately the drafter predicts, the more tokens are accepted at once;
• If the draft is too inaccurate, the costs of verification and rollback will consume the acceleration gains.

There is an often-overlooked prerequisite here: the value of speculative decoding lies not only in making the model “generate faster,” but also in preserving the target model’s original output distribution as closely as possible. Rigorously designed speculative sampling can mathematically maintain sampling results consistent with those of the target model, rather than simply replacing the large model with a smaller one.

It is therefore not quite the same as quantization or pruning. Quantization primarily reduces the cost of each computation and lowers memory usage, while speculative decoding reduces the number of expensive steps required for serial decoding without directly altering the target model’s capabilities. The two types of optimization can even be combined: the target model can be quantized, with a drafter placed in front of it.

The Real Challenge Is Not the “Guessing,” but the Entire Pipeline

Achieving a speedup with speculative decoding in an experiment is not difficult. The challenge is making it consistently effective under production workloads.

A deployable solution must address at least four problems simultaneously.

1. The Drafter Must Be Both Fast Enough and Accurate Enough

If the drafter is too small, generating candidates may be inexpensive, but the acceptance rate may be too low. If the drafter is too large, its candidates may be more accurate, but the additional computation can offset the time saved.

This problem cannot be solved by simply choosing a “smaller version” of the model. Differences in vocabulary, hidden states, output patterns, and task distributions between the drafter and target model can all affect the acceptance rate. Code completion, mathematical reasoning, long-form continuation, and open-ended dialogue will not share the same optimal configuration.

Drafter training is therefore a core asset of any speculative decoding system. AngelSpec’s decision to open-source the training code alongside the framework is more practically significant than releasing only an inference operator: developers can retrain or adapt the drafter to their own traffic distributions instead of being locked into fixed weights.

2. Longer Lookahead Is Not Always Better

Having the drafter predict more tokens at once may appear to reduce the number of target-model invocations, but the longer the candidate chain, the more likely later predictions are to be wrong. Frequent rejections and rollbacks result in wasted GPU computation.

Production systems need to determine draft length dynamically: highly predictable content can support more speculative steps, while open-ended responses may require a more conservative approach. Parameters that perform best in a static benchmark may no longer be optimal in a mixed request queue.

3. Continuous Batching Changes the Economics

In production environments, requests are typically grouped for execution through continuous batching. Different requests vary in input length, generation progress, and acceptance rate, while speculative verification introduces irregular sequence advancement.

This affects:

• KV Cache allocation and reclamation;
• Alignment among different requests within a batch;
• Scheduling between the drafter and target model;
• Throughput and tail latency as concurrency increases;
• Communication overhead in multi-GPU deployments.

A 2× speedup for a single request does not necessarily mean that online serving throughput per GPU will also double. For an inference platform, the ultimate metric is not the tokens/s shown on a demo page, but how many more requests can be handled under a fixed latency target and whether the actual cost per million tokens decreases.

4. The Framework Must Be Operable in Production

A model inference service is not a one-off experimental script. It requires version management, canary releases, monitoring, failure rollback, and capacity planning.

Speculative decoding also introduces a new set of metrics that must be monitored:

• Candidate-token acceptance rate;
• Average number of tokens advanced per verification step;
• Additional computation time consumed by the drafter;
• Rejection and rollback rates;
• Differences in acceleration across request types;
• Changes in P50, P95, and P99 latency.

If a framework reports only average generation speed without exposing these metrics, developers will struggle to determine whether performance fluctuations stem from the model, the scheduler, or the request distribution.

MTP and DFly Represent Two Drafter Approaches

With this release, Tencent Hunyuan has made available the weights and training code for two types of Hy3-A21B drafters: MTP and DFly. Based on the information disclosed so far, AngelSpec does not restrict developers to a single draft architecture. Instead, it places drafter training, candidate generation, and deployment within a unified framework.

MTP generally refers to multi-token prediction: rather than predicting only the immediately following token, the model attempts to provide predictions for multiple subsequent positions simultaneously. Compared with deploying a separate, complete small model, this approach may be able to reuse some of the target model’s representations and reduce the cost of the drafting stage.

DFly is another drafter solution that Tencent has released at the same time. Because the official public information currently focuses primarily on the scope of the open-source release, details such as its network architecture, training objectives, acceptance rate, and performance across different hardware should be assessed based on the repository code, technical documentation, and subsequent benchmarks. At this stage, its internal implementation should not be inferred from the name alone.

Releasing both approaches allows developers to compare the cost curves of different drafters in their own scenarios, rather than seeing only a single, carefully selected best-case figure.

Inference systems should, at minimum, be tested against the following matrix:

| Dimension | Recommended Test Items | | --- | --- | | Request type | Dialogue, code, mathematics, summarization, structured output | | Input length | Short, medium, and long context | | Generation length | Short responses and long-form generation | | Sampling settings | Greedy, low temperature, high temperature, top-p | | Concurrency level | Single request, low concurrency, high concurrency | | Core metrics | Acceptance rate, throughput, TPOT, P95 latency, memory usage |

A system can be considered truly production-ready only if it delivers consistent gains across most cells in this table.

AngelSpec’s Advantage Is That It Shifts the Optimization Target From the Model to the System

The current large-model inference ecosystem has no shortage of speculative decoding implementations. Mainstream inference engines have gradually added support for draft models, MTP, and other speculative decoding approaches, while hardware vendors are also optimizing kernels and graph execution.

To differentiate itself, AngelSpec cannot merely prove that it “can also run speculative decoding.” It must demonstrate three things:

1. Whether training and deployment are genuinely connected. Can a trained drafter enter the production inference pipeline with minimal modification?
2. Whether it can handle real-world concurrency. Can acceleration gains be sustained from single-request tests through continuous batching and multi-GPU serving?
3. Whether it is sufficiently modular. Can developers replace the target model, drafter architecture, and inference backend without rewriting the entire system?

Judging by the scope of Tencent’s current open-source release, AngelSpec has at least chosen the right entry point: rather than providing another isolated set of weights, it includes the framework, drafter weights, and training code together. This is more pragmatic than merely publishing a model’s acceleration ratio under ideal conditions.

Still, some caution is warranted. A framework that “covers everything from training to deployment” is not necessarily ready to integrate into any service at zero cost. For development teams, adoption will ultimately depend on hardware compatibility, inference-backend support, licensing, the pace of community maintenance, and test results under their own request distributions. These details will need to be continuously verified against the official repository.

Who Should Try It

AngelSpec is best suited to three types of teams.

The first is teams that have already deployed Hy3-A21B and face notably high GPU costs on the generation side. The ready-made drafter weights lower the barrier to validation and make it easier to run benchmarks before deciding whether to train a custom version.

The second is platform teams that maintain private inference clusters. Compared with teams that call managed APIs, these teams have control over scheduling, caching, and batching strategies, putting them in a better position to realize the full benefits of speculative decoding.

The third is developers researching drafter architectures and inference scheduling. Releasing the training code alongside two types of weights allows researchers to conduct reproducible experiments involving acceptance rates, draft lengths, and online scheduling.

Conversely, speculative decoding may not be worthwhile if request volume is low or if latency is primarily consumed by retrieval, tool calls, and network transmission. It increases system complexity, while the drafter also requires additional GPU memory and operational resources. For such services, optimizing quantization, batching, KV Cache management, and request routing is usually a more direct approach.

The Direction of Tencent’s Open-Source Release Matters More Than Its Benchmark Scores

Competition among large models is shifting from “who has more parameters” to “who can deliver capabilities at lower cost.” At a stage when gains in model capabilities are slowing while inference demand continues to grow, decoding efficiency will directly affect product margins and service capacity.

AngelSpec is based on the view that speculative decoding should not be merely a piece of experimental code embedded in an inference engine, but should have its own integrated training, architecture, and deployment system. That assessment is fundamentally sound.

Whether it can become general-purpose infrastructure will depend on whether it can expand to support more models, more inference backends, and more complex production scheduling scenarios. Cross-model portability will be particularly important: if AngelSpec’s primary benefits are limited to Hy3-A21B, it will be more of a companion tool for Hunyuan models. If developers can readily train drafters for other mainstream open-source models and integrate them into existing services, it may become a component of the broader inference framework ecosystem.

At this stage, AngelSpec’s greatest value is not an acceleration figure that still requires independent verification, but the fact that Tencent has released the drafter weights, training methods, and deployment pipeline together. For teams already under intense pressure from large-model inference costs, that is more useful than yet another position on a model leaderboard.

References

• GitHub: Search portal for public repositories related to AngelSpec — Used to locate framework code, training scripts, and licensing information subsequently released by Tencent Hunyuan. Repository ownership should be verified against official Tencent Hunyuan announcements.
• Public information released by Tencent Hunyuan on July 29, 2026 — Confirms the open-source release of AngelSpec and the simultaneous release of the MTP and DFly drafter weights and training code for Hy3-A21B. No external link is provided because the source domain is outside the range permitted for citation in this article.