Microsoft Introduces NextLat: Making Transformers No Longer "Short-Sighted" — Achieving Both Long-Range Reasoning and Faster Inference

In the past couple of days, a preprint from Microsoft Research resurfaced on r/MachineLearning and has been intensely discussed. The title is provocative — Next-Token Prediction is Myopic. Microsoft Research, together with external collaborators, proposed a self-supervised training objective called Next-Latent Prediction (NextLat). The core idea can be summarized in one sentence: in addition to having the Transformer predict the next token, also have it predict its own next latent state.

It sounds like an auxiliary “self-forecasting” task for the model, but the gains are far from trivial: according to data in the authors' blog, generalization in reasoning and planning tasks improves noticeably, and even more impressively — via self-speculative decoding, inference speed can increase by up to 3.3×.

Why Next-Token Prediction is “Short-Sighted”

Let’s clarify the problem. The standard GPT-style training objective is simple: given the first t tokens, predict the distribution of the (t+1)-th token using cross-entropy. This paradigm has been in use for 7–8 years, powering the product lines from GPT-2 to GPT-5 and Claude 4.5. But it has a long-standing issue — the supervision signal is too sparse.

A one-hot token label, from an information-theoretic perspective, contains only log|V| bits — for a vocabulary of 100,000, that’s about 17 bits. But a hidden layer vector is often 4,096 or 8,192 dimensions, carrying far more information. In other words, the dimensions along which the model is “graded” at each step are severely mismatched with the dimensions of its internal representations.

More critically, this per-token objective naturally encourages the model to take a “greedy shortcut”: just get the next word right, and ignore whether the hidden state has compressed into a stable, long-range-planning-capable world model. The training objective doesn’t care at all. This is what the paper calls myopic — the model can excel at next-token prediction while its internal world model remains loose and fragmented.

In fact, the industry has been trying to break this limitation for the past two years. Meta’s JEPA series and Yann LeCun’s repeated emphasis on “predicting in representation space rather than pixel space” share the same direction: stop fixating on the output space, learn structure in the latent space. NextLat follows the same path, but the clever part is it does not aim to replace next-token prediction — it is added on top of it, integrating into existing architectures at almost zero cost.

What Exactly Does NextLat Do

The mechanism is quite straightforward. Let the current latent state be hₜ, and the next input token be xₜ₊₁. A standard Transformer computes the next latent state hₜ₊₁. NextLat introduces an auxiliary prediction head that makes the model use (hₜ, xₜ₊₁) to directly predict hₜ₊₁, and then match it with the actually computed hₜ₊₁ (the paper uses a latent-space loss like cosine similarity).

The total training objective becomes:

L_total = L_next_token + λ · L_next_latent

It may look like just adding an auxiliary loss, but the resulting effects are interesting:

• Representation compression: To predict what it will look like in the next step, the model must compress history into a compact belief state. Otherwise, if the latent is full of noise, the next step cannot be reliably predicted. This effectively forces out an internal world model.
• Denser supervision: Compared to the 17-bit token label, predicting a high-dimensional vector inherently provides much richer gradient signals. This explains why data efficiency improves — given the same number of tokens, the model learns more.
• Recursive lookahead unlocked: Since the model can predict hₜ₊₁ from hₜ, it can also predict hₜ₊₂ from hₜ₊₁, and so on — theoretically “looking ahead” multiple steps recursively. This is the source of inference acceleration.

How the 3.3× Speedup is Achieved: Self-Speculative Decoding

This is, in my opinion, the most practical part of the work. Speculative decoding is well known — typically, you use a small draft model to quickly guess several tokens, then have the large model verify them in one go, accepting the matching ones. The problem: you need to additionally train or deploy a draft model, which is a hassle engineering-wise.

NextLat’s method is to be its own draft: since the model has learned to predict its next latent state, you can recursively step forward using the latent prediction head to get predicted hₜ₊₁, hₜ₊₂, hₜ₊₃, derive the corresponding candidate tokens using the LM head, and then run a full Transformer forward pass to verify and accept the matches.

The benefits are clear:

1. No extra model required; deployment pipeline unchanged;
2. No reward model or alignment needed — the draft and main model share the same source;
3. Under the paper’s reported settings, yields 2.x to 3.3× end-to-end acceleration with generation quality fully matching the main model (thanks to speculative decoding’s mathematical guarantees).

Compared to methods like Medusa or EAGLE that require separately trained draft heads, NextLat builds the capability right into pretraining, so at deployment it’s just a matter of “flipping the switch.”

Where the Long-Range Reasoning Gains Come From

Speed alone isn’t enough — benchmarks speak louder. The authors’ blog shows several task categories:

• Algorithmic reasoning (like graph traversal, synthetic algorithmic reasoning tasks): NextLat shows clear advantages over the baseline in out-of-distribution length generalization. This is a long-standing weakness of next-token models — train on length 32, test on 64, and they collapse.
• Planning tasks (mazes, blocksworld, etc.): Gains are most obvious here, as planning inherently requires multiple-step internal simulation.
• Standard language modeling: Perplexity doesn’t drop, and even improves slightly. This means the auxiliary objective doesn’t hurt the main task.

It’s worth noting: this approach aligns with another recent research trend — latent reasoning / continuous chain-of-thought. Earlier this year, Meta’s Coconut (doing CoT in latent space instead of emitting tokens) was driven by the same intuition: token space is too discrete and narrow — real “thinking” should happen in the latent. NextLat bakes this idea into the pretraining objective rather than as an inference-time trick, laying a deeper foundation.

A Bit of Sober Assessment

I’m optimistic about this work overall, but a few questions remain:

First, how to tune the auxiliary loss weight λ? The preprint’s experiments aren’t at very large scale yet, so whether λ becomes irrelevant or turns into a critical hyperparameter at, say, 70B or 200B scale is unclear. Auxiliary losses often face the “effective on small models, disappears on large models” issue.

Second, latent objective collapse risk. Predicting one’s next latent state is, in essence, a form of self-distillation, with a theoretical risk of representation collapse (all h becoming similar). The paper likely uses stop-gradient and other tricks, but whether stability holds after large-scale RL fine-tuning needs more validation.

Third, compatibility with MoE and long context. Today’s frontier models are mostly MoE with ±1M context. NextLat integrates smoothly into dense models, but whether it remains as clean amid MoE router noise and sliding-window attention is something the engineering side has to verify.

That said, these are engineering details and don’t diminish the value of the core idea. Predicting latent space vs predicting tokens has been hotly debated over the past year, and NextLat offers a low-intrusion, easy-to-reproduce example — plus the very tangible benefit of acceleration, which, in today’s context of GPU scarcity and high inference costs, is genuinely attractive.

What This Means for Developers

In the short term, you won’t see GPT-6 or Claude 5 explicitly branded with the NextLat name, but the idea of “adding self-supervised objectives in latent space” will likely be absorbed into the next round of pretraining across the board — especially in open source. Teams like DeepSeek or Qwen, which move fast, could replicate a NextLat variant in just a few weeks.

If you’re doing small-model pretraining or fine-tuning in vertical domains, it’s even more worth considering:

• Adding a latent prediction head to your existing training pipeline is very low cost;
• Especially meaningful for low-data-efficiency scenarios (domains with limited corpus);
• Self-speculative decoding can directly cut inference costs.

The authors have already released the GitHub repo, so getting a minimal-demo replication running shouldn’t be hard. Once the community reproduces it at the 7B scale, we’ll know much more about its scalability.

On running comparison experiments across frontier models, OpenAI Hub lets you access GPT, Claude, Gemini, DeepSeek with a single key, direct connection in China, OpenAI-format compatibility, and saves you the hassle of switching SDKs — quite handy for running long-range reasoning benchmarks.

References

• Reddit r/MachineLearning: Next-Latent Prediction Transformers discussion thread — Original discussion of the Microsoft Research preprint and the authors’ explanation
• GitHub: Jayden NextLat code repository — Public implementation and training scripts