Liquid AI Takes on LLM “Repetition Syndrome”: FTPO Penalizes Only the Final Token

Last week, Liquid AI dropped a technical blog post about how they tackle an old LLM problem: the Doom Loop (infinite repetition loops). Their method is called FTPO (Final Token Preference Optimization). The idea is simple and direct: when things go wrong, it’s often because of a single token, so just punish that one token instead of penalizing the whole sequence.

If you’ve worked with small-model APIs—especially for long-form writing or multi-step agent reasoning—you’ve probably seen this before: the model suddenly gets stuck repeating the same sentence over and over until it hits max_tokens. In the open-source community, the common hacks are basically three things: increase temperature, add repetition_penalty, or use a DRY sampler. They all help somewhat, but only treat the symptoms. Liquid AI is trying to fix the problem at the training stage itself.

How Doom Loops Actually Happen

First, the phenomenon itself. In the blog, the Liquid team gave several typical examples: after generating a block of code, the model suddenly repeats a function name dozens of times; or while answering an open-ended question, it gets trapped in a sentence pattern and keeps going: “In short, ... In short, ... In short, ...” endlessly.

This is not a new issue. It already existed back in the GPT-2 era, and engineering explanations usually point to two causes:

1. Tail collapse in the training distribution — under certain contexts, the model concentrates too much probability mass onto a small number of tokens. Once sampling lands on one of them, escaping becomes difficult.
2. Self-reinforcement — once the model repeats itself once, that repeated content becomes part of the context, further increasing the probability of repeating again.

The second point is the critical one. It creates a positive feedback loop. Temperature can delay the explosion, but it cannot stop it.

Why Existing Solutions Aren’t Enough

DPO (Direct Preference Optimization) seems like the most natural choice at first: treat good outputs as chosen samples and bad repetitive outputs as rejected samples, then align the model accordingly. But the Liquid team points out several major problems with DPO in this scenario.

First, the penalty granularity is too coarse. DPO computes loss over the entire rejected sequence. The model learns that “this whole output is bad,” even though maybe 95% of the tokens are perfectly fine and only the final step slipped into a loop. Penalizing the whole sequence lowers the probability of good parts too, causing style drift and reduced diversity.

Second, training efficiency is low. Every rejected sample requires a chosen counterpart, and you still need long-sequence log-probability computation. Fixing a single token issue becomes disproportionately expensive.

Third—and more subtly—over-alignment smooths out the model too much. Previous research such as DivPO has already warned that preference optimization without diversity considerations gradually makes models more “standardized,” reducing creativity and long-tail expression.

FTPO: Surgical Precision Penalties

FTPO’s core insight can be summarized in one sentence: the formation of a Doom Loop can usually be traced back to one specific token decision. Once that token is chosen incorrectly, everything afterward avalanches. So training only needs to focus preference optimization on that one token.

The workflow roughly looks like this:

• Sample a batch of model outputs and use a detector (such as n-gram repetition rate) to identify Doom Loop samples.
• Locate the “critical token” — the exact step where the loop truly begins.
• At that position, construct a preference pair: the sampled token becomes the rejected token, while an alternative token (either the second-highest probability token or one sampled from a reference model) becomes the chosen token.
• Compute preference loss only at this token position, leaving gradients at other positions untouched.

This approach has several direct advantages:

The training signal is extremely sparse but highly precise. It effectively tells the model: “Everything else you did was fine, just don’t make this specific move.” The model doesn’t need to reshape its entire behavior to fix one bug.

High sample efficiency. A long output may contain only one critical token, and FTPO trains specifically on that token. There’s no need to pair entire chosen sequences, reducing engineering complexity and VRAM usage.

Better diversity preservation. This resembles the motivation behind DivPO: if you leave the “correct” tokens untouched, the model’s output distribution avoids large-scale distortion.

Experimental Results and Some Limitations

In the evaluation results shared by Liquid, they mainly tracked two metrics: Doom Loop occurrence rate and changes in general benchmark performance. The former dropped significantly, while the latter remained largely unchanged—which is rare in preference optimization, since most alignment methods reduce general capability to some extent.

Still, FTPO is not a universal solution. It has several clear limitations:

• Critical token localization depends on the detector. If the detector mistakenly classifies legitimate repetitive patterns (such as lists or repeated code structures) as Doom Loops, the training signal becomes incorrect.
• Limited effectiveness against semantic-level “hidden repetition.” Some models repeat the same idea using different wording. These semantic loops cannot be solved through single-token penalties alone.
• Needs to be applied in late-stage training. Liquid recommends using FTPO after SFT + DPO as a final refinement step. Applying it immediately after pretraining may reduce effectiveness due to insufficient base alignment.

What This Means for Developers

If you’re building products on top of open-source models—especially in long-text generation, agents, or coding assistants—this FTPO approach is worth copying. You don’t need to retrain the base model; you can simply add a Doom Loop–focused “patch training” stage after your existing fine-tuning pipeline.

The pseudocode roughly looks like this:

# 1. Generate samples and identify doom loops
loop_samples = detect_doom_loops(model.generate(prompts))

# 2. Locate the critical token position
for sample in loop_samples:
    critical_idx = find_loop_start(sample.tokens)
    rejected_token = sample.tokens[critical_idx]
    chosen_token = pick_alternative(sample.logits[critical_idx], exclude=rejected_token)
    
    # 3. Compute preference loss only at this position
    loss = ftpo_loss(
        logits_at=critical_idx,
        chosen=chosen_token,
        rejected=rejected_token,
    )
    loss.backward()

One hidden engineering advantage is that this method is very VRAM-friendly. Traditional DPO requires full forward passes through two models (policy and reference), and long sequences can quickly blow up memory usage. Since FTPO only cares about one token position, many optimizations become possible—for example, retaining hidden states only around the critical token.

Looking at the Bigger Picture

Over the past two years, preference optimization has consistently moved toward finer granularity. The progression has gone from sequence-level RLHF, to pairwise DPO, to KTO requiring only one-sided labels, to step-level Process Reward Models, and now FTPO pushes granularity all the way down to individual tokens.

Behind this trend is an emerging consensus: model errors are often local, but our training signals have traditionally been global, wasting a huge amount of learning efficiency in between. FTPO is just one application of this broader direction. It’s likely we’ll soon see more “localized preference optimization” methods targeting specific failure modes—for example, hallucination-token PO for hallucinations or refusal-token PO for over-refusal behavior.

Liquid AI itself has always focused on efficiency. Their earlier LFM small-model series also emphasized achieving stronger performance with fewer parameters. FTPO fits perfectly with that philosophy: no grand narratives, just identifying a concrete problem and solving it with the most economical method possible.

For developers in China, if you’re building products with open-source models like Qwen, DeepSeek, or GLM, you no longer need to rely solely on brute-force repetition_penalty tweaks to fight repetition issues. Borrowing FTPO’s ideas and doing a targeted fine-tuning round with just a few hundred labeled examples may deliver much larger gains than you expect.

References

• DivPO paper interpretation - Zhihu: Related work on preserving output diversity during preference alignment, closely aligned with FTPO’s motivation for maintaining diversity.