RL and OPD: Hands-On Post-Training for Large Models

Recently, an in-depth tutorial discussing reinforcement learning (RL) and On-Policy Distillation (OPD) attracted attention in Reddit’s Machine Learning community. It touches on one of the most important technical directions in post-training large models today: GRPO-style reinforcement learning and OPD are appearing with increasing frequency in technical reports for models such as Kimi, DeepSeek, Qwen, and GLM, while the training paradigm is shifting from standalone SFT toward a combination of “reward-driven exploration + dense teacher guidance.”

This is not just another case of renaming algorithms.

RL and OPD solve two different problems: RL tells the model, “Did you ultimately solve this problem correctly?” OPD tries to tell the model, “At each step you just reached, in which direction should the next token be adjusted?” The former excels at discovering new, verifiable strategies, while the latter excels at compressing a strong model’s local decisions into a student model.

As we enter 2026, these two methods are converging. A Distilled Reinforcement Learning study released in July made the debate even more explicit: pure RL provides only coarse outcome rewards, making credit assignment difficult; pure OPD, meanwhile, can easily devolve into indiscriminate imitation, copying the teacher’s stylistic habits, redundant reasoning, and even erroneous preferences. A truly effective training system should not choose one over the other. Instead, it should determine what each type of supervision signal is responsible for.

Rather than restating paper abstracts, the discussion below breaks the topic down directly through mathematics and code.

1. First, Put Everything in the Right Place: Pretraining, SFT, RL, and OPD Each Handle a Different Stage

Large-model training is often described broadly as “predicting the next word,” but the data distributions and optimization objectives differ across stages.

1. Pretraining: Learning a Distribution from Human-Generated Corpora

The pretraining objective is the simplest: given the preceding context, maximize the likelihood of the next token.

Let the sequence be x = (x₁, x₂, ..., xₜ). The loss can be written as:

L_pretrain = -Σₜ log πθ(xₜ | x＜ₜ)

In essence, pretraining compresses the statistical patterns found across the internet, code repositories, and books. The model acquires knowledge, grammar, and some reasoning patterns, but it does not know how developers want it to respond, nor does it explicitly optimize for mathematical accuracy or code pass rates.

2. SFT: Learning “What a Standard Answer Looks Like”

Supervised fine-tuning still uses cross-entropy, except that the data changes from natural corpora to instruction–answer pairs:

L_SFT = -Σₜ log πθ(yₜ | x, y＜ₜ)

SFT is highly stable and relatively inexpensive to engineer, but it can only imitate trajectories already present in the data. If the training set contains only one solution method, the model will usually learn only to reproduce that path. More importantly, SFT treats all tokens equally: a crucial step that makes a proof valid and connective phrases such as “therefore” or “next” are included in the same averaged loss.

3. RL: The Trajectory Does Not Need to Match, as Long as the Outcome Improves

Reinforcement learning changes the form of supervision. The model samples answers on its own, and a reward function then evaluates the outcome. For math problems, the final numerical result can be verified; for coding tasks, tests can be run; for tool use, the system can check whether the task was completed.

This allows the model to discover paths that never appeared in the training data. That is RL’s most important value for reasoning models: rather than making the model memorize reference answers more faithfully, it lets the model search for strategies under reward constraints.

4. OPD: Letting the Teacher Guide the Student on States the Student Actually Encounters

Traditional knowledge distillation usually trains the student to match the teacher’s distribution on a fixed dataset. However, the contexts in that fixed dataset may not be the same contexts the student visits during inference.

The difference in OPD lies in “On-Policy”:

1. The student model generates a response using its current policy;
2. The prefixes actually visited by the student are collected as states;
3. The teacher outputs token probability distributions on those states;
4. The student learns to match the teacher’s judgments on those states.

This can be understood through driving instruction. Traditional distillation asks the learner to watch a prerecorded standard route driven by the instructor; OPD lets the learner drive, and at every intersection they reach, they ask the instructor which way to turn the steering wheel. The latter covers the places where the student actually makes mistakes.

2. What Exactly Does GRPO Do?

PPO was once the dominant approach for language-model RLHF, but it usually requires training an additional value model to estimate the value of each state. As model scale increases, keeping the policy model, reference model, reward model, and value model resident at the same time imposes substantial memory and communication costs.

The core idea of GRPO is to compare a group of sampled responses to the same problem, using those comparisons to approximate advantage values without training a separate critic.

Suppose the model samples G responses for prompt x:

y₁, y₂, ..., y_G ~ π_old(· | x)

The reward function returns:

R₁, R₂, ..., R_G

The most common group-normalized advantage is:

Aᵢ = (Rᵢ - mean(R)) / (std(R) + ε)

A response that performs better than the group average receives a positive advantage, while one that performs worse receives a negative advantage. The model then increases the probability of high-advantage responses and decreases the probability of low-advantage responses.

For token t in response i, the importance ratio is:

rᵢ,ₜ(θ) = πθ(yᵢ,ₜ | sᵢ,ₜ) / π_old(yᵢ,ₜ | sᵢ,ₜ)

After applying PPO-style clipping, the core term in the objective is:

min(rᵢ,ₜ Aᵢ, clip(rᵢ,ₜ, 1-ε, 1+ε) Aᵢ)

The purpose of clipping is not to raise the performance ceiling, but to prevent any single update from moving too far. If a high-reward response happens to be sampled, an unconstrained policy gradient may rapidly amplify its probability, eventually causing format collapse, excessively long answers, or reward exploitation.

A complete implementation usually also includes a KL constraint against a reference model:

L = -L_policy + β D_KL(πθ || π_ref)

The reference model is generally the initial SFT checkpoint. It acts as a safety line, preventing the policy from losing its original language capabilities while pursuing a single reward.

However, stronger KL regularization is not always better. If β is too large, the model barely explores; if it is too small, reward hacking becomes more likely. In practice, it is more reasonable to monitor the actual KL and adjust it dynamically than to hard-code the coefficient and look only at the total loss.

3. A Minimal Implementation of the GRPO Core

The code below omits generation, distributed communication, and reward executors, retaining only the key path of the policy loss. Here, old_logp must come from the old policy used during sampling. It cannot be recomputed after the update, or the importance-sampling ratio loses its meaning.

import torch


def masked_mean(x, mask, dim=None):
    x = x * mask
    return x.sum(dim=dim) / mask.sum(dim=dim).clamp_min(1)


def grpo_loss(
    new_logp,       # [batch, group, seq]
    old_logp,       # [batch, group, seq]
    rewards,        # [batch, group]
    response_mask,  # [batch, group, seq]
    clip_eps=0.2,
):
    reward_mean = rewards.mean(dim=1, keepdim=True)
    reward_std = rewards.std(dim=1, keepdim=True, unbiased=False)
    advantage = (rewards - reward_mean) / (reward_std + 1e-6)

    # Broadcast a sequence-level outcome reward to all generated tokens in the response
    advantage = advantage.unsqueeze(-1)

    log_ratio = new_logp - old_logp
    ratio = torch.exp(log_ratio)
    ratio_clipped = ratio.clamp(1.0 - clip_eps, 1.0 + clip_eps)

    objective_1 = ratio * advantage
    objective_2 = ratio_clipped * advantage
    token_objective = torch.minimum(objective_1, objective_2)

    return -masked_mean(token_objective, response_mask)

The code is short; all the hard parts lie outside it.

Reward Design Matters More Than the Optimizer

For mathematical tasks, one cannot simply determine whether output strings are identical. The model may produce equivalent fractions, answers with units, or different formats. Therefore, the system generally needs to:

• Parse the final answer first, then perform symbolic or numerical verification;
• Track formatting rewards and correctness rewards separately;
• Prevent formatting rewards from outweighing task rewards;
• Distinguish between unparseable outputs and explicitly incorrect answers;
• Set timeouts, sandboxes, and hidden tests for coding tasks.

If the reward function contains a loophole, RL will not “understand your true intent.” It will simply find the loophole faster.

Within-Group Variance May Drop Directly to Zero

If all G responses to a problem are correct or all are incorrect, the normalized advantages will be close to zero, and the batch will produce almost no gradient. This can happen when problems are too difficult early in training or too easy later in training.

Possible solutions include dynamic curriculum sampling, adjusting problem difficulty based on success rate, increasing the sampling temperature, or using R - mean(R) without dividing by the standard deviation. The latter reduces the risk of abnormally amplifying batches with very small variance, but makes gradient scales less consistent across different problems.

Length Normalization Is Not a Harmless Detail

After copying a sequence-level reward to every token, directly averaging over all tokens can weight long and short responses differently depending on the implementation. Some systems average by sequence, some average globally over tokens, and others introduce length penalties.

This materially changes the model’s style. When reasoning models become increasingly verbose, the cause is often not just the data. It may also result from the interaction between loss normalization and reward design.

4. The Key to OPD Is Not “Distillation,” but the State Distribution

Let the student model be πθ and the teacher model be πT. The student first samples a trajectory y ~ πθ(· | x), after which both teacher and student compute next-token distributions on the prefixes produced by the student.

One common objective minimizes token-level KL divergence:

L_OPD = Σₜ D_KL(πT(· | x, y＜ₜ) || πθ(· | x, y＜ₜ))

Cross-entropy can also be used:

L_OPD = -Σₜ Σᵥ pT(v | sₜ) log pθ(v | sₜ)

The most important point here is that the teacher does not generate an entire standard answer for the student to copy. The trajectory comes from the student, and the teacher only provides a probability distribution at the positions the student actually reaches. This reduces the state-distribution shift between training and inference.

Consider a code-generation scenario: the student has already chosen a data structure that is not ideal but can still complete the task. An offline-distillation reference answer might use an entirely different structure from the beginning, making its subsequent tokens less useful in the current context. An OPD teacher, after seeing the code the student has already written, can suggest a more reasonable next step for that specific situation.

This is also where OPD is more fine-grained than sequence-level distillation: it transfers not only “what the teacher ultimately wrote,” but also the teacher’s relative preferences among candidate tokens. How far the second-best token trails the best one, and whether an alternative remains acceptable, are all encoded in the logits.

5. A PyTorch Skeleton for OPD

The following assumes that the student has already generated input_ids, with response_mask covering only the portion generated by the student. The teacher’s parameters are frozen, but the teacher must perform a forward pass over the same batch of student trajectories.

import torch
import torch.nn.functional as F


def opd_loss(student, teacher, input_ids, attention_mask, response_mask, temperature=1.0):
    student_logits = student(
        input_ids=input_ids,
        attention_mask=attention_mask,
    ).logits[:, :-1]

    with torch.no_grad():
        teacher_logits = teacher(
            input_ids=input_ids,
            attention_mask=attention_mask,
        ).logits[:, :-1]

    # The logits at position t predict position t+1
    mask = response_mask[:, 1:].float()

    student_logp = F.log_softmax(student_logits / temperature, dim=-1)
    teacher_logp = F.log_softmax(teacher_logits / temperature, dim=-1)
    teacher_prob = teacher_logp.exp()

    token_kl = (teacher_prob * (teacher_logp - student_logp)).sum(dim=-1)
    loss = (token_kl * mask).sum() / mask.sum().clamp_min(1.0)

    return loss * (temperature ** 2)

This code illustrates the mathematical relationship, but using it directly for large-model training would be extremely expensive. Full-vocabulary logits have shape [batch, seq, vocab], and both teacher and student must compute and retain the relevant tensors. With a vocabulary of 150,000 and contexts containing tens of thousands of tokens, communication and memory pressure can quickly exceed that of the parameters themselves.

In practice, at least one of the following optimizations is usually required:

• Transmit only the Top-K logits from the teacher distribution and approximate the tail probabilities;
• Use tensor parallelism for the teacher and data parallelism for the student, with asynchronous transmission of the distillation signal;
• Shorten the effective response length used for OPD;
• Cache teacher logits on rollout workers while strictly controlling staleness;
• Use models from the same family or a shared tokenizer to avoid vocabulary-alignment problems;
• Calibrate teacher and student temperatures separately to prevent the teacher distribution from becoming overly sharp.

The last point is often overlooked. If the teacher assigns a probability close to 1 to a single token, OPD degenerates into hard-label SFT. If the temperature is too high, however, the distribution becomes excessively flat, and the student learns only noise.

6. Why RL and OPD Complement Each Other

The signals provided by the two methods can be laid out along two dimensions:

| Method | Signal Granularity | Requires a Teacher | Can Explore New Trajectories | Main Risks | |---|---:|---:|---:|---| | SFT | Token-level | Requires reference answers | Weak | Limited by the data distribution | | GRPO/RL | Sequence- or process-level | Not necessarily | Strong | Credit assignment, reward exploitation | | OPD | Token-level distribution | Requires a teacher | Depends on student sampling | Blind imitation, high cost |

RL rewards are usually sparse. If a 4,000-token proof is ultimately correct, the policy gradient broadcasts a positive advantage across the entire sequence, but it does not know whether the key turning point occurred at token 317 or whether the model merely corrected an error by chance in the final step. This is the credit-assignment problem.

OPD can provide exactly the dense directional signal that is missing: at every prefix, the teacher indicates which tokens are more reasonable. But OPD also has its own fundamental weakness—teacher preferences are not the same as task rewards. The teacher may prefer longer answers, may be biased toward a particular coding style, or may provide distributional targets that the student architecture cannot realize.

A reasonably designed hybrid objective is therefore:

L_total = λ_RL L_RL + λ_OPD L_OPD + λ_KL L_ref

Going further, the same OPD weight should not be used for every token. It can be gated based on reward, teacher confidence, or student–teacher divergence:

• The response is ultimately correct, but the teacher and student disagree substantially: distill cautiously to avoid erasing a novel strategy discovered by the student;
• The response is incorrect, and the teacher has high confidence at critical positions: increase the OPD weight;
• The teacher has high entropy: this indicates that the teacher is also uncertain, so reduce the distillation signal;
• The teacher and student are nearly identical: continued distillation adds little new information and can be skipped;
• The teacher and student differ extremely: this may indicate a capability gap or tokenizer issue, so direct strong matching is inappropriate.

This goes one step beyond simply “adding an RL loss and a KL loss”: it acknowledges that teacher reliability varies across samples and positions.

7. Why “Finding the Strongest Teacher” May Not Be Optimal

OPD presents a seemingly counterintuitive teacher-selection problem.

If the teacher is very similar to the student, it provides little new knowledge. If the teacher is far stronger than the student, however, the distribution gap may be too large. A 7B student may not be able to reproduce the long-horizon planning of an enormous MoE teacher. Even if it tries hard to match the teacher’s local logits, it may learn only superficial phrasing.

This is like asking a student who has just learned calculus to imitate a graduate-level proof word for word. Each individual step may be copyable, but the student may still be unable to internalize the complete abstract structure.

In practice, several approaches can be considered:

1. Switch teachers in stages: First use a teacher with similar capabilities to establish a stable policy, then switch to a stronger teacher to expand the frontier;
2. Distill only high-value positions: For example, branch choices, tool-call parameters, and critical derivation steps, rather than every connective word;
3. Perform sequence filtering before token distillation: Invoke the teacher only for student trajectories that receive high rewards or can be repaired;
4. Preserve student advantages: When the student’s outcome reward exceeds that of the teacher’s generated response, do not forcibly pull it back toward the teacher distribution;
5. Use teacher ensembles: Route tasks to specialized math, coding, and agent teachers instead of expecting a single model to cover every domain.

OPD moves capability integration from parameter space into output-distribution space. Compared with directly merging model weights, it provides finer control over “what capabilities are absorbed on which data, in which states, and at which tokens.” But the cost is equally clear: the training system expands from a single model into a pipeline composed of online rollouts, teacher inference, reward execution, logits transmission, and student updates.

The algorithmic formulas are simple. The systems engineering is not.

8. In Real Deployments, Monitor These Six Groups of Metrics First

Training loss alone is almost useless for determining whether RL or OPD is effective. At a minimum, the following metrics should be recorded:

1. Reward Decomposition

Do not use only a single aggregate reward. Correctness, formatting, length, tool success rate, and safety constraints should be tracked separately. Otherwise, it is impossible to tell whether the model has learned to solve problems or merely learned to satisfy formatting requirements.

2. Within-Group Success Rate per Problem

Record pass@G, the proportion of groups in which all responses are correct, the proportion in which all are incorrect, and the reward standard deviation. If many groups receive identical rewards, GRPO is not obtaining an effective comparison signal.

3. Policy KL and Clipping Fraction

If the actual KL continues to increase, the model is rapidly diverging from the reference policy. If the clipping fraction remains high for a long time, the learning rate, advantage scale, or number of update epochs may be too aggressive.

4. Teacher–Student KL and Teacher Entropy

A decreasing OPD loss does not necessarily indicate improved capabilities. It may simply mean that the student has learned the teacher’s stock phrases. Metrics should be broken down by task type, token position, and final reward.

5. Response Length and Termination Behavior

Reasoning RL can easily cause response lengths to balloon. An increase in average length does not necessarily indicate stronger reasoning. It may instead reduce training throughput, increase timeout rates, and drive evaluation costs out of control.

6. Real Performance on Isolated Evaluation Sets

Once a reward model participates in training, the training reward is no longer a trustworthy evaluation metric. Math evaluations should use different problem types, coding tasks should use hidden tests, and agent tasks should change environment parameters. Only then can reward overfitting be detected.

9. Our View: OPD Will Become a Mainstream Component, but It Will Not Replace RL

OPD’s recent rise is not because it is more “advanced” than RL, but because the industry has begun to take information efficiency in post-training seriously.

Assigning only a 0 or 1 to the final answer and then expecting the model to determine which step went wrong among thousands of tokens is indeed extremely sample-inefficient. Teacher logits provide denser local direction, making them especially suitable for mathematics, coding, tool use, and small models in vertical domains. But without externally verifiable rewards, the student can easily become a compressed package of the teacher’s style rather than a policy model capable of independent search.

The configuration most likely to become standard is a hybrid system with clear division of responsibilities: SFT handles cold starts, RL drives exploration using verifiable outcomes, OPD adds token-level guidance on states the student actually visits, and reference KL plus capability replay prevent forgetting.

For developers, the most important step is not to immediately reproduce some complicated acronym, but to answer three questions first:

• Does your task have a reliable reward that is resistant to exploitation?
• Can the teacher provide a distribution on the student’s real trajectories that is more valuable than hard labels?
• Can the training infrastructure afford the dual cost of rollouts and teacher inference?

If the first condition holds, start with GRPO. If the second also holds and teacher invocation costs are manageable, add OPD. If neither holds, a clean SFT dataset may still be more effective than a complex post-training pipeline.

Large-model post-training has moved beyond “which algorithm should we choose?” and entered the stage of “how should supervision signals be allocated?” The true value of RL and OPD lies not in one replacing the other, but in finally bringing outcome-level feedback and token-level feedback into the same training pipeline.

References

• Reddit: Deep Dive on RL and OPD for Training LLMs: An introduction to the recently discussed tutorial on the mathematics and code behind RL, GRPO, and OPD.
• GitHub: Datawhale, “Build a Large Language Model from Scratch,” Reinforcement Learning Chapter: Includes the GRPO objective, the GSM8K training workflow, and practical code examples.
• Zhihu: The Rise of On-Policy Distillation Through the Lens of Technical Reports: Reviews the use of OPD in recent large-model technical reports and approaches to capability integration.