On-Policy Distillation: When the Teacher Has to Keep Up With the Student

Self-distillation has a quiet paradox at its center. In on-policy self-distillation, a frozen copy of the model — given the ground-truth solution in its context — plays teacher, supplying token-level supervision while the student learns from its own generations. Freezing the teacher stabilizes training. But the student keeps learning and the teacher doesn't, so over time the supervisor falls behind its own pupil — it can't teach revision behaviors (backtracking, self-correction) that only emerged after it was frozen. This 2026 Meta paper names the problem precisely, measures it, and fixes it by letting the teacher co-evolve with the student. The teacher, too, must be recursively self-improved.

Diagnosing the frozen teacher

The background is OPSD (on-policy self-distillation): the student sees only the problem and generates a response; a privileged teacher — a frozen copy of the base checkpoint additionally given the verified solution — supplies next-token logits at every prefix of the student's response; the student minimizes the divergence (forward KL) between its predictions and the teacher's. Dense, token-level supervision with no external teacher model and no step-level annotations — a middle path between sparse outcome rewards (RLVR) and expensive process supervision.

The paper's diagnostic is devastating in its specificity. On fixed incorrect trajectories from AIME problems, the frozen teacher — despite seeing the verified solution — assigned an average 94.5% probability to EOS (end-of-sequence, i.e., "stop, you're done") at wrong endpoints, but only 22.0% to reflection cues like "wait." The teacher could see the answer and still couldn't teach the student how to recover from wrong reasoning. Meanwhile the student was becoming substantially more reflective during training — a growing mismatch between the evolving model and its fixed supervisor. Revision behaviors strengthen during post-training, but a teacher frozen at the initial checkpoint can never acquire them. Formally, the student minimizes the divergence between its own next-token distribution and the teacher's at every prefix of its own generated response — the teacher sees the problem, the verified solution, and the prefix, while the student sees only the problem and the prefix. Everything that follows is about what happens when that teacher is allowed to learn.

Dynamic Co-Evolution + Self-Refined Concise Learning

The fix has two components. Dynamic Co-Evolution (DCE): after each training round, the resulting checkpoint initializes both the next student and a fresh detached gold-conditioned teacher. Revision learned in one round supervises the next — the teacher perpetually re-absorbs the student's improvements, creating a genuinely recursive loop. Self-Refined Concise Learning (SRCL): stronger revision has a side effect — verbosity. Models that check and backtrack more also ramble more. So SRCL additionally trains on shorter, answer-verified rewrites of the model's own on-policy responses, keeping the revision capability while cutting the token cost. The construction is deliberately on-policy: rewrite the model's own responses more concisely, keep only rewrites whose answers still verify, and train on those. DCE teaches the model to revise; SRCL teaches it to revise efficiently — and the ablations show this isn't cosmetic, because under tight evaluation budgets revision-per-token is what actually wins.

The teacher-evolution measurements confirm the mechanism: across three AIME cohorts, the co-evolving teacher's EOS probability at wrong endpoints fell from 90.4% to 41.3% — it learned to not terminate bad reasoning — while its probability on observed reflection tokens rose from 32.8% to 77.4%. The teacher genuinely learned to teach recovery.

The numbers

On Qwen3-8B across four competition-level math benchmarks, DCE+SRCL reached 65.97% Average@12, outperforming frozen-teacher OPSD by 35.62 percentage points — while generating 7.80% fewer mean output tokens than DCE alone. DCE alone beat OPSD by 35.41, 37.15, and 12.98 points at 8B, 4B, and 1.7B scales. Under a tight 8K evaluation budget, DCE+SRCL hit 35.07% accuracy at ~7,500 tokens per response — 6.6 points better than an OPSD test-time-scaling control while using ~692 fewer tokens. Matched-budget controls confirm longer generation alone doesn't explain the gains, and updating the teacher every round beat frozen, EMA, and periodic alternatives.

The supervisor is part of the system being improved. A frozen teacher — however privileged — becomes the bottleneck the moment the student outgrows it; co-evolving the teacher turns the bottleneck into a second engine of improvement.

Limitations worth naming

The privileged teacher needs ground-truth solutions in its context — the method inherits the verifier-dependence that limits all RLVR-style approaches to domains with checkable answers. The evaluation is math-only; whether co-evolution helps (or even works) for open-ended reasoning is untested. There's a co-adaptation hazard the paper doesn't fully explore: when teacher and student evolve together from the same lineage, they can collude — converging on shared stylistic tics or mutually-reinforcing verbosity that SRCL only partially constrains. And maintaining two evolving roles doubles the compute and complexity versus single-model post-training, a real cost at frontier scale.

Where it sits in the RSI arc

This paper closes a loop the field has been circling since 2024. Self-Rewarding LMs argued the reward model must improve alongside the policy; RLAIF showed AI feedback works but left its labeler frozen. DCE is that argument pushed down to the token level: not just the judge but the per-token supervisor must co-evolve, with measurements (EOS vs. reflection probabilities) precise enough to watch it happen. It's also the natural complement to Absolute Zero's proposer/solver split — there, one model plays two roles in task space; here, two roles co-evolve in supervision space. For anyone working on post-training, the takeaway is architectural: any fixed component in a self-improvement loop — reward model, teacher, labeler, curriculum — is a future bottleneck. The recursion has to include everything, or it stalls at the part you froze.

Paper: Recursive Self-Improvement via On-Policy Distillation for Reasoning — Shangjian Yin et al., 2026.