Deleting the Critic

Buried in a 2024 paper about math reasoning was an algorithmic detail that would end up running some of the most important RL training runs in the world. Group Relative Policy Optimization — GRPO — was introduced almost as an aside in the DeepSeekMath paper: a variant of PPO that deletes the value model entirely. Two years later it was the engine behind DeepSeek-R1. The story of how a memory optimization became the default RL algorithm for reasoning models is worth understanding.

What GRPO actually changes

Standard PPO needs two networks: the policy (which acts) and a value function (which estimates how good each state is, feeding the advantage computation). The value model is roughly the same size as the policy — so half your memory, half your compute, and a whole second training dynamics to babysit goes to a network whose only job is helping the first network learn.

GRPO's move is embarrassingly simple: for each question, sample a group of outputs from the current policy, score them all, and compute each output's advantage relative to the group — the reward minus the group mean, divided by the group standard deviation. No value model. The baseline is the group's own performance.

A_i = (r_i − mean of group rewards) / (std of group rewards)

Everything else stays PPO-like: the clipped surrogate objective, the KL penalty against a reference model. The DeepSeekMath authors framed it modestly, as a way to "enhance mathematical reasoning abilities while concurrently optimizing the memory usage of PPO." In practice, removing the critic roughly halved the memory footprint of RL training — which, at the scales involved, is the difference between feasible and not.

There's an elegance to the group-relative framing beyond efficiency. The advantage now means something intuitive: how much better was this attempt than my other attempts at the same question? The baseline adapts per-question automatically — hard questions get a low bar, easy ones a high bar — without any learned value function to go stale as the policy improves. And because every output in the group answers the same question, the baseline automatically controls for difficulty: a brilliant solution to a hard problem and a mediocre one to an easy one are judged against their own peers, not a global average that would conflate the two.

Why the math paper mattered

DeepSeekMath itself is a serious result: a 7B model continued-pretrained on 120B math-heavy tokens, scoring 51.7% on the competition-level MATH benchmark without tools or voting (60.9% with self-consistency over 64 samples) — approaching Gemini-Ultra and GPT-4 territory at a fraction of the size. The paper attributes this to two things: a carefully engineered data pipeline over public web data, and GRPO.

The data story deserves its own mention. The 120B-token math corpus was built by an iterative bootstrapping pipeline: seed a classifier with high-quality math examples, crawl Common Crawl, filter, retrain the classifier on what survived, repeat. The paper shows this curated corpus beat simply using more raw tokens — data quality dominated quantity. It's a useful corrective to the "just scale" narrative: the MATH score came from curation as much as from GRPO, and the pipeline design is arguably the more transferable contribution.

But the paper's lasting influence isn't the 7B model — it's the algorithm section that R1 later lifted wholesale. GRPO turned out to be particularly suited to reasoning RL: with rule-based rewards (right/wrong), the group-relative advantage is clean and stable, and dropping the value model removes a major source of training instability. The R1 paper adopts it directly, and DAPO's authors treat it as the baseline to beat.

The tradeoffs nobody advertises

Group-relative advantages have a failure mode worth knowing: when every output in the group gets the same reward — all right or all wrong — the advantage is zero for everything, and the group contributes nothing to the gradient. Early in training (everything wrong) or late (everything right), large fractions of your compute can produce literally zero learning signal. DAPO's "dynamic sampling" was invented specifically to fix this: oversample and discard the uninformative groups.

GRPO also inherits a subtle bias from its normalization. Dividing by the group standard deviation means questions with high reward variance get downweighted relative to questions where the group disagrees sharply — which isn't obviously what you want. It's also worth noting what GRPO assumes about reward density: with sparse outcome rewards, most groups early in training are all-wrong, which is exactly the zero-gradient regime DAPO later patched with dynamic sampling. GRPO didn't solve that; it just made the remaining problems cheap enough to brute-force. And without a value model, there's no learned notion of partial progress; a reasoning trace that gets 90% of the way there but flubs the final answer gets the same reward as one that's nonsense throughout, at least under pure outcome rewards. The process-supervision debate — whether to reward intermediate steps — is partly a debate about this blindness.

GRPO's real lesson: in RL for language models, the value function was load-bearing scaffolding from the robotics era — and once rewards became verifiable, the scaffolding could come down.

Where it sits in the arc

GRPO is the hinge between the RLHF era and the reasoning-RL era. PPO needed a critic because its rewards were learned, noisy, and dense. Once the field moved to verifiable, rule-based rewards on reasoning tasks, the critic's job got simple enough for group statistics to handle — and the memory savings unlocked training at unprecedented scale. Every "PPO is dead" take of 2025 is really a GRPO story: the clipped objective survived, the value network didn't.

Paper: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al., 2024.