What SFT Can't Learn

Everyone in post-training "knows" the pipeline: supervised fine-tuning first, RL after. SFT teaches the format; RL teaches the behavior. But surprisingly little work had directly compared what each stage actually contributes — which one memorizes, which one generalizes, and whether you need both. This paper runs the controlled experiment, and its title is the result: SFT memorizes, RL generalizes.

The experimental design

The authors needed tasks where memorization and generalization could be cleanly separated — where you could train on one set of rules and test on different rules, so that only genuine rule-learning (not pattern-matching) would transfer. They built two such testbeds:

• GeneralPoints, an original arithmetic card game: given four cards, combine their values with arithmetic to reach a target (24 by default). The twist is rule variants — train on one rule set, test on unseen variants of the rules. A model that memorized training examples fails; a model that learned arithmetic reasoning transfers.
• V-IRL, a real-world visual navigation environment: spatial reasoning with photographic inputs, testing whether post-training methods generalize across visual variations (colors, layouts) and even across textual action spaces.

Both are evaluated on large vision-language models, and the RL setup is multi-turn with a verifier: the model acts, gets feedback, and can revise — PPO as the backbone, with outcome-based rewards. The comparison is apples-to-apples: same base model, same data budget, SFT vs. RL.

What the results show

The headline finding is clean and replicated across both environments: RL with outcome-based rewards generalizes; SFT memorizes. In-distribution gains from RL transferred to unseen rule variants; SFT's gains didn't. On the visual navigation benchmark, the RL-trained model reached 77.8% on V-IRL mini — up from 44.0%, a state-of-the-art result at the time — while SFT lagged on every out-of-distribution measure.

Two secondary findings are arguably more interesting than the headline:

• RL improved visual recognition itself. Training with outcome-based rewards didn't just teach better reasoning over visual inputs — it improved the model's underlying visual perception. That's a surprising direction of transfer: a reward signal about task success sharpening the encoder's ability to see. It suggests RL's benefits aren't confined to the "reasoning head" but propagate through the whole model.
• SFT is still necessary. The paper doesn't conclude "skip SFT" — it finds that SFT stabilizes the model's output format, which RL then needs as a foundation to achieve its gains. Without the format-stabilizing stage, RL's exploration is wasted on malformed outputs. The pipeline's order isn't arbitrary; each stage solves a problem the other can't.

The RL setup itself is worth noting: a multi-turn formulation where the model generates an answer, receives verifier feedback, and revises — the "sequential revision" framing. This matters because it gives RL something SFT fundamentally lacks: a loop. The model can be wrong, see that it's wrong, and try again, with the reward reinforcing the recovery rather than just the final answer. SFT on expert traces can show a model what correct reasoning looks like; only the interactive loop can teach it to find correctness from incorrectness. That loop, more than any particular algorithm, may be what "RL generalizes" actually means.

The authors also found that scaling inference-time compute — letting the model take more revision steps — improved generalization further, connecting their results to the broader test-time scaling literature. Training-time and test-time compute turn out to be complements here, not substitutes: RL teaches the model to use feedback, and more inference steps give it more feedback to use.

Why this matters for the post-training debate

This paper is the empirical backbone for a claim the reasoning-RL wave keeps making implicitly: that RL does something qualitatively different from SFT, not just quantitatively more. SFT is behavioral cloning — it teaches the model to reproduce the training distribution. RL is optimization against outcomes — it teaches the model to find any path to success, which is why it discovers strategies (and generalizations) absent from the data.

That distinction reframes several live debates. DPO's offline preference learning looks, in this light, more like SFT than like RL — it clones preferences rather than exploring toward them, which may explain its generalization limits. R1-Zero's decision to skip SFT entirely looks bolder but also more principled: if SFT's main job is format stabilization, maybe a format reward (as R1 used) is enough, and full SFT is overkill that risks capping exploration. The paper doesn't settle these questions, but it gives them an empirical vocabulary.

The honest limitation: GeneralPoints and V-IRL are still synthetic testbeds, and "generalization" here means transfer across rule variants and visual perturbations — not the open-ended generalization frontier models need. Whether the SFT-memorizes/RL-generalizes split holds at the scale and messiness of real post-training is the question this paper sets up but can't answer alone.

The paper's real contribution isn't the ranking — it's the mechanism: SFT teaches a model to reproduce, RL teaches it to succeed, and those are different skills with different scaling properties.

Where it sits in the arc

This is the paper to cite when someone asks why the field moved from SFT-heavy pipelines to RL-heavy ones. It provides the controlled evidence behind the intuition driving R1, Kimi, and DAPO: that if you want capabilities beyond your training data, you need the stage that optimizes rather than imitates. As post-training shifts toward agentic tasks — where the right behavior can't be demonstrated, only discovered — the memorizer/generalizer distinction is becoming the central design principle of the whole stack.

Paper: SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training — Chu et al., 2025.