Reasoning Without Teachers

In January 2025, a lab most Western researchers had barely heard of published a result that rearranged the field's assumptions. DeepSeek-R1 showed that a language model could learn sophisticated reasoning — self-verification, reflection, trying alternative approaches — through pure reinforcement learning, with no human-written reasoning traces at all. No supervised fine-tuning first. No demonstrations to imitate. Just a base model, a verifier, and a reward for correct answers.

The experiment that shouldn't have worked

The conventional post-training wisdom, inherited from InstructGPT, was SFT-then-RL: first teach the model the shape of good behavior with human demonstrations, then refine with reinforcement learning. The reasoning was practical — RL from scratch on a base model was assumed too unstable, too sample-hungry, too likely to wander.

DeepSeek-R1-Zero ignored all of that. The team took DeepSeek-V3-Base and applied reinforcement learning directly, skipping supervised fine-tuning entirely. Their hypothesis, stated plainly in the paper, was that human-defined reasoning patterns might actually limit model exploration — that teaching a model to imitate human thought could cap it at human thought. Better to let RL discover its own strategies.

The reward design was deliberately minimal: rule-based rewards for accuracy (is the answer right, verified against ground truth?) plus a small format reward (is the reasoning wrapped in the expected tags?). No neural reward model — the authors explicitly avoided them, citing susceptibility to reward hacking at scale. Just: solve the problem, show your work in the right format.

What happened next is the paper's famous "aha moment." Mid-training, the model spontaneously learned to rethink — to catch its own errors mid-derivation and restart, in an almost anthropomorphic tone. Nobody taught it to do that. The training dynamics told the story in miniature: response lengths grew steadily, the model learned to allocate more tokens to harder problems, and behaviors like self-verification appeared without being programmed — evidence, the authors argue, that sophisticated reasoning is a natural attractor for RL under verifiable rewards, not a human invention to be imitated. AIME 2024 accuracy climbed from 15.6% to 77.9% over training.

From Zero to R1

R1-Zero was a proof of concept with rough edges: poor readability, occasional language mixing, and narrow competence confined to reasoning tasks. The full DeepSeek-R1 added a multi-stage pipeline around the RL core — cold-start data, rejection sampling, a second RL stage, and supervised fine-tuning to align general behavior with human preferences — while preserving the reasoning capabilities RL had discovered.

The full R1 pipeline is worth spelling out, because "pure RL" describes only R1-Zero. The released model went through four stages: a cold-start phase with a small amount of long-CoT data to fix readability and language mixing; a reasoning-oriented RL stage; rejection sampling, where the RL-trained model generated large volumes of reasoning traces that were filtered and used for supervised fine-tuning; and a final RL stage over all scenarios to align with human preferences. Then came distillation: the reasoning patterns were transferred into smaller Qwen and Llama models, which inherited much of the capability at a fraction of the size. Notably, the cold-start data was itself generated by prompting the base model for long chains of thought and filtering — a bootstrapping trick that kept the whole pipeline free of human-written reasoning traces. The open release of the distillations is what turned a lab result into an ecosystem event.

The limitations the authors actually list

The conclusion section is unusually candid, and it's worth reading as a map of where reasoning-RL still falls short:

• Verifiers are the bottleneck. Pure RL works when you can reliably check answers. For math and code, rules suffice; for writing and open-ended tasks, they don't. The authors are direct: scaling pure RL "remains an open challenge" where reliable reward models can't be built, because model-assigned rewards get hacked as training progresses.
• Token efficiency. The model overthinks — generating long reasoning traces even for simple questions. It allocates compute dynamically, but wastefully.
• Tool use and structured output lagged behind existing models, and software engineering saw little improvement (long evaluation times made RL impractical there).
• Prompt sensitivity. Few-shot prompting degraded performance — a genuinely strange finding that hints at how different these models are from their instruction-tuned cousins.

The deepest limitation is philosophical: R1 shows RL can discover reasoning strategies better than human demonstrations, but only in domains with ground truth. The vast territory of tasks where "correct" isn't checkable — judgment, taste, wisdom — remains outside this paradigm's reach.

There's a broader historical rhyme worth naming. R1 is arguably the clearest vindication yet of the "bitter lesson" in the post-training era: the system that improved by optimizing a simple objective at scale beat the systems carefully taught human methods. SFT-first pipelines assumed human reasoning traces were the ceiling; R1 treated them as a possible constraint. The result doesn't prove demonstrations are useless — the cold-start stage exists for a reason — but it reframes them as scaffolding to be discarded, not foundations to build on.

R1's thesis, in one line: the key to unlocking reasoning isn't more human annotation — it's hard questions, a reliable verifier, and enough compute for RL to explore.

Where it sits in the arc

DeepSeek-R1 split post-training into two eras: before, when human demonstrations were assumed necessary; after, when they became optional for verifiable domains. It made GRPO famous, launched a thousand reproduction attempts (DAPO being the most instructive), and established RLVR — reinforcement learning from verifiable rewards — as the dominant paradigm for reasoning models. Every lab training a reasoning model today is, knowingly or not, working in the experimental frame this paper defined.

Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI, 2025.