The Three-Step Recipe Behind ChatGPT

The most startling number in the InstructGPT paper isn't about scale. It's about anti-scale: a 1.3-billion-parameter model, fine-tuned with human feedback, produced outputs that human evaluators preferred over those of a 175-billion-parameter GPT-3 — a model with a hundred times more parameters. The paper's thesis, stated plainly in its opening line, is that making language models bigger does not inherently make them better at following a user's intent. What followed was the three-step recipe that now underlies virtually all post-training.

Why pretraining wasn't enough

A base language model is trained to predict the next token on internet text. That objective produces a remarkable simulator of how people write — but "what would a human plausibly write next" is not the same as "what would be helpful here." The paper documents the gap crisply: large models generate outputs that are untruthful, toxic, or simply not helpful, because helpfulness was never in the training objective. Alignment, in this framing, is the problem of redirecting a capable simulator toward user intent.

The paper's answer is a pipeline with three stages, each addressing a different deficiency:

1. Supervised fine-tuning on demonstrations. Human labelers write example responses to a set of prompts — including prompts submitted through the OpenAI API, so the distribution reflects real usage rather than researcher imagination. GPT-3 is fine-tuned on these demonstrations. This teaches the format of helpful behavior but is bottlenecked by how many demonstrations humans can write.

2. Reward modeling from rankings. Labelers rank several model outputs for each prompt. A reward model is trained to predict these rankings. This is the Christiano et al. preference-learning idea, transplanted from Atari trajectories to text — and it's the step that scales, because ranking outputs is far cheaper than writing demonstrations.

3. RL fine-tuning with PPO. The supervised model is optimized with reinforcement learning against the reward model, with a KL penalty keeping it from drifting too far from the supervised starting point. This is where the model learns to generalize the preferences beyond the ranked examples.

What the experiments actually showed

The headline result — 1.3B InstructGPT preferred over 175B GPT-3 — is doing deliberate rhetorical work. It's not a claim that small models beat big ones in general; it's a claim that alignment is orthogonal to scale, and that a smaller aligned model can be more useful than a larger unaligned one. That distinction mattered enormously for productizing language models: it meant you could ship something genuinely useful without waiting for the next scale-up.

Two more results deserve attention. First, InstructGPT showed improvements in truthfulness and reductions in toxic output — the human-feedback signal transferred to qualities beyond "followed the instruction." Second, and just as important for adoption, the paper reported minimal performance regressions on public NLP benchmarks. Alignment didn't cost capability, at least as measured then. That finding — that you could have both — is what made RLHF the default rather than an exotic alignment technique.

One underappreciated detail: the labelers worked from extensive written guidelines specifying what counts as helpful, truthful, and harmless — dozens of pages of edge cases. Labeler agreement was measured and imperfect, which the paper reports honestly. That imperfection matters, because the reward model can only be as coherent as its labels: noise in preferences becomes noise in the reward, which becomes exploitable surface for the policy. Later work on labeler disagreement and pluralistic alignment starts here, in the gap between "human preferences" as an abstraction and real contractors trying to apply a style guide consistently.

The paper also ran ablations that get less attention than they deserve: removing the RL step (SFT only) or the SFT step produced measurably worse human-preference scores, confirming each stage earned its place. And the reward model generalized beyond its training distribution well enough that optimizing against it improved outputs on prompts never seen during ranking — the first real evidence that learned human preferences could be a generalizable training signal rather than a lookup table.

The parts that didn't work

The paper is admirably candid about its limits, and they read as a roadmap for the next three years of research. InstructGPT still made simple mistakes. Its alignment was to the preferences of its labelers — a specific group of contractors, mostly American, whose judgments encoded particular values — which raises the question the field is still wrestling with: aligned to whom? The reward model was trained once and frozen, so the policy could over-optimize against its blind spots; the KL penalty mitigated this but didn't solve it. And the whole pipeline was expensive and fiddly: three stages, PPO hyperparameters, a separate reward model to maintain. Every simplification since — DPO most prominently — has been an attempt to keep the gains while shedding the machinery.

There's also a subtle methodological point worth noticing. The human evaluations were done on the paper's own prompt distribution — prompts from the API and labeler-written ones. "Preferred by humans" means preferred by their labelers on their prompts. That's not a flaw exactly, but it's a scope condition worth remembering whenever alignment claims travel beyond the lab that produced them.

InstructGPT's enduring contribution isn't the model — it's the demonstration that intent-following is a trainable property, separable from raw capability, and that human preference is a sufficient training signal for it.

Where it sits in the arc

This paper turned RLHF from an alignment research program into an industry. Everything after it is commentary: DPO asked whether the RL step was necessary, Constitutional AI asked whether the humans were necessary, and the reasoning-RL wave asked whether preferences were even the right signal when verifiable rewards exist. But the three-stage skeleton — demonstrate, rank, optimize — remains the mental model the whole field reasons with, including when it's trying to replace it.

Paper: Training Language Models to Follow Instructions with Human Feedback — Ouyang et al., 2022.