Teaching AI to Criticize Itself

Every RLHF pipeline has the same bottleneck: humans. Someone has to rank the outputs, thousands of times, and the model's values can never exceed the quality and consistency of those judgments. Constitutional AI asked the obvious next question: what if the feedback came from the AI itself, guided only by a written list of principles — a constitution?

Two phases, zero human labels on outputs

The method has a supervised phase and an RL phase, and the constitution — a list of rules like "choose the response that is most helpful and harmless" — is the only human input to either.

In the supervised phase, the model samples its own responses, then critiques and revises them against the constitution. A response that violates a principle gets rewritten with an explanation of the violation; the original model is then fine-tuned on the revised responses. The model is, in effect, teaching itself to follow rules by practicing self-correction — and because the critiques use chain-of-thought reasoning, the process is inspectable in a way that pure reward optimization isn't.

In the RL phase, the model samples pairs of responses and an AI — not a human — judges which better follows the constitution. Those AI preferences train a preference model, which becomes the reward signal for RL. The authors call this RLAIF: reinforcement learning from AI feedback. The result, they report, is a harmless but non-evasive assistant — one that engages with harmful queries by explaining its objections rather than refusing blankly.

The conceptual move is worth pausing on. RLHF treats human judgment as the ground truth to be approximated. Constitutional AI treats human judgment as one implementation of something more abstract — principles — and bets that a model can apply those principles to its own outputs about as well as a contractor could. It's a shift from "learn what humans want" to "learn to follow rules humans wrote," and those turn out to be different tasks.

Why it mattered beyond the technique

The paper landed two ideas that outlived its specific recipe. First, self-critique as a training signal: the supervised phase showed that a model revising its own outputs against explicit criteria produces genuinely better behavior, a finding that now shows up everywhere from self-refinement loops to agent training. The chain-of-thought critiques also prefigured today's interest in making model reasoning legible — the revision traces are a primitive form of the transparency the field keeps reaching for.

Second, RLAIF as a scaling strategy. Human feedback doesn't scale to superhuman tasks — a human can't reliably rank outputs they can't evaluate. AI feedback, whatever its flaws, doesn't have that ceiling. The paper is explicit that this is the motivation: enlisting AI help to supervise other AIs. Every subsequent attempt to train models on tasks humans can't judge — from debate to weak-to-strong generalization — is working in the space this paper opened.

The constitution itself is worth reading as an artifact — it mixes commonsense duties ("do not help with wrongdoing") with specific behavioral prescriptions ("explain objections rather than refusing evasively") and even nods to the UN Declaration of Human Rights. The harmlessness evaluations showed the RLAIF-trained model rated comparably to a human-feedback baseline while being notably less evasive: where an RLHF model might refuse a sensitive question outright, the constitutional model would engage, explain its reasoning, and offer safe alternatives. That "harmless but non-evasive" property was the paper's real product goal — a model you could actually talk to about hard things.

The honest limitations

A constitution is only as good as its authors, and the paper's principles were written by the research team — which just moves the "aligned to whom?" question one level up rather than answering it. AI judges inherit the biases of the models doing the judging; if the base model has blind spots, the constitution-following it teaches itself will have them too, now laundered through an official-sounding process. The paper also evaluates harmlessness primarily through human judgment of its outputs, which means the thing being optimized (AI preferences) and the thing being measured (human preferences) can drift apart silently.

There's a subtler worry: self-critique works when the model can recognize its own mistakes, but the errors that matter most are the ones it can't see. A model confidently wrong about a principle will revise its outputs confidently in the wrong direction, and the constitution provides no external check. The method amplifies the model's existing judgment; it doesn't supply judgment the model lacks. And there's an open question about whose constitution it is: the principles reflect their authors' worldview, and the method offers no mechanism for contesting them. A constitution you can't amend is just a prompt with better branding — which is why later work on collective and democratic alignment treats the constitution itself as the thing to be negotiated, not merely followed.

Constitutional AI's real claim: that principles are a more scalable supervisory signal than preferences — because rules can be debated, revised, and applied by the very systems they're meant to govern.

Where it sits in the arc

Constitutional AI is the bridge between RLHF and the self-improvement era. It kept InstructGPT's two-phase skeleton (supervised then RL) but replaced the human at both stages — with the model's own critiques in phase one and AI preferences in phase two. Claude, the product this research fed directly, became the industry's proof that RLAIF could work at scale. And the paper's core bet — that AI systems can participate in their own supervision — is now one of the load-bearing assumptions of the entire alignment field.

Paper: Constitutional AI: Harmlessness from AI Feedback — Bai et al., 2022.