STaR: Teaching Yourself to Reason, Then Reasoning to Teach Yourself

By 2022, chain-of-thought prompting had shown that asking a language model to reason step by step improves its answers. But there was an annoying bottleneck: getting a model to generate good rationales reliably seemed to require either massive human-annotated rationale datasets or settling for few-shot prompting that underperformed fine-tuning. STaR — the Self-Taught Reasoner — asked the obvious next question: what if the model generates its own rationales, keeps the ones that lead to correct answers, trains on those, and repeats? It was, to the authors' knowledge, the first technique letting a pretrained LLM iteratively use its own language-modeling capacity to improve itself. The loop is almost embarrassingly simple, and it worked.

The loop: generate, filter, rationalize, retrain

You start with three ingredients: a pretrained LLM M, a large dataset of problems with ground-truth answers D = {(x_i, y_i)} (no rationales), and a tiny prompt set P of maybe ten examples that do include rationales. Few-shot prompt M with P and have it generate a rationale r̂_i and answer ŷ_i for each problem. Keep only the rationales that lead to the correct answer (ŷ_i = y_i) — the paper's core assumption is that rationales producing correct answers are better training data than those producing wrong ones. Fine-tune on that filtered set, then repeat with the improved model generating the next round of rationales.

The clever bit is rationalization, which fixes the loop's failure mode. Plain bootstrapping stalls: the model gets no training signal from problems it can't solve, so it eventually stops making progress on the hard tail. For each failure, STaR re-prompts the model with the correct answer included and asks it to produce a rationale justifying that answer — reasoning backward from the hint. Those rationales are then added to the training data as if the model had generated them unprompted. This recovers a large fraction of otherwise-wasted problems and, the ablations show, meaningfully accelerates and improves the whole process.

One implementation detail matters: each iteration fine-tunes from the original pretrained M, not from the previous iteration's model — a deliberate choice to avoid overfitting to the model's own early outputs. And the loop stops when performance plateaus, i.e., when no new problems in the training set get solved.

The ablations justify both choices. Without rationalization, the loop plateaus early, starved of training signal on hard problems; with it, accuracy keeps climbing because previously-unsolvable problems re-enter the dataset as backward-reasoned examples. And the fresh-start fine-tuning held up empirically — continually training on self-generated data overfits to the model's early mistakes, while restarting from M each round lets every generation benefit from strictly better data without inheriting the last round's bad habits.

It's (approximately) reinforcement learning

STaR isn't just a data-augmentation hack; the paper shows it approximates a policy-gradient objective. View the model as a latent-variable model p_M(y|x) = Σ_r p(r|x)·p(y|x,r): it first samples a latent rationale, then predicts the answer. Define the reward as the indicator 1(ŷ = y) — one if the answer is right, zero otherwise. Then the filtering step (discarding wrong-answer rationales) is exactly discarding zero-reward samples, greedy decoding reduces variance of the gradient estimate, and taking multiple gradient steps on the same batch mirrors standard policy-gradient practice. This framing quietly reframes "self-improvement via fine-tuning on own outputs" as RL with a correctness verifier as the reward — a lens that predicts the entire RLVR wave that followed.

The numbers

On CommonsenseQA, STaR improved 35.9% over the few-shot baseline and 12.5% over a model fine-tuned to directly predict answers — and reached 72.5% accuracy, essentially matching a fine-tuned model 30× larger (73.0%). Similar gains held on arithmetic and math word problems. The message: a small number of rationale examples, amplified through the model's own generation and filtered by correctness, can substitute for both massive annotation and massive scale.

STaR's real move isn't the rationale trick — it's the filter. Correctness of the final answer becomes a reward signal, and the model's own generations become the training data. Self-improvement, operationalized as RL with a verifier.

Limitations worth naming

The loop leans entirely on ground-truth answers. No verifiable answer, no filter, no STaR — which is why it shines on math and multiple-choice QA and says nothing about open-ended generation. Rationalization is the shakier half: a rationale generated given the answer is a post-hoc justification, and training the model to produce such justifications fluently is not the same as training it to reason soundly. There's a real risk of the model learning to confabulate convincing wrong reasoning that happens to land on right answers, then amplifying that style in the next round. And like all self-training, errors the filter can't catch — problems where the model is confidently, consistently wrong — get baked in deeper each iteration.

Where it sits in the RSI arc

STaR is the LLM era's first clean RSI loop, and nearly everything after it is a variation on its theme. ReST generalized the generate-filter-retrain pattern; Self-Rewarding LMs replaced the ground-truth filter with the model's own judgment; RLAIF replaced human preference labels with AI labels; Absolute Zero replaced the human-curated question set with a self-proposing curriculum. STaR also foreshadowed the field's central tension: the filter is everything. Whoever supplies the correctness signal — a gold answer, a code executor, a learned reward model, the model itself — determines what "improvement" means, and every RSI method since has been, at bottom, an argument about what to use as the filter.

Paper: STaR: Self-Taught Reasoner — Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, 2022.