AlphaZero: The Self-Play Proof That Tabula Rasa Scales

Every discussion of recursive self-improvement eventually has to answer one question: has it ever actually worked? AlphaZero is the cleanest yes in the field's history. Starting from random play, knowing nothing but the rules, a single general algorithm reached superhuman level in chess, shogi, and Go — each within 24 hours — and convincingly beat the world-champion program in each game. No human games, no handcrafted evaluation functions, no domain-specific search heuristics refined over decades. Just self-play, a neural network, and a loss function. It's the existence proof the entire RSI literature leans on.

Tabula rasa, one algorithm, three games

The setup is deliberately spartan. A single deep neural network f_θ(s) = (p, v) takes a board position and outputs a move-probability vector p and a scalar value v estimating the expected game outcome. No handcrafted features — the board is encoded as spatial planes derived only from the basic rules. Instead of alpha-beta search with decades of chess-specific enhancements, AlphaZero uses general-purpose Monte Carlo tree search: each simulation walks the tree selecting moves with low visit count, high prior probability, and high value, and the search returns move probabilities derived from visit counts.

Training is pure self-play reinforcement learning from random initialization. Games are played by both sides using MCTS; at the terminal position the outcome z (−1/0/+1) is scored by the rules; the network is updated to minimize (z − v)² − πᵀlog p + c‖θ‖² — value regression plus policy cross-entropy against the search's move distribution, plus regularization. The updated network plays the next games. One detail distinguishes it from AlphaGo Zero: there's no gating — no "new player must beat the old champion by 55% to take over." A single network updates continually, with self-play always generated from the latest parameters, which removes an entire evaluation phase from the training loop.

Three deliberate simplifications made it more general than AlphaGo Zero, not less. First, AlphaGo Zero exploited Go's rotational and reflectional symmetries — eightfold data augmentation plus random board transformations during MCTS. Chess and shogi are asymmetric (pawns march forward, castling differs by side), so AlphaZero drops symmetry exploitation entirely: no augmentation, no transformations. Second, identical hyperparameters are reused across all three games — same architecture, same settings — with a single exception: the exploration noise added to the prior policy is scaled to each game's typical number of legal moves. Third, training ran a fixed 700,000 steps; the authors explicitly note they did not reproduce AlphaGo Zero's 100× longer regime. Generality, here, meant removing game-specific crutches and showing the core loop still worked.

The numbers, which are absurd

Training ran 700,000 steps (minibatches of 4,096), with 5,000 first-generation TPUs generating self-play games and 64 second-generation TPUs training the network. AlphaZero surpassed Stockfish — the 2016 TCEC world champion — after 4 hours of training; Elmo, the shogi champion, after under 2 hours; AlphaGo Lee after 8 hours. In final 100-game matches at tournament time controls: 28 wins, 72 draws, 0 losses against Stockfish; 90–8 against Elmo; 60–40 against the previous 3-day AlphaGo Zero.

The most telling statistic isn't the score, it's the search budget. AlphaZero evaluated 80,000 positions per second in chess versus Stockfish's 70 million — nearly a thousand times less search — and won anyway. It compensated with a neural network that focused search far more selectively on promising variations, a more "human-like" allocation of computation that, the paper notes, called into question the long-held belief that alpha-beta search is inherently superior. The authors also measured strength directly as a function of thinking time: AlphaZero's MCTS scaled more effectively with additional compute than either Stockfish's or Elmo's alpha-beta search.

AlphaZero's real result isn't "AI beats chess." It's that a tabula rasa learner, given a perfect simulator and a verifiable outcome, can compress decades of human domain expertise into hours of self-play — searching 1,000× less and still winning.

Limitations worth naming

The asterisks are load-bearing. AlphaZero had a perfect world model (the game rules), a cheap infinite supply of experience (self-play games cost compute, nothing else), and a crisp verifiable reward (win/loss/draw). Remove any one of those and the recipe breaks — which is exactly why two decades of "AlphaZero for X" mostly failed: the real world has no rules engine, no reset button, and no score. Self-play in non-zero-sum or sparse-reward settings can also collapse into degenerate equilibria rather than climb to superhuman skill. And the knowledge it discovered is locked inside network weights — AlphaZero plays beautifully but can't explain a single principle it found, unlike the human grandmasters whose heuristics it replaced.

Where it sits in the RSI arc

AlphaZero is the template every later RSI method is either imitating or trying to escape. The formula — self-play + verifiable reward + iterative policy improvement from the latest checkpoint — reappears in RLVR, in Absolute Zero (which cites it explicitly and replaces game rules with a code executor), and in the self-distillation literature. Its deepest lesson for RSI is about the verifier: self-improvement only compounds when there's a ground-truth signal that can't be gamed, and the history of the field since 2017 is largely the search for verifiers outside board games — math checkers, code executors, learned reward models, the model itself. Every one of those is an attempt to answer the question AlphaZero left open: what plays the role of the rules when there is no game?

Paper: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm — David Silver et al., 2017.