The Spring That Remembers

The mammalian brain has a trick that artificial networks, in 2016, conspicuously lacked. When a mouse learns a new motor skill, a fraction of its excitatory synapses physically enlarge — dendritic spines swell — and those enlarged spines persist for months despite subsequent learning. Erase those specific spines and the skill vanishes; it's causal, not correlational. The brain protects the synapses that matter by making them less plastic. Elastic Weight Consolidation asks: what if we did that to a neural network, and what if "which synapses matter" could be computed from the math of the loss itself?

The mechanism

EWC reframes continual learning as Bayesian inference. Training on task A gives you a posterior over parameters, p(θ|D_A). When task B arrives, that posterior becomes your prior. The true posterior is intractable, so EWC applies the Laplace approximation: treat p(θ|D_A) as a Gaussian centered at the task-A solution θ*_A, with precision given by the diagonal of the Fisher information matrix F. The loss for task B then becomes:

L(θ) = L_B(θ) + Σᵢ (λ/2) Fᵢ (θᵢ − θ*_A,i)²

Read it as physics: each parameter is anchored to its old value by a spring, and the spring's stiffness is Fᵢ — the Fisher, which measures how much the task-A loss would change if you moved that parameter. Parameters that mattered for A get stiff springs; parameters that didn't get loose ones. The optimizer can still find a good solution for B, because over-parameterized networks have many solutions, and the springs just steer it toward one that's also good for A. Uniform L2 regularization toward the old weights (the "green arrow" in the paper's famous schematic) fails because it constrains everything equally — too rigid to learn B, or too loose to remember A. Selectivity is the whole game.

The Fisher is a clever choice for three reasons the paper spells out: near a minimum it equals the second derivative of the loss (so it really measures curvature-based importance), it can be computed from first-order gradients alone, and it's guaranteed positive semi-definite. For a third task C, you just add another quadratic penalty — or fold it into one, since the sum of quadratics is quadratic.

What it proved

Two demonstrations, one supervised and one RL. On sequential permuted-MNIST tasks (each task a fixed random pixel permutation, so all tasks are equally hard but need different solutions), plain SGD catastrophically forgot; EWC maintained accuracy on early tasks while learning later ones. On sequential Atari 2600 games, EWC let a single DQN learn multiple games without the earlier ones collapsing — the first demonstration that a deep RL agent could accumulate games this way.

But the result I keep coming back to isn't a number. It's the paper's framing of why a nearby solution should exist at all: over-parameterization means the set of good solutions for B is large, so one probably lies near θ*_A. EWC doesn't create the overlap between task solutions; it exploits the geometry that over-parameterization provides. That observation quietly predicts a decade of later findings — that wider networks forget less, that the loss landscape's flat directions are where continual learning lives.

Where the springs creak

EWC's limitations are well-mapped, and the paper is honest about the central one: the Laplace approximation is local. The Fisher is computed at θ*_A, the endpoint of task-A training. If the loss landscape between the task-A and task-B solutions is non-quadratic — and in deep networks it always is — the Gaussian approximation degrades, and springs calibrated at the old minimum misjudge importance along the path. This is exactly the gap Synaptic Intelligence would attack months later by integrating importance along the entire trajectory rather than estimating it at the endpoint.

Other issues, mostly from later literature: the diagonal Fisher ignores interactions between parameters; λ (how much the old task matters relative to the new one) is a sensitive hyperparameter with no principled setting; and EWC assumes clean task boundaries — you need to know when A ends to compute F and θ*_A. In the streaming, boundary-free setting, the method needs scaffolding the paper doesn't provide. Empirically, EWC also underperforms replay-based methods on harder benchmarks, a gap that GEM would quantify directly.

The brain's solution to forgetting isn't to remember the data — it's to remember which synapses mattered, and to make those synapses stubborn. EWC is the first algorithm to take that literally: importance-weighted stubbornness, computed from curvature.

The arc: from springs to credit assignment

EWC sits at the hinge of the field's history. Before it: McCloskey and Cohen's diagnosis, interleaving, tiny-scale fixes. After it: an explosion of "which parameters matter" methods — SI's path integral, Memory Aware Synapses' unsupervised importance, Riemannian approaches that drop the diagonal approximation. The paper also cemented the regularization family as one of continual learning's three great strategies, alongside replay and architecture.

For my research direction — counterfactual credit assignment for lifelong networks — EWC is both inspiration and foil. It's inspiration because it frames protection as a credit-assignment problem: figure out which parameters deserve credit for past performance, and spend your plasticity budget accordingly. It's a foil because the Fisher answers "which parameters were important" with a local, backward-looking estimate. The counterfactual question is sharper: which parameters, if changed, would most damage past performance — and which changes would most improve the future? EWC approximates the first with curvature at a point. The trajectory matters, the interactions matter, and the future matters. That's the gap I'm working in.

Paper: Overcoming Catastrophic Forgetting in Neural Networks — Kirkpatrick, Pascanu, Rabinowitz, Veness, Desjardins, Rusu, Milan, Quan, Ramalho, Grabska-Barwinska, Hassabis, Clopath, Kumaran & Hadsell, 2017.