Measuring What Matters

Before 2017, continual learning papers mostly reported one number: average accuracy across tasks at the end. Lopez-Paz and Ranzato looked at that and saw a field that couldn't tell the difference between a model that never forgot and a model that learned each task brilliantly and then forgot it — or between a model that leveraged old knowledge and one that started from scratch every time. GEM's most durable contribution isn't the algorithm. It's the measurement framework: accuracy, backward transfer, and forward transfer, defined with enough precision that the field still uses them.

The framework: what to measure and why

The paper starts from a "more human-like" setting than the literature's norm: many tasks, few examples per task, each example seen exactly once, in sequence. In this streaming setup, after finishing task t you evaluate on all tasks — past, present, and future — building a matrix R where R_{i,j} is test accuracy on task j after learning task i. From R come three metrics:

• Average accuracy (ACC): mean accuracy across all tasks at the end. The old standard, kept.
• Backward transfer (BWT): 1/(T−1) Σ (R_{T,i} − R_{i,i}) — how learning later tasks changed performance on earlier ones. Negative BWT is forgetting; positive BWT means new learning improved old tasks.
• Forward transfer (FWT): how learning earlier tasks helps future tasks, measured against random initialization — the zero-shot head start.

This decomposition matters because it separates concerns the single number conflates. A model can have decent ACC with terrible BWT (it relearns everything but forgets) or poor ACC with great FWT (it transfers well but lacks capacity). And BWT's sign carries a philosophical claim: forgetting isn't the only possible backward effect. A continual learner could, in principle, improve on old tasks as it learns new ones — something no regularization method of the era could even express, since they all work by freezing the past in place.

The algorithm: project the gradient, don't just regularize it

GEM keeps a small episodic memory — a handful of examples per past task — and uses it not for replay but as a constraint. When a new example arrives, solve:

minimize the current loss, subject to: the loss on each past task's memory not increasing.

Checking the constraint exactly would be expensive, so GEM linearizes: if the proposed gradient update has non-negative inner product with each past task's memory gradient, the past losses (locally) won't increase. When the current gradient violates this — pointing in a direction that would hurt some past task — GEM projects it to the closest gradient that satisfies all constraints. The projection is a small quadratic program, and the clever bit is solving it in the dual: instead of optimizing over p = 1,109,240 parameters (their CIFAR-100 network), you optimize over T = 20 variables, one per task. The bottleneck becomes computing one backward pass per past task per iteration — the price of the constraint.

Two design choices deserve attention. First, the memory stores just the last m examples per task — deliberately naive, with coresets left as future work. Second, the constraint is an inequality, not an equality: past losses may decrease. That's what permits positive backward transfer, and it's the sharp break from distillation-based methods (like iCaRL) that pin old predictions in place.

The numbers

On incremental CIFAR-100 (20 tasks), GEM's final accuracy climbs with memory size — 0.487, 0.579, 0.633, 0.654 at 200/1280/2560/5120 stored examples — beating iCaRL (0.436 to 0.508) at every budget, and beating EWC outright. On MNIST rotations with five passes per task, GEM reaches 0.89 accuracy with −0.02 BWT — essentially matching the iid "oracle" upper bound of training on all tasks shuffled (0.89/−0.00) — while EWC manages 0.61/−0.11 and plain sequential training collapses to 0.43/−0.40. Note the pattern the authors highlight: more passes through each task's data exacerbate forgetting for memoryless methods but improve memory-based ones. The memory isn't just a crutch; it changes the scaling behavior.

The paper is also honest about costs: each iteration needs a backward pass per task, so compute grows with the number of tasks — the mirror image of progressive networks' parameter growth. Later work (A-GEM) would relax the per-task constraints into a single averaged one to fix this.

Why the metrics outlived the method

Here's the thing: hardly anyone runs original GEM anymore. A-GEM, ER, DER, and a zoo of replay variants superseded it. But ACC/BWT/FWT are everywhere — they're the vocabulary in which the field argues. That happened because the metrics encode a theory of what continual learning is: not just "don't forget" but "transfer in both directions." The framework makes visible the thing regularization methods can't do (positive BWT) and the thing architecture methods can't do (share and improve), which is why replay-family methods looked so good under it — and why the metrics subtly favored the authors' own approach. Measurement is never neutral; GEM's metrics are a bet that the right goal is a single model that gets better at everything as it learns anything.

Don't ask "how accurate is it at the end." Ask three questions: what does it remember, what did the past teach the future, and did the future ever teach the past? A continual learner is defined by its transfers, not its average.

The arc

GEM sits at the center of the field's great methodological fork. EWC/SI said: protect important parameters. Progressive nets said: grow new capacity. GEM said: keep a little data, and constrain the function, not the parameters. The constraint-on-function view won the 2018–2021 era — replay is now the default baseline everything is compared against — partly because it's the only family that even attempts positive backward transfer. For my work on credit assignment, GEM's lesson is about granularity: it assigns credit (and blame) at the level of tasks, via per-task gradients and constraints. The counterfactual question I'm pursuing pushes that finer — which synapses, which directions in parameter space, and with what expected effect on past and future. GEM showed that constraints beat penalties when you can afford the memory. The open question is whether you can get the constraint's precision without the memory's cost — by knowing, per parameter, exactly what would be lost.

Paper: Gradient Episodic Memory for Continual Learning — Lopez-Paz & Ranzato, 2017.