Forget Less by Training Less

The numbers in this paper stopped me cold. Continually fine-tune a language model on a stream of new facts, and its accuracy on previously learned facts collapses: an 89% F1 drop with full fine-tuning, 71% with LoRA. Then do the same training but update only a sparse subset of the model's memory layers, and the drop is 11% — while acquiring the new knowledge just as well. Same data, same objective, same base model. The only difference is where the gradient is allowed to land. Forgetting, this paper argues, is largely a targeting problem.

The mechanism: train the memory, spare the rest

The work builds on memory layers (Berges et al., 2024) — transformer layers augmented with large key-value memories where factual knowledge tends to localize. The authors' method, sparse memory finetuning, works in two phases:

1. Score the slots. Given the new training data, score each memory slot by TF-IDF: how distinctive is this slot's content relative to the incoming facts? Slots whose stored knowledge is most relevant to (or most entangled with) the new information get the highest scores. This is cheap, interpretable, and — crucially — computed from the data, not from gradients.
2. Finetune sparsely. Update only the top-scoring slots; freeze everything else.

The intuition is almost embarrassingly simple once stated: catastrophic forgetting happens because gradient updates to shared parameters overwrite old knowledge as a side effect of fitting new data. If new facts only need to modify the memory slots that actually store related knowledge, then updating only those slots confines the blast radius. Everything unrelated — the vast majority of the model's knowledge — is never touched, so it can't be forgotten. It's EWC's "protect important parameters" idea, but inverted: instead of estimating importance and penalizing movement, just don't move anything except the slots the new data actually implicates.

The experiments

Two continual QA settings, both designed to look like real deployment:

• Stream of facts: the model must absorb new factual information over time — the knowledge-updating scenario.
• Stream of documents: the model consolidates knowledge from incoming documents — the messier, more realistic variant.

Across both, the pattern holds: full finetuning destroys old knowledge (89% F1 drop on Natural Questions), LoRA — the supposed safe default — still destroys most of it (71%), and sparse memory finetuning holds the line at 11%, with no penalty on new-knowledge acquisition. Read that LoRA number again. The community's standard "safe" fine-tuning method, the one everyone reaches for to avoid breaking models, still forgets most of what the model knew. Low-rank updates are parameter-efficient, but they're not knowledge-efficient: a low-rank change to attention weights still perturbs the shared computation that every fact flows through. Sparsity in the right substrate beats low-rankness in the wrong one.

The authors are from FAIR at Meta and UC Berkeley (Lin, Zettlemoyer, and collaborators), and the work has a distinctly systems-aware flavor: the method is cheap, the scoring is interpretable, and the result is a practical recipe rather than a theoretical statement.

Why it's surprising

Three surprises, in increasing order of importance. First, that TF-IDF — a decades-old information retrieval heuristic — is sufficient as the targeting mechanism. No learned router, no gradient-based importance, no Fisher information. The signal for "which memory to update" was already in the data distribution; the field's elaborate importance-estimation machinery (EWC's Fisher, SI's path integral) may be overkill when the architecture localizes knowledge in addressable slots.

Second, that LoRA does so badly. This should update everyone's priors about what PEFT methods actually guarantee. They guarantee compute and memory efficiency. They guarantee nothing about forgetting — and this paper puts a number on it: 71%.

Third, and most theoretically interesting: the result suggests forgetting is not an inevitable consequence of the stability-plasticity dilemma in the strong sense. The dilemma says you can't learn new things without changing the substrate, and changing the substrate disturbs old things. Sparse memory finetuning doesn't resolve the dilemma — it routes around it by making the substrate modular enough that "changing" and "disturbing" decouple. The 11% residual is the honest remainder: some interference is genuinely unavoidable when new and old knowledge share slots. But most of what we call catastrophic forgetting was just sloppy targeting.

Forgetting is a targeting problem. When knowledge lives in addressable slots and updates land only where the new data points, 89% forgetting becomes 11% — not through a cleverer penalty, but through a narrower blast radius.

Limitations and open questions

The method leans hard on memory layers: it works because factual knowledge localizes in key-value memories. For capabilities that live in the computation — reasoning patterns, style, multi-step planning — there's no slot to target, and it's unclear the trick transfers. The TF-IDF scoring is a heuristic; adversarial or highly entangled knowledge streams might defeat it, and the paper doesn't stress-test that regime. Task boundaries are assumed (you score slots per incoming batch), so the streaming, boundary-free setting still needs work. And 11% isn't 0% — the residual forgetting, concentrated where new and old knowledge genuinely collide in the same slots, is the hard core of the problem, and it's exactly where importance-weighting methods like EWC/SI still have something to say.

The biggest open question is composition with the rest of the CL toolkit. Sparse updates solve the where; they don't solve the what (which knowledge to keep — a memory management question) or the when (consolidation over long timescales). A system that routes updates to the right slots, replays the right memories, and consolidates on the right schedule would be the full picture. This paper nails one piece.

The arc

This is the paper in the series that most directly vindicates the classical intuitions while overturning the classical methods. McCloskey and Cohen said interference happens when new learning alters the weights involved in old learning; sparse memory finetuning takes that literally and simply doesn't alter those weights. EWC and SI estimated importance to penalize movement; this paper observes that with the right architecture, you can skip the estimation and just not move. And it reframes the LoRA era: parameter efficiency was never the right objective — knowledge efficiency was. For my credit-assignment work, the lesson is sharp: the cheapest importance signal is often in the data, not the gradients. The counterfactual question — "which parameters, if changed, would damage the past?" — might have a surprisingly simple answer when the architecture makes knowledge addressable.

Paper: Continual Learning via Sparse Memory Finetuning — Lin, Zettlemoyer et al., 2025.