Grow the Brain, Freeze the Past

Continual pre-training has a scaling problem, and it's not the one you're thinking of. The field spent years scaling parameters — bigger models, more data — while the structure of continual learning stayed naive: keep training the same weights on new data and hope regularization or replay holds the past together. SCALE's provocation, stated plainly in the abstract, is that progress now depends more on scaling the right structure than on scaling parameters alone. Their structure: make the model wider, freeze everything it already knows, and train only the new capacity.

The mechanism: grow wider, freeze the past

SCALE (from Jin-woo Lee and colleagues) is a width-upscaling architecture for continual pre-training. When new knowledge needs to be absorbed — say, continual pre-training on a Korean corpus starting from an English-trained model — the method inserts lightweight expansion modules into the model's linear layers while freezing all pre-trained parameters. The residual and attention topologies are preserved; capacity increases without perturbing the base model's original function. Two principles govern the design:

• Persistent Preservation. Preservation-oriented initialization plus freezing of pretrained weights keeps the base model's behavior intact. The paper doesn't just assert this — it provides analysis clarifying when preservation provably holds, which is rare air in a literature full of empirical claims. Freezing is the strongest possible anti-forgetting guarantee: a frozen parameter cannot forget, because forgetting is a change and frozen things don't change.
• Collaborative Adaptation. A subset of the expansion components is selectively trained to acquire new knowledge with minimal interference. The "collaborative" part matters: the new capacity isn't a separate column (as in progressive networks) but integrated into the existing computation, sharing a cross-domain matrix that decouples shared from domain-specific knowledge.

Three variants instantiate the stability-plasticity trade-off explicitly: SCALE-Preserve (preservation-first), SCALE-Adapt (adaptation-first, training only the new domain knowledge), and SCALE-Route, an optional extension doing token-level routing between preservation and adaptation heads — a mixture-of-experts flavor where each token decides whether it needs the old knowledge or the new.

The experiments: synthetic control and a real language

The evaluation is two-tiered, and the first tier is the more interesting one. On a controlled synthetic biography benchmark — fictional people with invented attributes, where memorization can be measured exactly — SCALE mitigates the severe forgetting seen with depth expansion while still acquiring the new knowledge. The depth-vs-width contrast is doing theoretical work here: adding layers perturbs the function compositionally (every downstream layer sees shifted inputs), while widening with frozen originals preserves the existing computation paths. It's an architectural fact with the force of a theorem, and the synthetic benchmark makes it measurable.

Then the real test: continual pre-training on a Korean corpus. SCALE variants show less forgetting on English evaluations and competitive gains on Korean benchmarks — the best overall stability-plasticity trade-off among the compared methods. The analysis goes further, characterizing when preservation provably holds and why the preservation-adaptation interplay stabilizes optimization relative to standard continual setups.

Why width, why now

The depth-vs-width finding deserves emphasis because it cuts against a lot of folk practice. When practitioners want more capacity for continual learning, the instinct is often to add layers or adapters sequentially — deeper. SCALE's evidence says that's the worse direction: depth expansion composes new transformations after old ones, so errors and distribution shifts propagate; width expansion alongside frozen originals leaves existing paths untouched. This is progressive networks' column idea, but rotated 90 degrees: instead of growing a new network per task (linear cost, no sharing), grow the same network wider per knowledge domain, with the frozen original as shared substrate. It's the compression that progressive nets' appendix called for, implemented as architecture.

The token-level routing in SCALE-Route is the forward-looking piece. Static preservation/adaptation splits assume you know which knowledge a token needs; routing learns it. That moves the method toward the boundary-free setting — the model itself decides, per token, whether to draw on preserved or adapted capacity — which is where real deployment lives.

Limitations and open questions

Width expansion still expands: parameters grow with each continual phase, so this is a slower version of progressive nets' growth problem, not an escape from it. The "provable preservation" holds under conditions the analysis specifies — and conditions are where guarantees go to be violated in practice (distribution shift in the frozen model's operating regime, for instance, can degrade behavior without any parameter changing). The Korean/English experiment is a clean two-domain case; the many-domain, long-horizon setting — dozens of continual phases — is where growth costs compound and where the method's real test lies. And there's a subtle question about the frozen substrate going stale: freezing prevents forgetting, but it also prevents improvement of old knowledge from new experience. Positive backward transfer — GEM's great unfulfilled promise — is structurally impossible here, just as in progressive nets.

The deepest open question is the routing one. SCALE-Route's token-level decisions are learned, which means they can be wrong, and wrong routing in a frozen/adapted split fails silently — the model confidently uses the wrong knowledge store. Understanding and supervising those routing decisions is the difference between a clever architecture and a reliable system.

Don't scale parameters and hope structure follows. Scale the structure: widen alongside the frozen past, prove what preservation guarantees, and let each token route between what was known and what's being learned.

The arc

SCALE is where the architectural family of continual learning has arrived in the LLM era — and it's striking how directly it descends from 2016. Progressive networks: freeze old columns, grow new ones. SCALE: freeze the pretrained model, grow it wider. The differences — shared substrate instead of separate columns, width instead of columns, provable preservation analysis, token-level routing — are a decade of engineering progress. But the core move is identical, and it embodies the field's most durable lesson: the strongest anti-forgetting guarantee isn't a penalty or a memory buffer, it's not changing the parameters. Everything else — EWC's springs, SI's ledgers, GEM's constraints, RECALL's merging — is for the cases where you can't afford to freeze. SCALE's provocation is that, with width upscaling, maybe you can afford it more often than you think. For my research, the routing variant is the tantalizing piece: per-token decisions about which knowledge store to use are a form of online credit assignment — the model learning, as it runs, where each piece of its knowledge lives.

Paper: SCALE: Upscaled Continual Learning of Large Language Models — Jin-woo Lee, Junhwa Choi, Bongkyu Hwang, Jinho Choo, Bogun Kim, JeongSeon Yi, Joonseok Lee, DongYoung Jung, Jaeseon Park, Kyoungwon Park & Suk-hoon Jung, 2025.