Elastic Weight Consolidation

Elastic Weight Consolidation (EWC) is a regularization method for continual learning that protects previously learned knowledge by estimating the importance of each neural network parameter for past tasks and penalizing changes to important parameters during new learning. The importance measure is derived from the diagonal of the Fisher information matrix, which approximates how much each parameter contributed to the likelihood of the previously learned task data. EWC treats continual learning as a Bayesian updating problem: the posterior after learning task A becomes the prior before learning task B, and the Fisher information serves as a quadratic approximation of that prior's curvature.

The biological analogue is synaptic consolidation — the molecular strengthening of synapses that makes them resistant to subsequent modification. EWC is artificial synaptic consolidation implemented in gradient descent. The limitation is that the Fisher diagonal is a local approximation that ignores parameter interactions, and EWC accumulates penalties across many tasks until the network becomes effectively frozen.