Catastrophic Interference

Catastrophic interference — also called catastrophic forgetting — is the phenomenon in neural networks where learning new information destroys previously learned information. A network trained to classify handwritten digits, then fine-tuned on street signs, may forget digits entirely. The new task overwrites the weights that encoded the old, not because the network is too small but because gradient descent is greedy: it optimizes for the current objective with no mechanism to preserve past competencies.

The problem is not merely technical; it is a fundamental tension in any learning system that uses shared resources. In continual learning — the sequential acquisition of skills over a lifetime — catastrophic interference is the central obstacle. Biological brains appear to solve it through a combination of synaptic consolidation, complementary learning systems, and memory replay during sleep. Artificial systems have borrowed these ideas: elastic weight consolidation penalizes changes to important weights; memory replay interleaves old and new training data; modular architectures isolate different competencies in different subnetworks. None fully solves the problem.

The deeper systems question is whether interference is a bug or a feature. Catastrophic interference arises from distributed representations — the very property that enables generalization. A system that cannot forget cannot learn, because unlearning is the flip side of updating. The challenge is not to eliminate interference but to make it selective: forgetting the irrelevant while preserving the essential. The brain does this. We do not yet know how.

A promising direction for mitigating catastrophic interference is memory replay — the reactivation of past experiences during new learning, inspired by the consolidation mechanisms observed during sleep in biological brains.