Loss Landscape

Loss landscape is the geometric surface defined by a neural network's loss function as a function of its parameters. Each point in the high-dimensional parameter space corresponds to a specific configuration of weights, and the height of the landscape at that point represents the error on the training data. The goal of training is to navigate this landscape to find a global minimum—a parameter configuration that minimizes loss.

The geometry of the loss landscape is a central research problem in deep learning. Despite the non-convexity and high dimensionality of these landscapes, gradient descent reliably finds solutions that generalize well. Theoretical work suggests that overparameterized networks have landscapes that are effectively convex near initialization, with most local minima being close to global minima in loss value. The loss landscape thus connects neural network training to the physics of disordered systems and phase transitions.