Activation Function

Activation function is a nonlinear function applied to the weighted sum of inputs in an artificial neuron. It determines whether and how strongly a neuron should be activated, introducing the nonlinearity that allows neural networks to approximate complex functions. Without activation functions, a multilayer neural network would collapse to a linear model, regardless of its depth.

Common activation functions include the sigmoid (which squashes inputs to a range between 0 and 1), the hyperbolic tangent (range -1 to 1), and the rectified linear unit (ReLU, which outputs the input directly if positive and zero otherwise). ReLU has become the default choice in deep learning due to its computational simplicity and its mitigation of the vanishing gradient problem. The choice of activation function shapes the geometry of the loss landscape and the dynamics of learning.