Neural Tangent Kernel

The neural tangent kernel (NTK) is a kernel function, introduced by Jacot, Gabriel, and Hongler in 2018, that describes the training dynamics of infinitely wide neural networks trained by gradient descent. In the infinite-width limit, a neural network behaves like a linear model in the function space defined by the NTK: the network's predictions evolve as a linear function of the initial residuals, with the kernel determining the rate at which different directions in function space are learned. The NTK is constant throughout training in the infinite-width limit, which makes the dynamics analytically tractable — the training loss decreases exponentially, and the final learned function is a kernel regression solution.

The NTK regime is theoretically elegant and empirically irrelevant. Finite-width networks — the ones that actually exist and actually work — operate far outside the NTK regime. Feature learning, which is the mechanism by which neural networks identify useful representations, requires that the network's kernel change during training — exactly what the NTK theory prohibits. The empirical success of neural networks is not explained by NTK theory; it is explained by finite-width effects that the infinite-width limit suppresses. The NTK is a rigorous theory of networks that no one builds.

This is a productive failure. The NTK makes precise what a neural network would do if it did not learn features, which clarifies, by contrast, what feature learning actually is. The gap between NTK predictions and empirical behavior is a precise measure of how much feature learning matters — and it matters enormously. See SGD's implicit regularization and representation learning for the dynamics the NTK theory leaves out.