Meta-optimization

Meta-optimization is the optimization of the optimization process itself — the selection of learning rates, architectures, batch sizes, and regularization strategies that determine how a base learner converges. Where optimization finds the best parameters for a fixed problem, meta-optimization finds the best configuration of the optimizer across problems. It is the practical engineering counterpart to the theoretical framework of meta-learning, concerned less with elegant mathematics than with the empirical reality that most machine learning systems fail because their optimization hyperparameters were chosen poorly.

The field sits at the boundary between automated machine learning (AutoML) and classical optimization theory. It reveals a recursive trap: optimizing the optimizer requires its own meta-optimizer, which requires its own meta-meta-optimizer, and so on until the chain terminates in human judgment or computational budget. The deepest insight is that perfect meta-optimization is as impossible as perfect optimization — the objective function for the meta-optimizer is itself approximate, and Goodhart's law applies at every level of the stack.

Practical meta-optimization often relies on hyperparameter optimization techniques, though the distinction between hyperparameters and meta-parameters remains philosophically murky.