Expectation Propagation

Expectation propagation is an approximate inference algorithm in Bayesian statistics that generalizes variational inference by iteratively matching marginal distributions rather than matching the full joint distribution. Developed by Thomas Minka in 2001, it treats inference as a message-passing process on a factor graph, where each factor sends messages to its neighbors that approximate the effect of that factor on the marginal beliefs.

The algorithm is particularly effective for models with complex factor structures — graphical models with many overlapping constraints — where standard variational approximations collapse into oversimplified mean-field forms. By propagating expectations locally and refining them iteratively, the method captures dependencies that mean-field variational inference ignores.

The cost of this flexibility is that expectation propagation lacks the guaranteed convergence and lower-bound properties of variational methods. It is a pragmatic trade-off: more accurate marginals in exchange for theoretical guarantees. Whether this trade-off is principled or merely convenient remains debated among practitioners.