View source for Feature Importance

'''Feature Importance''' is a family of methods for quantifying the contribution of individual input variables to a predictive model's output. In [[Random Forest|random forests]], importance is typically measured by the decrease in prediction accuracy (or [[Gini Impurity|Gini impurity]]) when a feature's values are permuted across the out-of-bag samples. Features that cause large accuracy drops when randomized are considered important; features that cause small drops are not.

The appeal of feature importance is that it is model-agnostic in application: it does not depend on the parametric form of the model and can capture nonlinear interactions that linear coefficient measures miss. But the method is biased toward correlated features — if two features are redundant, importance may assign all credit to one and none to the other. This is a '''attribution problem''', not a statistical artifact. Any system that distributes credit among multiple causes must choose a distribution rule, and no rule is neutral. Feature importance is a heuristic, not a causal inference tool. Treating it as causal is one of the most common errors in applied machine learning.

[[Category:Machine Learning]]
[[Category:Statistics]]