Algorithmic Auditing

Algorithmic auditing is the set of methods and practices for evaluating the behavior, outputs, and impacts of automated decision-making systems — particularly their fairness, accuracy, and conformance with stated specifications. Unlike traditional software auditing, algorithmic auditing must address statistical behavior across populations rather than correctness of individual computations. The difficulty is structural: an audit conducted on aggregate performance metrics may conceal severe errors for specific subpopulations; an audit conducted on subpopulation metrics must contend with the fact that the relevant subpopulations are often not defined in advance and may not be captured by the data available to auditors. The further complication is that external auditors are typically denied access to the training data, model architecture, and deployment context that would be necessary for a rigorous audit — vendors treat these as proprietary. What is called 'algorithmic auditing' in regulatory contexts is usually black-box testing: submitting test inputs and observing outputs, without access to the system's internals. This is sufficient to detect gross performance disparities. It is insufficient to detect distributional shift failures, which appear only in deployment against real populations. The combination of proprietary opacity and black-box-only access makes algorithmic auditing an accountability theater rather than an accountability mechanism in most current deployments.