Fairness assessment is the most common focus of AI audits. It requires understanding different fairness definitions and measurement approaches.

Fairness Metrics

No single metric captures all aspects of fairness. Key metrics include:

• Demographic Parity — positive outcomes at equal rates across groups
• Equal Opportunity — equal true positive rates across groups
• Equalized Odds — equal true positive AND false positive rates
• Calibration — predicted probabilities match actual outcomes within each group
• Individual Fairness — similar individuals receive similar predictions
• Counterfactual Fairness — prediction wouldn't change if protected attribute were different

The Impossibility Theorem

A critical insight: most fairness metrics are mathematically incompatible. You cannot simultaneously satisfy demographic parity, calibration, and equal opportunity (except in trivial cases). This means:

• You must choose which fairness definition is most appropriate for the context
• The choice depends on the specific use case, stakeholders, and potential harms
• There is no universally "correct" fairness metric

Conducting a Bias Assessment

Step-by-step process:

1. Define Protected Groups — identify demographics relevant to the use case
2. Collect Disaggregated Data — ensure sufficient data for each group
3. Choose Metrics — select fairness metrics appropriate for the context
4. Measure Baselines — calculate metrics across all protected groups
5. Intersectional Analysis — check combinations (e.g., young Black women)
6. Root Cause Analysis — identify sources of measured bias
7. Mitigation — implement pre-processing, in-processing, or post-processing corrections
8. Monitor — continuously track fairness metrics in production

Bias Mitigation Techniques

• Pre-Processing — reweight or resample training data to balance representation
• In-Processing — add fairness constraints to the model's objective function
• Post-Processing — adjust model outputs to equalize metrics across groups
• Causal Methods — model causal pathways to directly address discrimination mechanisms

Tools for Fairness Assessment

• Fairlearn (Microsoft) — fairness metrics and mitigation algorithms
• AI Fairness 360 (IBM) — comprehensive bias detection and mitigation toolkit
• What-If Tool (Google) — interactive fairness exploration interface
• Aequitas — open-source bias audit toolkit for classification models