22 min
Lesson 2 of 4
Risk Assessment and Mitigation
From the course AI Risk Management
Conduct systematic risk assessments and implement effective mitigation strategies.
Risk Assessment Process
Pre-Deployment Assessment
Step 1: System Characterization
- What decisions does the AI make?
- Who and how many people are affected?
- What data is used?
- What are the failure modes?
Step 2: Risk Identification
- Brainstorm failure scenarios
- Review similar AI incident cases
- Consult affected stakeholders
- Red team exercises
Step 3: Risk Analysis
- Score each risk (Likelihood × Impact)
- Map risk relationships
- Identify root causes
Step 4: Risk Evaluation
- Compare against risk appetite
- Prioritize high-risk items
- Determine acceptable residual risk
Step 5: Risk Treatment
- Avoid, reduce, transfer, or accept
- Design specific controls
- Assign ownership
Bias Risk Assessment
from sklearn.metrics import confusion_matrix
import numpy as np
def assess_bias_risk(predictions, actuals, protected_attribute):
groups = np.unique(protected_attribute)
metrics = {}
for group in groups:
mask = protected_attribute == group
tp = np.sum((predictions[mask] == 1) & (actuals[mask] == 1))
fp = np.sum((predictions[mask] == 1) & (actuals[mask] == 0))
fn = np.sum((predictions[mask] == 0) & (actuals[mask] == 1))
tn = np.sum((predictions[mask] == 0) & (actuals[mask] == 0))
metrics[group] = {
"positive_rate": (tp + fp) / len(predictions[mask]),
"true_positive_rate": tp / (tp + fn) if (tp + fn) > 0 else 0,
"false_positive_rate": fp / (fp + tn) if (fp + tn) > 0 else 0
}
# Calculate disparate impact
rates = [m["positive_rate"] for m in metrics.values()]
disparate_impact = min(rates) / max(rates)
# Flag if < 0.8 (80% rule)
risk_level = "HIGH" if disparate_impact < 0.8 else "LOW"
return {"disparate_impact": disparate_impact, "risk_level": risk_level}
Mitigation Strategies
Technical Controls
- Regular model retraining to address drift
- Robust input validation to prevent adversarial attacks
- Ensemble methods to improve reliability
- Uncertainty quantification to know when AI is uncertain
- Canary deployments for gradual rollout
Organizational Controls
- Human-in-the-loop for high-stakes decisions
- AI ethics review board approval
- Regular third-party audits
- Incident response planning
- AI-specific insurance coverage
Key Takeaways
- •Systematic 5-step assessment covers characterization, identification, analysis, evaluation, and treatment
- •Disparate impact testing identifies bias risk — below 0.8 ratio triggers high-risk flag
- •Technical controls include drift monitoring, input validation, and canary deployments
- •Organizational controls with human oversight manage risks that technology cannot fully address
0 of 4 lessons marked complete
Progress is kept in this browser only.