AI Model Wiki

    Skip to main content
    20 min
    Lesson 3 of 4

    Risk Monitoring and Reporting

    From the course AI Risk Management

    Establish ongoing monitoring programs and reporting cadences for AI risk management.

    Monitoring Program

    Automated Risk Indicators

    class AIRiskMonitor:
        thresholds = {
            "accuracy_degradation": 0.05,  # 5% drop triggers alert
            "bias_disparate_impact": 0.8,   # Below 0.8 is high risk
            "error_rate": 0.001,             # 0.1% error rate threshold
            "data_drift_score": 0.1          # KS statistic threshold
        }
        
        def check_risks(self, model_id, metrics):
            alerts = []
            
            if metrics["accuracy"] < self.baseline[model_id] - self.thresholds["accuracy_degradation"]:
                alerts.append({"type": "ACCURACY_DRIFT", "severity": "HIGH"})
            
            if metrics["disparate_impact"] < self.thresholds["bias_disparate_impact"]:
                alerts.append({"type": "BIAS_DETECTED", "severity": "CRITICAL"})
            
            if metrics["drift_score"] > self.thresholds["data_drift_score"]:
                alerts.append({"type": "DATA_DRIFT", "severity": "MEDIUM"})
            
            return alerts
    

    Risk Reporting

    Risk Register Template

    Risk IDDescriptionLikelihoodImpactScoreOwnerControlsStatus
    AI-001Model bias in lending3515Head of RiskBias audit, human reviewActive
    AI-002Data poisoning248CISOInput validation, training data scanningActive

    Board-Level Reporting

    • Quarterly AI risk dashboard for executive team
    • Annual AI risk report for board presentation
    • Incident reports for material AI failures
    • Regulatory filing support for compliance matters

    Key Risk Indicators (KRIs)

    • Model accuracy vs baseline (weekly)
    • Bias audit results (quarterly)
    • AI incident count and severity (monthly)
    • Regulatory inquiry count (quarterly)
    • Staff AI ethics training completion (quarterly)

    Key Takeaways

    • •Automated monitoring with threshold-based alerting catches risks before they escalate
    • •Risk registers provide structured tracking of identified risks and their controls
    • •Board-level reporting translates technical risks into business impact language
    • •Key Risk Indicators provide leading indicators of emerging AI risk
    0 of 4 lessons marked complete

    Progress is kept in this browser only.