AI Model Wiki

    Skip to main content
    15 min
    Lesson 6 of 6

    Common Pitfalls & How to Avoid Them

    From the course AI for Compliance Officers

    Compliance officers deploying AI tools encounter predictable failure modes. Understanding them in advance prevents costly missteps.

    Pitfall 1: Treating AI Output as a Final Decision

    What happens: A team configures AI to auto-close low-risk alerts or auto-approve low-risk third parties without human review. A genuine risk case scores just below the threshold and is missed.

    Why it matters: Regulators hold the compliance function — not the AI vendor — responsible for missed violations. "The algorithm said it was fine" is not a defence in an enforcement action.

    How to avoid it:

    • Maintain human review for any AI output that feeds a compliance decision
    • Set AI thresholds conservatively: catch more rather than less, and tune over time
    • Document review steps so you can demonstrate human oversight during examinations
    • Build exception escalation paths for edge cases near threshold boundaries

    Pitfall 2: Deploying AI on Dirty Data

    What happens: AI risk models trained or run on incomplete, inconsistent, or stale data produce unreliable scores. Analysts lose trust in outputs and revert to manual methods, eliminating the efficiency gain.

    Why it matters: Garbage in, garbage out applies more acutely to AI than to manual processes, because AI amplifies patterns at scale. A data quality problem that a human would notice and question gets embedded in thousands of automated decisions.

    How to avoid it:

    • Conduct a data quality audit before selecting AI tools, not after
    • Establish data governance for the specific fields your AI models rely on (customer data, transaction attributes, counterparty identifiers)
    • Build data quality checks into your AI pipeline and alert on quality degradation
    • Do not migrate to AI-assisted workflows until your underlying data is clean enough to support them

    Pitfall 3: No Model Governance Framework

    What happens: An AI tool is deployed, works well initially, but degrades as the underlying data distribution shifts (new products, new geographies, changed customer behaviour). Nobody notices because there is no monitoring in place.

    Why it matters: Regulators increasingly expect firms to validate AI models used in compliance decisions, document their limitations, and demonstrate ongoing monitoring. Absence of model governance is itself a finding in examinations.

    How to avoid it:

    • Document every AI model in use: vendor, version, training data vintage, intended use case, known limitations
    • Establish a periodic review cadence (quarterly minimum for high-stakes models)
    • Define performance metrics you will track: precision, recall, false-positive rate, coverage
    • Maintain a model inventory as a living document, not a one-time deliverable

    Pitfall 4: Over-Automation of Vendor Due Diligence

    What happens: AI vendor screening passes a third party based on public-data checks. The vendor's actual risk — known to industry contacts but not reflected in databases — is missed.

    Why it matters: Third-party risk is one of the most active areas of regulatory enforcement. Automated screening can miss reputational and conduct risks that human judgement and industry knowledge would catch.

    How to avoid it:

    • Use AI screening as a first pass, not a complete due diligence substitute
    • Tier your vendor population: reserve enhanced human due diligence for high-risk or high-value relationships
    • Supplement database screening with analyst review for vendors with any flags
    • Periodically test your screening tool against known-bad cases to verify coverage

    Pitfall 5: Ignoring Change Management

    What happens: A capable AI tool is deployed but compliance analysts do not trust it, do not understand its outputs, or feel their jobs are threatened. Adoption stalls and the tool is used inconsistently.

    Why it matters: Technology that is not used does not deliver ROI and may create audit evidence problems if the team is supposed to be using it but isn't.

    How to avoid it:

    • Involve analysts in tool selection and piloting — their feedback improves configuration and builds ownership
    • Communicate clearly that AI handles volume so analysts can focus on higher-value work
    • Provide training on how each tool works, what it can and cannot detect, and how to interpret outputs
    • Celebrate early wins publicly to build momentum and confidence

    Key Takeaways

    • •Never use AI output as a final compliance decision — document human review for regulator scrutiny
    • •Data quality audits must precede AI deployment; dirty data amplifies errors at scale
    • •Model governance frameworks (inventory, metrics, periodic review) are now a regulatory expectation
    • •Change management and analyst buy-in determine whether a capable tool actually delivers ROI
    0 of 6 lessons marked complete

    Progress is kept in this browser only.