Explainability, often abbreviated XAI (Explainable Artificial Intelligence), refers to methods and practices that make an AI model's decisions or predictions understandable to humans. A model's internal computation may technically produce a correct output but be opaque about why — XAI tries to bridge that gap by attributing predictions to specific inputs, features, or intermediate representations. It is distinct from interpretability, which describes inherently transparent models; XAI often applies post-hoc analysis to black-box models that were not designed to be transparent.

Think of it like a doctor ordering extra tests not because the diagnosis is wrong but because the hospital requires written justification for every prescription. The explanation is required for accountability even when the conclusion is sound.

A widely used XAI technique is SHAP (SHapley Additive exPlanations), which borrows from cooperative game theory to assign each input feature a contribution score. For example, a credit-risk model might produce a SHAP waterfall chart showing that a customer's high debt-to-income ratio contributed +0.34 to the default probability while a long employment history contributed -0.21. In Transformer-based models, Attention visualisation is another popular but contested approach — highlighting which tokens the model attended to most when producing an output.

XAI connects closely to Hallucination (understanding why a model confabulates), RLHF (where human feedback implicitly encodes what kinds of reasoning are acceptable), and Prompt Engineering (carefully structured prompts can force the model to externalise its reasoning via Chain of Thought, which is itself a form of self-explanation). It also intersects with Overfitting diagnosis, where explainability tools reveal whether a model is relying on spurious correlations.

You will encounter XAI requirements most in regulated industries — financial services (the EU AI Act and US fair lending laws require adverse-action notices), healthcare (clinicians need to understand model recommendations), and criminal justice — and in any internal ML system where engineers need to debug model failures.