Explainable AI (XAI):
Opening the Black Box of Machine Learning

A model that denies someone a loan, flags a transaction as fraud, or recommends a medical treatment isn’t very useful if nobody — including the people building it — can explain why it made that call. Explainable AI (XAI) is the set of techniques developed to answer that “why,” and it has become essential wherever machine learning decisions affect real people.

Why explainability matters

• Trust — people are more willing to accept a decision they can understand, even when they disagree with it.
• Debugging — explanations often reveal when a model is picking up on the wrong signal, like a spurious correlation instead of a genuine pattern.
• Regulatory requirements — sectors like finance and healthcare increasingly require some form of explanation for automated decisions.
• Fairness auditing — you can’t check whether a model is discriminating against a group without understanding what’s driving its predictions.

Interpretable models vs. post-hoc explanation

There are two fundamentally different strategies here. The first is to use a model that’s inherently interpretable — a decision tree, a linear regression, a rule list — where you can read the logic directly off the model itself. The second, needed for complex models like deep neural networks or gradient-boosted trees, is post-hoc explanation: treating the model as a black box and using separate techniques to approximate why it behaved a certain way for a given input.

A feature importance chart is one of the most common outputs of a post-hoc explanation method. (Image by author)

The two workhorse techniques

SHAP (SHapley Additive exPlanations) borrows a concept from cooperative game theory — Shapley values — to fairly distribute “credit” for a prediction among all the input features. For a given prediction, SHAP tells you exactly how much each feature pushed the output up or down relative to a baseline, and the underlying math guarantees these contributions add up consistently. It’s computationally expensive in its exact form, but efficient approximations exist for tree-based models and neural networks.

LIME (Local Interpretable Model-agnostic Explanations) takes a different approach: it perturbs the input slightly, observes how the model’s output changes, and fits a simple, interpretable model (usually linear) to approximate the black box’s behavior in that local neighborhood. The explanation is only valid near that specific input — LIME doesn’t claim to explain the model globally, only its local behavior around one prediction.

Global vs. local explanations

It’s worth distinguishing between explaining a single prediction (local explanation — “why did the model deny this specific application?”) and explaining the model’s overall behavior (global explanation — “which features matter most across all predictions?”). SHAP and LIME are both primarily local techniques, though SHAP values can be aggregated across many predictions to build a global picture, which is one reason it’s become the more widely adopted of the two.

The limits of explainability

Explanations from post-hoc methods are approximations, not ground truth — they describe correlational relationships the model has learned, not necessarily causal ones, and two different explanation methods can disagree with each other on the same prediction. There’s also a well-documented risk of explanations being used to build unwarranted confidence in a model rather than genuine understanding — a plausible-looking explanation doesn’t guarantee the model’s reasoning is sound. Explainability tools are aids for human judgment, not a replacement for it.

Conclusion

As machine learning models take on more consequential decisions, understanding why they make the calls they do has moved from a nice-to-have to a practical necessity. Whether through inherently interpretable models or post-hoc tools like SHAP and LIME, explainability gives practitioners, regulators, and the people affected by these decisions a way to look inside the black box — imperfectly, but usefully.