Explainable AI: explained

Imagine your mortgage application is rejected. The first question you would probably ask is: why? A mortgage lender can explain why you do not meet the requirements for a mortgage. But what if a machine learning model makes that decision? We will not settle for “computer says no”. The lack of an explanation is a problem for several people involved. If the decision is about you, you need to understand the reasoning behind it. You also want to make sure the decision is not based on unintentional discrimination: someone must not be rejected because of their ethnicity. For the data scientists and data engineers training a model, understanding its behaviour is essential to spotting errors or weaknesses. eXplainable AI (XAI) can help improve a model. We increasingly encounter automated systems that use machine learning algorithms. They are used by government organisations, the financial sector and healthcare providers, among others. And there is good reason for this. In many cases, these algorithms perform well, sometimes even better than a person. But can we blindly trust a model with an accuracy of 90% or higher, or should we ask how the algorithm reached its decision? Research into XAI has increased significantly in recent years. Ethics in artificial intelligence has also received more attention. That may be because AI is being used on an ever larger scale and more often for important decisions, but also because algorithms are becoming more complex. Too complex to understand. So is XAI the answer?

What is explainability?

Let’s start by defining explainability . Unfortunately, there is not (yet) a single agreed definition. The literature usually uses this definition: “the degree to which a human can understand the cause of a decision” [1]. Here, the decision is made by a machine learning model or algorithm. We can generally distinguish between two types of explainability:

Ante-hoc explainability

Explainability by design. In this approach, you develop a model that is transparent by nature. Think of a simple decision tree or a simple linear model. These models are easy for people to understand and therefore ‘explainable’: we can understand how they reach a decision.

Post-hoc explainability

But what if a model is not transparent? Such a model is called a ‘black box’: only its input and output are visible, and we do not know what happens inside. In this case, you can use post-hoc explainability: we generate an explanation for an existing black box model that is not itself ‘explainable’. This often involves using another algorithm to explain the original black-box algorithm and, in effect, open it up.

Which approach is preferable?

A simple model that is already explainable is usually preferable to a complex neural network that still needs to be explained using post-hoc XAI techniques. However, there are many situations where a simpler model is not sufficient and simply cannot achieve the performance of a more complex model. That creates a trade-off between accuracy and explainability: you have to weigh a model’s performance against how easy it is to explain. As the figure below shows, greater explainability often comes at the expense of a model’s accuracy.

A major advantage of post-hoc methods is that you do not have to sacrifice performance: we keep the model as it is and add a technique afterwards to generate an explanation. This makes it possible to use a black-box neural network for a prediction task and explain its decisions afterwards without compromising its power or performance. Another advantage is that the technique can be applied to existing models of different kinds. This is why discussions of XAI often focus on post-hoc explainability.

Why is XAI important?

Data scientists and data engineers need to understand what a model does so they can debug it and address any unintended discrimination. Before a model goes into production, people need to trust its predictions. But they are unlikely to trust an outcome they do not understand. A lack of explainability may mean that end users do not use the model. A model that can explain its outcomes is transparent, which helps build trust in it.

EU privacy legislation also needs to be taken into account. The requirements of the General Data Protection Regulation make it harder to put a model into practice if it cannot be explained. The European Commission has gone a step further with its recent proposal for an AI regulation: in April 2021, it announced that we can expect stricter rules for high-risk AI applications.

How do we approach this?

There are already many methods for explaining machine learning models, and the number is growing considerably. We can categorise these methods by their explanation scope. Global XAI methods try to explain the behaviour of the entire model, while local XAI methods focus on explaining an individual decision. A global explanation is particularly valuable to a data scientist; a local explanation is useful to the person affected by a decision.

The two best-known examples of local (and post-hoc) methods are LIME (Local Interpretable Model-agnostic Explanations)[2] and SHAP (SHapley Additive exPlanations)[3]. These algorithms differ somewhat, but the idea is the same: they change parts - or features - of the input data and examine how much each change influences the final decision.

For a mortgage application, the applicant’s income can be changed, as can whether or not they have a debt. For text classification, some words are replaced or removed, and the new outcome is then determined. For an image, it is divided into super-pixels: groups of pixels with similar colours and brightness. The result is a set of scores showing how important each feature is to the decision.

What a LIME or SHAP explanation looks like depends on these features, which in turn depend on the type of data. For tabular data, the features and their scores can be shown in a chart:

A fictional example of a LIME explanation for a prediction of whether a mortgage applicant has a low or high risk of default. Red indicates features that contribute to the outcome ‘high risk of default’; green indicates ‘low risk of default’.

For recognising dogs and cats in images, the explanation consists of super-pixels: Why is LIME so popular? Its software is accessible and applying it is not too complicated. More importantly, it treats every model as a black box it cannot access, which means it can be applied to any model. SHAP could also be an option. It produces a more robust result, but takes longer to compute. This differs from model-specific XAI methods, which can only be applied to a particular type of model and cannot be implemented for just any model.

What next?

We have a model whose decisions can be explained, but what comes next? Can we assume that LIME’s explanation is correct? The answer is no. Just as a predictive model needs to be evaluated, so does an explanation. The explanations generated by LIME may look plausible to us without actually reflecting the model’s behaviour. There is also a reasonable chance that another XAI method, such as SHAP, will give a different explanation from LIME. Evaluating explanations is a field of study in its own right, and much research is still being done on it.

Even so, it is worth paying attention to a model’s explainability, especially when it is used to make high-risk decisions with implications for society. Think of systems in healthcare, government or education. These sectors need a higher degree of explainability than systems such as spam filters, chatbots or recommendation systems.

Conclusion

As algorithms become more complex, it becomes harder to explain and understand exactly what they do. Yet it is important to understand how a model reached a decision and which features played a part. Explainability matters not only to data scientists and data engineers, but also to end users, domain experts and everyone else involved. It can build trust in a system, help uncover errors and weaknesses, and prevent unfair outcomes. Not every algorithm needs the same degree of explainability, though. An explanation is essential for a medical diagnosis, but less necessary for a personalised Netflix recommendation. Does model performance matter less than a high degree of explainability? Then a simpler model with transparent decision-making is preferable. Be careful with XAI techniques such as LIME and SHAP: although they can provide useful insights, different XAI techniques can generate different explanations for the same decision. So do not make a black box model ultimately responsible for a decision; treat it as a supporting system. A ‘second opinion’. And leave the final decision on whether someone gets a mortgage to a person.

References [1] T. Miller, “Explanation in Artificial Intelligence: Insights from the Social Sciences,” arXiv preprint, 2017. http://arxiv.org/abs/1706.07269
[2] https://github.com/marcotcr/lime
[3] https://github.com/slundberg/shap