Explainable AI: explained

Imagine your mortgage application is rejected. Your first question is probably: why? A mortgage lender can explain why you do not meet the requirements for a mortgage. But what if that decision is made by a machine learning model? We would not settle for “computer says no”. The lack of an explanation is problematic and affects several people. The person affected by the decision needs to understand the reasoning behind it. You also want to prevent the decision from being based on unintended discrimination, for example: someone must not be rejected because of their ethnicity. For the data scientists and data engineers training a model, understanding its behaviour is essential to finding errors or weaknesses. eXplainable AI (XAI) can help improve a model. We increasingly encounter automated systems that use machine learning algorithms. They are used by government organisations, the financial sector and healthcare providers, among others. And there is good reason for that. In many cases, these algorithms perform well, sometimes even better than a person. But can we blindly trust a model with an accuracy of 90% or higher, or should we ask how it reached its decision? Research into XAI has increased significantly in recent years. Ethics in AI has received more attention. This may be because AI is being used on a larger scale and more often for important decisions, but also because algorithms are becoming increasingly complex. Too complex to understand. Is XAI the solution?

What is explainability?

Let’s start by defining explainability . Unfortunately, there is no single agreed definition (yet). The literature generally uses the following definition: “the extent to which a human can understand the cause of a decision” [1]. Here, the decision is made by a machine learning model or algorithm. Broadly speaking, we can distinguish between two types of explainability:

Ante-hoc explainability

Explainability by design. In this approach, you develop a model that is inherently transparent. Think of a simple decision tree or a simple linear model. These models are easy for people to understand and therefore ‘explainable’: we can understand how they reach a decision.

Post-hoc explainability

But what if a model is not transparent? Such a model is called a ‘black box’: you can see its inputs and outputs, but you do not know what happens inside it. In this case, you can choose post-hoc explainability: we generate an explanation for an existing black box model that is not itself ‘explainable’. This often involves using another algorithm to explain the original black-box algorithm and, in effect, open it up.

Which approach is preferable?

A simple model that is already explainable is usually preferable to a complex neural network that still needs to be explained using post-hoc XAI techniques. However, there are often situations where a simpler model is not enough and cannot achieve the performance that a more complex model can. This creates a trade-off between accuracy and explainability: you have to weigh a model’s performance against how easy it is to explain. As the figure below shows, greater explainability often comes at the expense of a model’s accuracy.

A major advantage of post-hoc methods is that you do not have to compromise on performance: we keep the model as it is and add a technique afterwards that can generate an explanation. This makes it possible to use a black-box neural network for a prediction task and explain its decisions afterwards without sacrificing its power or performance. Another advantage is that this technique can be applied to existing models – and to different types of models. When people talk about XAI, they are often referring to post-hoc explainability.

Why is XAI important?

Data scientists and data engineers need to understand what a model does so they can debug it and address any unintended discrimination. Before a model goes into production, people need to trust its predictions. But they are unlikely to trust an outcome they do not understand. A lack of explainability can mean that end users will not use the model. A model that can explain its outcomes is transparent, which helps build trust in it.

EU privacy legislation also needs to be taken into account. The requirements of the General Data Protection Regulation will make it harder to put a model into practice if it cannot be explained. The European Commission has gone a step further with its recent proposal for an AI regulation: in April 2021, it announced that we can expect stricter rules for high-risk AI applications.

How do we approach this?

There are now many different methods for explaining machine learning models, and the number is growing considerably. We can categorise these methods by their explanation scope. Global XAI methods try to explain the behaviour of the entire model, while local XAI methods focus on explaining an individual decision. A global explanation is particularly valuable to a data scientist; a local explanation is useful to the person affected by a decision.

The two best-known examples of local (and post-hoc) methods are LIME (Local Interpretable Model-agnostic Explanations)[2] and SHAP (SHapley Additive exPlanations)[3]. These algorithms differ somewhat, but the idea is the same: they change parts - or features - of the input data and examine how much each change influences the final decision.

For a mortgage application, you could change the applicant’s income or whether they have any debt. For text classification, you replace or remove some words and then determine the new outcome. For an image, you divide it into super-pixels: groups of pixels with similar colours and brightness. The result is a set of scores showing how important each feature is to the decision.

What a LIME or SHAP explanation looks like depends on these features (which, in turn, depend on the type of data). For tabular data, you can show the features and their scores in a chart:

A fictional example of a LIME explanation for a prediction of whether a mortgage applicant presents a low or high risk of default. Red indicates features that contribute to the outcome ‘high risk of default’; green indicates ‘low risk of default’.

For recognising dogs and cats in images, the explanation consists of super-pixels: Why is LIME so popular? Its software is accessible and relatively straightforward to apply. More importantly, it treats every model as a black-box that it cannot access, which means you can apply it to any model. SHAP could also be an option. It produces a more robust result, but takes longer to compute. This contrasts with model-specific XAI methods, which can only be applied to a particular type of model and cannot be implemented arbitrarily.

What’s next?

We have a model whose decisions can be explained, but what comes next? Can we assume that LIME’s explanation is correct? The answer is no. Just as you need to evaluate a predictive model, you also need to evaluate an explanation. The explanations LIME generates may look plausible to us without actually reflecting how the model behaves. There is also a reasonable chance that another XAI method, such as SHAP, will give a different explanation from LIME. Evaluating explanations is a field of study in its own right, with much research still under way.

Even so, it is worth paying attention to a model’s explainability, especially when it informs high-risk decisions with implications for society. Think of systems used in healthcare, government or education. These sectors need a higher level of explainability than systems such as spam filters, chatbots or recommendation systems.

Conclusion

As algorithms become more complex, it becomes harder to explain and understand exactly what they do. Yet it is important to understand how a model reached a decision and which features played a role. Explainability matters not only to data scientists and data engineers, but also to end users, domain experts and everyone else involved. It can increase trust in a system, help uncover errors and weaknesses, and prevent unfair outcomes. Not all algorithms need the same level of explainability, though. An explanation is essential for a medical diagnosis, while it is less important for a personalised Netflix recommendation. If a model’s performance matters less and a high level of explainability is the priority, a simpler model with transparent decision-making is preferable. Be careful with XAI techniques such as LIME and SHAP: although they can provide useful insights, different XAI techniques can generate different explanations for the same decision. So do not treat a black-box model as ultimately responsible for a decision; use it as a supporting system. A ‘second opinion’. And leave the final decision on whether someone gets a mortgage to a human.

References [1] T. Miller, “Explanation in Artificial Intelligence: Insights from the Social Sciences,” arXiv preprint, 2017. http://arxiv.org/abs/1706.07269
[2] https://github.com/marcotcr/lime
[3] https://github.com/slundberg/shap