AI interpretability: innovation by the discovery of new combinations of variables

When can an algorithm be considered reliable? Artificial Intelligence, and more particularly Machine Learning, offer a huge range of opportunities for data scientists. And yet, there are still large gaps in our knowledge about how AI actually works. Senior Lecturer at the Centre for Applied Mathematics at France’s École Polytechnique (CMAP) since 2016, Erwan Scornet is a specialist in the fields of theoretical statistics and automated learning. Today, he is leading several research projects into Machine Learning, including one devoted to the interpretability of algorithms.

Our ability to understand Artificial Intelligence is based on its interpretability. Could you give us a definition of this? And what does it involve in practical terms?

Erwan Scornet: Interpretability is one of the research topics I’m working on, in collaboration with a PhD student and other colleagues. To understand the principle, imagine a production line for car parts. Based on a number of variables about the line (temperature, pressure, run rate, etc.), AI will be able to predict whether a part will be faulty. Now, for an industrial company, the object of the exercise is not to predict the fault, but to understand the process that caused it. Interpretability is the answer to the question “How did that happen?”

That said, there is no precise definition of interpretability. An initial approach to providing one would be based on three major principles:

• Simplicity: does the algorithm use a lot of parameters? Is the algorithm carrying out simple operations? 

• Stability: does the algorithm provide results that are reproducible and coherent over time, compared to the data that is has learned from?

• Predictability: how accurate are the algorithms predictions?

Interpretability, explainability, why are these ideas important in AI?

Erwan Scornet: These are essential notions in the use of AI. If you take the case of linear regression, these models are simple, predictable and highly comprehensible. For example, you could give a price for a property asset by taking into account several different parameters, such as the address, the surface area, what floor of a building it’s located in, the number of rooms, the presence of a terrace, etc. A coefficient is associated with each variable. Using a simple model, you will get a result that is interpretable, but the accuracy of its predictions will be poor. The reason being that simple models very often lack the flexibility to understand the true links between the different variables. In our example, the link between the variables and the price of the property is not necessarily linear.

We need more complex algorithms (random forests, neurone networks), commonly known as ‘black boxes’, that can identify non-linear correlations. They can be used for operations that are very simple in nature, but that so are entangled that a data scientist would never be able to explain how they operate. These methods perform very well in terms of prediction accuracy, but they are difficult to interpret. However, a data scientist can attempt to explain the algorithm’s end-product by looking at the importance of each variable. It would be difficult to get a full interpretation of the results, but a partial response would be possible. 

So basically, it’s possible to interpret a simple model and to explain a complex model. In one sense, the notion of explainability merges with the idea of interpretability: their shared aim is to help with decision-making by going further than simply making a prediction.

As a specialist in Machine Learning, and ‘random forests’ in particular, could you tell us what this involves?

Erwan Scornet: ‘Random forests’ are Machine Learning algorithms based on decision trees. These trees describe processes that are very simple to understand. The approach is similar to a doctor asking questions of a patient -- such as their age, their symptoms, and others depending on whether the patient is male or female -- in order to make a diagnosis. The series of questions being asked depends on the answers being given, with the decision-making process leading to a diagnosis, and therefore a prediction of whether you are ill or not. 

The predictive performance of a decision tree is fairly good. However, a single tree lacks stability: changing some of the data can lead to a completely different series of questions. It’s why data scientists base their work on ‘boosting groups’ and random forest methods in order to aggregate the results. 

To continue with the same metaphor, imagine 100,000 doctors all making a diagnosis at the same time. We could aggregate the results and make a recommendation, but although we would understand each individual diagnosis, we would be unable to understand the aggregation of the results. For that reason, we would need to carry out a set of tests to find the key variables in the aggregation process.