Lecturer introduction of this issue:
Byte, Chief Scientist of Guanyuan Data. Leading the application of multiple AI projects in the world's top 500 companies, and winning the Hackathon championship in the direction of smart retail for many times. He once worked in MicroStrategy and Alibaba Cloud, and has more than ten years of industry experience.
In the application of machine learning in the commercial field, allowing the business side to understand the model logic and improve confidence in the prediction results plays a key role in promoting the advancement and implementation of algorithm projects. Therefore, the interpretability of models and prediction outputs has become an increasingly important issue. In this sharing, we will introduce the technical principles of explainable machine learning, as well as some application cases in model development and project implementation.
Table of contents
- What is Explainable Machine Learning
- Technical realization
- Applications
Why explainable machine learning?
Let’s first introduce why explainable machine learning is needed. I usually encounter different scenarios in machine learning projects, such as the Taobao recommendation system, which recommends some products that you may be interested in. If the recommendation is inaccurate, it will not cause any big losses or bad results. Influence. In this scenario, you only need to pay attention to the accuracy of the model, and make it as high as possible.
Another scenario, like OCR optical character recognition, is a relatively mature scenario and can basically achieve 100% accuracy. And like some other demand forecasting products, such as financial credit rating system and medical diagnosis system, it gives forecasts. For example, demand forecasting may buy 10 boxes of Coca-Cola in the future. loss. In this case, we need to have requirements for the output of machine learning. There are also credit ratings. If I apply for a credit card and get rejected, I will want to know why it was rejected, whether it is discrimination against my age or gender, etc. For example, for the topic of Responsible AI, there will also be requirements related to this aspect, so we need explainable machine learning, so that while the model outputs predictions, it also outputs the reasons for the prediction changes.
The figure above shows that when faced with a black-box AI model, people with different functions, such as business managers, customer service, IT or model development engineers, have different demands on the output of machine learning. For example, as an algorithm engineer, if the prediction obtained from the black-box machine learning model is not good and deviates very far from the true value, then he may wonder why the model gave such a prediction and how to make the model better good.
To sum up, first, the model output has a significant impact. If a wrong prediction is given, the cost will be relatively high, and interpretable machine learning is required. Second, especially for algorithm engineers, it is convenient to debug in the development model. Of course, there are also some other application scenarios, such as the Bias of the avoidance model. There are also many discussions in this area in the community. If the model has some sexism or racial discrimination, the model cannot be launched. In addition, during model interaction, if you encounter challenges from the business side, some explanations can be given, which can also improve the acceptance of business users. There are also various audit requirements and regulatory requirements, such as credit rating systems. If the system is deployed online, it must meet the requirements of transparency or explainability.
Various approaches to explainable machine learning
Let's cut to the chase and explain the various approaches to machine learning. The picture on the right is divided into two dimensions, one is the accuracy of model prediction, and the other is interpretability.
When using some simple models, such as the linear model in the lower right corner, Decision Tree decision tree, or some probabilistic graphical models, their accuracy may be worse because the model itself is relatively simple in structure, but their interpretability will be very good . In the upper left corner, like neural networks or some integrated learning methods, the model complexity will be very high, and the accuracy will increase accordingly, but its interpretability will be poor.
In terms of method classification, linear models are inherently interpretable models. The neural network is a method of post-interpretation. Open the black box to see how it makes predictions. There are also some model-specific methods, such as neural networks that can be explained by the gradient of subsequent models. There are also some black-box model interpretation methods, such as SHAP, which do not have any requirements for the model, and can do some interpretation.
Local interpretation and global interpretation. The global interpretation is similar to a linear model, and its weight is how the global performance of the model makes decisions. Local explanations are equivalent to giving some specific explanations for a single sample.
Next, some inherently explanatory models are introduced, such as linear regression, linear logistic regression, decision tree, etc. After training these models, an explanation can be obtained through the coefficients and the structure of the model. The coefficient is relatively large, indicating that this feature has a relatively large impact on the entire output. The smaller the coefficient, the smaller the impact. This is an inherent explanatory property.
There are also some similar ideas, such as RuleFit or Skope-rules, which can extract some rules from the data and automatically build such rules as shown in the figure on the right. Its interpretability is very strong while predicting. Like Naive Bayes, kNN, can be explained by other samples.
There are some other internal explanation models, and the general method is to add some methods to some simple models. Like EBM or GAMxNN, they belong to the generalized additive model, which is equivalent to building several linear models based on the online model, each linear model describes a part of the system, and then adds them up. Facebook's Prophet is also a typical additive model, which will give some explanations for seasonality or trend. Recently, Google open-sourced TFT, which is a time-series model. It has a special treatment for the attention mechanism. After training, the attention mechanism can also be used as an output for model interpretation.
And like neural networks, most of them are explained afterwards after the training model is completed. We use the neural network to calculate its gradient and then do some backpropagation to do some interpretation. The picture above shows how we understand what the CNN model has learned in the early years of research. We can change its input to maximize the activation of each CNN filter. In this way, you can see that the first few layers of CNN can learn the edge, the middle one can learn some shapes, and the higher ones can learn some concepts. This is actually using the neural network to find the gradient, and then do the principle of backpropagation to make some explanations.
This example is also similar. It comes from the book "Deep Learning with Python", which introduces the method of implementing Grad-CAM. Like this picture, why is it predicted that this picture is an elephant? We will find the activation of the elephant class, and then send its gradient back in reverse, and then look at the pixels on the entire CNN layer that have the greatest influence on its activation, and then use the heat-map form draw it. It can be seen that because of the heads of two elephants, this picture is classified as an elephant with the most contribution, forming an explanation.
In addition to the neural network model, there are also a large number of tree models, Gradient Boosting, etc., which can be explained in some Post-hoc model-independent methods. The general process is that when you have a Black Box Model, you can use some explanation methods to finally output some instructions to the user. Let's focus on PDP, LIME and Shapley.
The idea of PDP is to modify the eigenvalues. After a eigenvalue is trained in the model, there is a black model. When inputting it again, make some changes to its eigenvalue, such as changing it from 0 to 100, to see what the output of the model looks like. It is very similar to the analysis of what-if, to see how its output will change as the characteristics change. The picture on the right is the interaction of two features, and the two features are modified separately to see its overall impact on the model output.
LIME is a locally interpreted method. Make a separate explanation for each instance, and its approach is to do some sampling near the instance. The picture on the left is to explain the bold red plus sign. First, a lot of samples are taken around it, and a linear model is built after sampling around the local samples. The coefficients of the linear model for individual features are themselves interpretable. Including xLIME on the basis of LIME, it is to construct two linear models to explain this feature from small to large, from the current point to a larger direction.
LIME can also be applied to text. The challenge with text pictures is that it is better sampled if it is tabular data. Adding or subtracting a little near the sampling point makes it easy to get a new sample, and the text and pictures will be a little more difficult. For example, for pictures, it may need to use the Super-pixel method to do some sampling of some pixel blocks. The author also mentioned some of his specific practices in this article, and the effect is still good.
The principle behind Shapley is a theory of game theory. For example, in the picture above, the problem to be solved is that three people go to carpool, and the three get off at different places, so how should the final fare be shared equally? The problem of migrating to machine learning explanation is that the final total fare is the predicted output of the model, and how much fare each person pays is the contribution of the three features. There is a set of very complex operation logic behind this, to calculate the attribution of each feature, that is, a local explanation, and make some explanations for each prediction sample.
Based on the Shapley theory, a Shap library is open sourced. It is also the most widely used library in the field of model interpretation. It has many built-in visualizations, and has done a lot of calculation optimization on the basis of the operation of the original Shapley. Because the calculation overhead of the original version is very high, Shap has made some optimizations in the tree model and deep learning model respectively, which can keep the running time within an acceptable range.
Next, we will talk about some sample-based methods. So how do you explain this prediction based on this sample? It is to find a sample similar to it, or find a representative sample, and then explain the behavior of this model, or explain the distribution of the training data in this way. This method must have a counterfactual method, or a prototype method, and an attack method like adversarial on neural networks.
The counterfactual explanation follows. For example, the model judges the model of the loan application, and after inputting data, it rejects the loan application. Shapley's counterfactual explanation is that what changes need to be made in the eigenvalues can be applied next time, and the predicted label will change from 0 to 1. This is a counterfactual explanation.
Next is the prototype sample explanation. Prototype is a representative sample in the data. On the contrary, if it is not very representative, it is called Criticism, which is similar to outlier. This method can help us discover the shortcomings of data and models. For example, we trained a CNN model for object classification and recognition. The prototype is some normal photos. Criticism found some photos that are not well-lit at night. If these photos are classified as Criticism, it means that our data set may be for The number of photos taken at night is not enough. The data accuracy of this model in this part is relatively poor and needs to be improved.
This last one is an impact sample. It is equivalent to the original full amount of data to train, it will fit the information model of this red line. After removing the influential instance in the lower right corner which is very far away from other points, it will fit a new blue line, and you can see that the slope changes greatly, so the point in the lower right corner can be called an influential instance is a very influential sample. However, this method also has some efficiency problems. It is similar to Shapley's principle. It needs to consider the influence of each sample, remove each sample, and retrain a model, which is very expensive. There are also other methods, such as neural network models, which can quickly find influential samples on a constructed model, and methods such as ensemble tree models that can quickly find influential samples.
We talked about a lot of explanation methods earlier, let's look at the comparison of explanation methods. At the top is an explanation of the image type. Compared with Grad-CAM and other methods similar to saliency graphs, there are also methods like LIME and SHAP, which are explained by examples of example data on the far right, some tasks of text in the middle, and a time series input at the bottom. explained in different ways. Then put these explanation methods on Amazon's Mechanical Turk crowdsourcing platform, and let humans judge which explanation is better. It can be seen that the most accepted one is Explanation by example. This also gives us an inspiration. If the explanation is used for this kind of business system, or needs to be explained in a business way, it is better to use sample explanation. But as a developer, Shapley might be more useful.
This is also a method choice for model interpretation on Twitter. Including whether you can accept a model with poor accuracy, whether you need to do a global interpretation, or a local interpretation of each instance, a similar decision diagram is listed, and you can also choose based on this.
So what is a good explanation? There is also a lot of research on this. Especially when the target audience of the explanation is a different group, special attention should be paid to some points. From the perspective of social psychology, it is a good explanation to see how people accept it. Academia also has some research in this area.
The recent AAAI 2021 tutorial summarizes the nature of a series of good explanations from the mathematical nature. The above is how many of these properties are explained by each method, and the two with a relatively large number are Shapley Value and Integrated Gradient. If these two methods use neural networks, you can use IG. If you use some traditional models, Shapley Value is also a very good choice.
Combat session
Earlier we talked about how to use the method of model interpretation to debug the model. The picture on the right is how to build the model. The person in the middle is standing on top of a bunch of matrix and computing things. The funnel on the left is where the data comes in, and the results come out from the outlet on the right. After building a very complex model, the data comes in and the results come out, and you see that the accuracy rate is not very good, so what will you do next?
This person fiddles with the complicated things in the middle, that is, adjusts the parameters, then changes the model structure, changes to a different activation function or loss function, and then sees if the effect has improved. If not, continue to debug. Such debugging may be relatively inefficient, and it is impossible to know exactly what the intermediate model does, so it is not of great help to the iterative development of the model.
Having model interpretation tools in place can help improve this process. When the results are available, model interpretation tools can be used to explain some items that are not well predicted. Check whether the features used meet expectations. If not, artificial experience can be embedded in the model construction process. After discovering the problem through model explanation, a new round of training was run and a new result was obtained. At this time, we can continue to use the explanation tool to verify whether the improvement is effective. When doing experiments, in addition to overfitting and underfitting, which can be seen directly from the indicators, other more detailed issues are very helpful after having model interpretation tools, which will help you speed up iterations, and It is a good way to integrate human business experience into it.
This is a very interesting example, called Dark Sight, which does a dimensionality reduction and model compression at the same time. After training the classifier multi-category model, the predicted output results of the model can be reduced in dimension and visualized on a two-dimensional plane. This dimensionality reduction has a very good feature. You can see the two in the middle, one is the blue block, which is a series of 9 numbers, and the pink block is the number 4. There is a junction between the two circles. If you click on the number instance at the junction, you will find that the number looks like 9 and 4. This feature can intuitively visualize the part that the model feels difficult to judge on the canvas, and analyze it by clicking and interacting. The samples in the middle are like 9 and 4, which is more difficult for the model, so it can be targeted to do some processing. This also brings us an inspiration. In fact, the dimensionality reduction of many models and data is also helpful to the interpretation of the model.
This is the use of IG - Integrated Gradients to do disease diagnosis on X-ray films. The picture where the original input can be seen is on the left. The output label is that the person has a disease. After we use IG to explain it, we can see some pink highlights on the right side, which means that the model on these pixels thinks that the sample is classified as having a problem, yes The pixel with the largest contribution.
We can zoom in on the pixels again to see what kind of judgment the model has made. It turns out that this is actually two lines drawn with a black marker on it. It should be that the doctor saw the X-ray film during the diagnosis process and diagnosed something wrong. He drew two lines with a marker pen here. What I got was the doctor's mark, not the pattern of the pathological tissue, so the judgment of the model is actually problematic. In this way, it is very convenient to find some weaknesses in the model through the method of model interpretation.
The following example is similar, and it also uses IG's method. This is a VQA - Visual Question Answering task of looking at pictures. The problem is how symmetrical the white bricks are in the picture. The answer given by the model is very, which is very symmetrical. This answer is a correct answer that makes sense. But after we used technology to analyze it, we found that the red text has a high contribution, the blue text has a relatively negative contribution, and the gray text has almost no contribution, so in fact the model gives this Answering the forecast, in fact, it focuses on the How, so the degree of attention is problematic.
Therefore, if we replace the middle symmetrical with other words, such as asymmetrical, is it asymmetrical, or how big is how big, or even change it into a sentence that has no meaning at all. White bricks speaking, how fast, it answered very. This helped us discover the problems of the model. It didn't really understand the sentence and the picture, but it seemed to give a good prediction, which also helped us discover some problems of the model.
If it is some traditional tabular data, then we can use SHAP for analysis. For example, like XGBboost or LightGBM, everyone will often analyze feature-importance and give a global explanation of the model. In fact, if you use shap, you can give a better explanation of feature-importance. In fact, it is not only the order of the contribution of the feature from high to low, but also whether the contribution of this feature is positive or negative, that is, the left side of the coordinate axis is Negative contributions, positive contributions to the right. In addition, the value and distribution of this feature can also be displayed on a graph.
For example, if its line is relatively thick, it is a conical distribution with a large amount of data; for example, the blue one is a case where the eigenvalue is relatively low, and the red one is a case where the eigenvalue is relatively high. The picture on the left is a well-trained model. You can roughly see most of the features. When the Feature value is high and low, it can clearly distinguish red as negative influence and blue as positive influence. It basically May be separated.
In the picture on the right, we randomly scattered the labels. It can be seen that its blue and red points are basically mixed together, and the degree of distinction is obviously not high, so it can be displayed through this pattern on the graph to help you find the problem. If you make a feature, it looks like the picture on the right, the high feature and the low feature will be mixed together, I don’t know if it is a positive impact or a negative impact, if there is no distinction, you need to see if your feature is made has a problem.
SHAP can also do some complex feature interaction analysis. This is an example about adult income, the two axes in the upper left corner are divided into two characteristics. One is the years of his education and below that is the age. The positive effect is that his income is relatively high, and the negative effect is that his income is relatively low. It can be seen that when the age is 20-30 years old, when the age of education is relatively low, it has a positive impact on the contrary. This is also easier to understand because between 20 and 30, if he has received more than 10 years of education, he may have just graduated at that time, and his income must be low in most cases. This can confirm whether some business knowledge can be reflected in your data and in your model. For example, on the right, you can see the impact of his occupation, age and income. We can analyze from it whether there is a certain age. The older he is, the higher his income will be. In some ages, the younger he is, the higher his income will be, and it is a profession that eats young people. All of these can be judged through the analysis of this specific interaction of SHAP value. If it does not conform to your business common sense, you can make some targeted data diagnosis and modification.
In this example, we did a binary classification problem and did a wave of feature engineering, and suddenly found that the AUC increased from the original 0.7 level to 0.9, which is obviously unreasonable, and some problems should have occurred. We observed the Feature importance before and after, and found that it seems that there is no obvious change in the Top feature. There is no feature that suddenly jumps up from the bottom. We did not find such an obvious leak feature. In fact, we can also use SHAP value to do this. You can choose the old and new models, and then find a forecast sample. For example, it was originally a false negative prediction that was wrong, but later it was predicted to be true positives and the prediction was correct. Let's look at the comparison of the SHAP value before and after, and then look at which feature has the biggest difference in the positive contribution of the SHAP value. We can pick 10 or 20 samples to verify separately, and find that if they all fall on the same feature, then it is easy to find out which leak feature is. This is also an example of directly using model detection technology to find leak features.
Then if it is a regression problem, it is similar. For example, you can find the predicted samples that are overestimated by Top and the predicted samples that are underestimated by Top, and then do some analysis related to SHAP value in the group respectively to see the characteristics of the most overestimated ones by TOP. What are they. If it is a problematic feature, we can do some targeted analysis. If the feature discrimination is not high enough, we may need to add some new features. Top underestimates the point, and it is a similar approach.
The same is true for classification problems. For example, if you focus on the wrong samples with the highest and lowest scores, you are most confident, but it predicts wrong. In addition, the prediction near the decision boundary is a sample near 50% probability. You can also use SHAP to analyze its characteristics to see if there are some conflicting characteristics. That is, there is actually some conflict between the a feature and the b feature, which causes their effects to be offset. In the end, the model is not sure whether it is a positive sample or a negative sample.
In addition to SHAP, there are a little more techniques that it can use for classification problems. Including using examples to do some explanations, such as ProtoDash to find some Prototype samples. There is also the data problem of using counterfactual samples to diagnose these samples.
The general meaning of this ProtoDash is a credit analysis of a credit card. If the value of the sample feature on the left is good, it means that the person is believed to be a creditworthy person. Then why the model gave him this prediction, we can find samples similar to him in the training set. So after ProtoDash finds a target sample, we find some prototype samples similar to it in the training set. After finding it, you can see that the first weight on the right is very high, indicating that it has a very high similarity with it. Then it can be well explained to the business side. When one of your training samples has a sample that is very close to his characteristics, because he was a trustworthy person at the time, then we think that he is also a new sample. A man of promise. But if there is a conflict between these two, for example, we find that the model prediction of this person is good, and the last label of the test sample on the left shows that this person has bad credit. At this time, we need to see why people similar to him have good credit, and whether there are any differences between these two people that we haven't caught. Then we need to add some features to strengthen the distinction between these two features, otherwise the model will be difficult to distinguish when they are similar. Or some hierarchical processing of features can help improve the model, and can be integrated into some of your business knowledge to improve the model.
The last one is a counterfactual case, or the example just now. For example, the X on the far left of the original person is predicted to be a person who does not keep his word. What should he do to make him pass the verification and consider him to be a person who keeps his word. The model suggested to him that there are three values to increase. The first one is ExternalRiskEstimate. This value is the bigger the better. Similar to Sesame Credit score, it is the average external score. The higher the Sesame Credit, the more trustworthy the person is. The following AverageMinFile and NumSatisfactoryTrades, one is the number of performances, the more times he fulfills the contract, theoretically speaking, he should be more trustworthy, and the number of months of his records is the length of his entire historical credit cycle. For example, if the credit card has been used for 10 years, then we will think that he is more likely to be a trustworthy person. If the counterfactual samples are more in line with your business judgment, it means that the overall training of this model is accurate.
But if there is something in the end, such as he wants to change his gender or race, there may be some problems with what the model has learned. We may find that this model is questionable in terms of fairness to him, so some improvements are needed. This is an example of such an example sample to explain the method and let it help us improve the model.
Regarding model development, there is another big application, which is how to use this model analysis technology to explain to the business side. Here is an example where we need to build models with different requirements. For example, we want to build a user repurchase model. We have three options here. The first is to be able to accurately predict which users will repurchase. The goal of your business is accuracy, that is, you can ignore other things and just improve the accuracy rate.
Model 2 is able to tell which combinations of features are highly correlated with repurchase behavior, which is an example of a global explanation. It is to provide insights to the business side, so that the business knows what needs to be done, which can increase the user's repurchase rate. It's a business guide for them. We can choose some relatively simple ones, such as decision trees and linear regression.
Model three is able to tell why this model thinks a certain user will repurchase. It is a finer-grained local interpretation. Its usage scenario is, for example, when we are doing customer marketing. After a customer makes a purchase, the model predicts that he may not repurchase later. We can explain why it is predicted that he will not come back to repurchase. Then you can make personalized settings for each user, and use some recommendations or business actions to enable him to repurchase.
This means that we need to build different models in different scenarios to achieve business goals. Of course, the business actually has many challenges. Most model explanations are correlational rather than causal. Sometimes you will see that it may suggest that you increase the user's age a little, but this is actually unreliable, because it is impossible to manually increase a person's age. In addition, when building an interpretable model, the construction of features must also be considered. If thousands of features are constructed and an explanation is given with SHAP value, the business side will not be able to understand it, even the explanation of thousands of features is incomprehensible. The forecasting process is actually very complicated. It is not just a model, it may be multiple models, and there are even some business processing rules behind the model, so the entire forecasting process needs to be explained. How to connect the whole process together is also a challenge. In addition, including the three different user scenarios mentioned above, the models and explanations we need to choose may be different, so different development methods are required for different business scenarios. Whether the explanation given at the end is an actionable step, that is, whether users can take some measures to change, is actually a very important topic. Not only does it involve model interpretation, machine learning, it may also involve some considerations in operational research optimization or knowledge graphs.
The following introduces some products and applications. For example, H2O.ai, SageMaker, including the AI platform on Google Cloud, they also provide components related to this model interpretation, and Fiddler and Truera both focus on the direction of model interpretation. start-up companies.
The table on the right shows some model interpretation applications in the industry. Let’s see in which scenarios they are applied, what technical means are used, and who are the final users? You can see a total of 8. Five of them are used by developers, that is, most engineers use some explanation techniques when developing models. There are only 3 relatively few, which will be explained to the final business side. And the explanation method for the business side is relatively simple, one is Feature importance. In terms of application scenarios, in addition to model development and evaluation, it is applied in the entire machine learning model life cycle. If you go to Fiddler's product introduction, you will also see it.
What I just talked about is interpretable machine learning. You can see the yellow boxes, such as machine learning, vision or NLP. If it is AI, its scope will be larger, such as some systems such as multi-agents, or some operations optimization, search optimization or planning, or something related to knowledge expression and reasoning, these may be all need explaining. When building the entire intelligent decision-making system, not only the algorithm, or the part of machine learning in a narrow sense, needs to be explained, but for other operations optimization, knowledge reasoning and logical reasoning, or reinforcement learning in a multi-intelligence environment The choice of strategy also requires some explanation. Or to help you improve the model, and help your business and audience understand why you make decisions, which are all very helpful.
Scan the code or click to use the demo to experience Guanyuan products