Original: https://www.researchgate.net/publication/332873867_Visualizing_Deep_Networks_by_Optimizing_with_Integrated_Gradients
Authors: Zhongang Qi, Saeed Khorram, Li Fuxin School of Electrical Engineering and Computer Science, Oregon State University
included in CVPR 2019
Abstract
Aiming at the problem that the heatmap has nothing to do with the network and misleading interpretation when the heatmap visualizes the deep network, the I-GOS (Integrated-Gradients Optimized Saliency) method is proposed to optimize the heatmap to minimize the masked image classification score (a bit like the perturbation based method) ).
Novelty: It proposes to use the integral gradient instead of the conventional gradient to calculate the descent direction, which avoids local optimum and speeds up the convergence speed.
Advantages: Compared with previous methods, our method can flexibly calculate heatmaps of any resolution according to different user needs; experiments prove that the heatmap generated by our method is more relevant to the decisions made by the underlying deep network (rather than repairing input image itself).
1.Introduction
An introduction to deep learning interpretability and heatmaps.
The methods that explain deep neural networks, GBP, DeconvNet, point out their problems: essentially doing (partial) image restoration, highlighting the boundaries of objects to produce more human-interpretable views, which does not necessarily represent what the model has really learned s things.
Perturbation Based early method - direct optimization of deletion-game: convergence is too slow.
Integral gradient method - the heatmap generated is unclear and difficult to understand.
Comparison of the above methods:
the original prediction probability of the model for the indigo bunting is 100%. When the proportion of pixels calculated by the three methods is 6.4%, delete these pixels, and the probability of I-GOS prediction is the lowest. Only display For this part of the pixels, the probability of I-GOS prediction is the highest, indicating that the heatmap calculated by I-GOS is most consistent with the model decision (model: I can’t recognize it without you, but I can recognize it with you, so you are the most suitable for me).
Our approach:
In this paper, we propose a new visualization method, I-GOS (Integral Gradient Optimized Significance), which improves upon the mask optimization method in "perturbation based..." using integral gradients. The idea is that the direction provided by the integrated gradient may lead to a better global optimum than the normal gradient, so we replace the gradient in the mask optimization with the integrated gradient. Due to the high computational cost of the integral gradient, we adopted a gradient projection method based on line search to maximize the use of each calculation of the integral gradient (do not integrate, directly accumulate the number of steps). Our method generates better heatmaps and uses less computation time than raw mask optimization, since line search is more efficient in finding suitable step sizes, allowing fewer iterations to be used.
The contributions are as follows:
1. We develop a new heatmap visualization method, I-GOS, which optimizes a mask using integrated gradients as a descent step.
2. Through regularization and perturbation, we better avoid adversarial masks at higher resolutions, making more detailed heatmaps more relevant to the model's decisions.
3. Extensive evaluations show that the proposed method outperforms the state-of-the-art methods, especially in insertion and deletion metrics.
2.Related Work
Backpropagation Based...
Perturbation Based...
Human Gaze...
3.Model Formulation
3.1 Gradient and mask optimization
Introduces the significance of gradients in the process of changes in input images leading to changes in classification scores.
The Mask calculation method in the perturbation based direction is introduced:
3.2 Integral Gradient
Formula 3 integrates the gradient of the entire process from the base pixel (the pixel of the base image, understood as an all-black and all-white image, used for insertion/deletion game) to the original pixel, not just the gradient of the original pixel, but is a gradient integral, a continuous process.
Some papers have proved that the cumulative gradient of the pixels is the difference of the picture score, and the formula 4 is established:
The integral in (3) is calculated using the approximation method. The integral interval is divided into sections and added up (here is no longer a pixel, but a picture): S is the
number of segments divided into, generally 20, and there is a mathematical proof in the appendix.
The following figure explains the process of integrating the gradient:
A to B, accumulated in segments, controlling gradient convergence.
3.3 Integrated Gradients Optimized Heatmaps
The improvement of this paper stems from the regular function g(M) in formula (2), and the defect of formula (2) directly calculating the gradient of the original image.
The original regular function is similar 
, which leads to the limitation of the optimization of the model. The black dotted line in the figure above is the limit range of the original method. In addition, the direct calculation of the gradient makes the model unable to perceive the global gradient change, and will find the local optimal C; while the integral gradient can improve the deficiency of the direct calculation of the gradient.
Therefore, this article brings the integral gradient into the formula (2), and calculates the partial derivative of the original regular function: 
the combination is IGOS:
3.4 Computing the step size
In order to optimize the calculation of the integrated gradient, the backtracking line search method was used and the Armijo condition was modified to help calculate the appropriate step size of the total gradient (the reason for the modification here feels a bit subjective, there is no mathematical proof, and the direction is guaranteed to be correct, but the numerical Will it affect the result, modified backtracking line search):
α is the step size; β is a parameter (0,1), the specific search method:
3.5 Avoiding adversarial examples
Reduce countermeasures: add noise and reduce mask pixels.
experiment
Clear thinking
