Interpretable machine learning task01-preliminary knowledge

Interpretable machine learning task01-preliminary knowledge

What is explainable artificial intelligence?

• 
• Modern machine learning is statistical machine learning, especially deep learning, which uses data to fit data distribution and decision-making edges. It is high-dimensional and non-convex, and decision-making is a black box.
• Some soul questions
  • What is the brain circuit of AI? How does AI make decisions? Is it consistent with human intuition and common sense?
  • What features will AI focus on, and are these features really useful?
  • How to measure the different contributions of different features to Al prediction results?
  • When does AI work and when does it not work?
  • Is AI overfitting? What is its generalization ability?
  • Will he be misled by hackers and let the AI ​​turn a deer into a horse?
    • Adversarial examples
  • If a certain feature of the sample becomes larger, what impact will it have on the Al prediction results?
  • If Al misjudges, why does it make a mistake? How can it not make a mistake?
  • The two Al predictions have different results, which one should I believe?
  • Can Al teach the learned characteristics to humans?
• AI still has some problems in the AIGC field
  • It shows that he has not really learned this part of the knowledge
  • A black box, prone to mistakes
• In some key areas such as autonomous driving, how to make humans believe in black box algorithms
• Explainable learning is the study of opening up black box learning

Why should we learn

Brother Tongji Zihao’s topic selection suggestions

• Explainable learning can intersect with various directions of AI
  • data mining, NLP, RL, KG, federated learning
  • CV
    • For example, the identification basis for target detection
  • NLP
    • Decision words/words for text classification
  • Recommended system
    • The rationale behind the recommendation
• common research methods
• Combination of specific tasks
  • Large model, weak supervision, defect anomaly detection, fine-grained classification, decision-making
    AI and reinforcement learning, graph neural network, Al correction , Al4Science, Machine
    Teaching, adversarial examples, trusted computing, federated learning.
• Writing papers can be combined with an interpretable algorithm to analyze tasks in subdivided fields.
  • [Research on bearing fault diagnosis method and interpretability based on acoustic imaging and convolutional neural network](https://kns.cnki.net/kns8/Detail?sfield=fn&QueryID=10&CurRec=4&DbCode= CJFD&dbname=CJFDLAST2022&filename=ZDCJ202216029&urlid= &yx=)
  • [Research and exploration of the interpretability of radar image deep learning models](https://kns.cnki.net/kns8/Detail?sfield=fn&QueryID=10&CurRec=7&DbCode= CAPJ&dbname=CAPJLAST&filename=PZKX20220613003&urlid=11.5846.TP.20220613.0913.008&y x =Y)

Some cross-cutting research directions in interpretability

Machine Teaching

• Teaching humans to learn through AI

• eg.Making a Bird AI Expert Work for You and Me

  • Teaching humans to identify birds through AI
  • human-centered-ai

• eg. Neural networks teach humans to predict process parameters

Fine-grained image classification

• For fine-grained classification, there are various subcategories under major categories, which are difficult for humans to classify. AI can be used to classify them better.
• Heat map of medical image classification giving decision-making
  • kaggle:SIIM-ACR Pneumothorax Segmentation
  • 
  • bg: paramedicine, teaching
• Industrial defect detection
  • There are fewer high-quality annotations in the industry
  • Ability to locate defect locations (detection/segmentation) by interpretability by training only one classification model
• Bioinformatics
  • protein
  • genetic research
• Object classification
• architecture

AI safety/adversarial examples

Frontier AI Direction

• chatGPT
  • Completely black box, suitable for interpretable analysis
• AIGC
  • Stable diffusion and other drawings
• large model
• protein
  • alphaFold

Learning that is inherently interpretable

• KNN
  • 
• logistic regression
  • The input features can be understood, and the weight intuitively reflects the decision-making importance of that feature.
  • 
• linear regression
• decision tree
  • Decision making using if else
• Naive Bayes

Some interpretability analysis methods

• Algorithm built-in visualization
• decision tree
• Feature weights that come with the algorithm
  • logistic regression
  • random forest
  • lgb
• Permutation ImportancePermutation Importance
  • Randomly disrupt a column of features. If it has a greater impact on the model, it means it is more important.
  • sklearn demo
• PDP diagram, ICE diagram
  • sklearn demo
  • 
• Shapley值
  • Library official github
  • Can be used to interpret tree models
  • deep learning model
    • 
  • Model agnostic explanation
  • paper
• Lime
  • paper
  • 
  • Build an interpretable model locally around the prediction, display representative samples at the same time, and model the task into an optimization task of a sub-module

Interpretability and model performance

• Traditional machine learning algorithms generally have good interpretability, but their prediction effects are slightly poor.
• Neural network has the best prediction effect but the weakest interpretability
• How performance and interpretability trade off ^{1 , 2}

Interpretable algorithm classification

• 
• all

Interpretability Analysis of Deep Learning

• Neural networks are composed of layers. The higher the level, the more abstract the corresponding high-dimensional features are, making it difficult for humans to understand directly.
• So related interpretability algorithms are needed
• eg. Feature map for handwritten array recognition
  • 
  • playground

CNN interpretability analysis

• Visualized convolution kernel/feature map
  • CS231N Visualization
• ZF Net
  • Introducing a new visualization technique that provides insights into the functionality of intermediate feature layers and the operation of classifiers
  • Understand CNN through the impact of indirect methods of occlusion, scaling, translation, and rotation on predictions
  • Use deconvolution to find the pixel or small image that activates a certain neuron
• RCNN
  • Find the original small image from neuron activation
  • Original figure 3, experiment 3.1
    • 
  • The author visualizes the neurons from the fifth layer of pooling. The white box is the receptive field and activation value.
  • It can be seen that some neurons capture specific concepts, such as the person in the first line and the words in the fourth line.
• Some neurons capture textures and materials
• CAM-based visualization
  • A large class of algorithms, including various improved algorithms based on the original CAM
  • Constructed a general localizable depth representation that can be applied to a variety of tasks
  • torch-cam
  • 
  • Example of CAM visualizing image segmentation
  • Explanation for wrong predictions
    • 
  • Explore whether the model is biased
    • 
    • (b) is a biased model. If you judge a nurse based on her hair, your hair (gender) will be used as a characteristic to judge a nurse, which is against ethics.
• Semantic dimensionality reduction visualization
  • 
  • By reducing the dimensionality of high-dimensional sample features to low dimensions for visualization, the distribution is related to semantics
  • The word vector representation of eg.word2vec. After dimensionality reduction, it is found that words with similar meanings are distributed close in the space.
    • cs224N
    • 
    • Dimensionality reduction algorithm
      • PCA
      • TSNE
      • UMAP
• Generate images that meet your needs
  • By optimizing samples, the image can meet certain requirements, such as the maximum activation of a certain neuron or the maximum prediction of a certain category.
  • Application scenario: Adversarial sample attack
    • FGSM etc.
    • By continuously iterating samples and imposing constraints (minimum disturbance), the model misjudges

Summary and further reading

• Method comparison
  • 
• Compared
  • 
  • 
• Zhang Zihao codebase
• pytorch-cnn-visualizations
• pytorch-grad-cam
• torch-cam
• Related papers
  • Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
  • Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
  • “Why Should I Trust You?”: Explaining the Predictions of Any Classifier
  • Visualizing and Understanding Convolutional Networks
  • Show and tell: A neural image caption generator
  • LayerCAM: Exploring Hierarchical Class Activation Maps for Localization
  • Learning Deep Features for Discriminative Localization
  • Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
  • Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Introduction to OpenMMLab category activation heat map visualization tool
• DataWhale 6 machine learning interpretability frameworks!

Thinking questions

Reference

all