《ECKPN: Explicit Class Knowledge Propagation Network for Transductive Few-shot Learning》

Published in CVPR2021! ! !
Paper link: https://arxiv.org/pdf/2106.08523.pdf
Code link: None

1. Question

In recent years, methods based on direct inference graphs have achieved great success in few-shot classification. However, most existing methods neglect to explore class-level knowledge, which is easily learned by humans from a few samples.

2. Contribution

1) Proposed for the first time a graph-based end-to-end small sample learning architecture that can explicitly learn rich class knowledge to guide graph reasoning for query samples 2) Established a multi-head sample
relationship to explore pairwise samples Fine-grained comparison between each other, which helps to learn richer class knowledge based on pairwise relationships.
3) Use the semantic embedding of class names to construct multi-modal knowledge representations of different classes to provide more differentiated knowledge for reasoning about query samples. .
4) Extensive experiments are conducted on four benchmarks (i.e. miniImageNet, tieredImageNet, CIFAR-FS and CUB-200-2011), and the results show that the method has better classification performance.

3. Method

Consider how to explicitly learn richer class knowledge to guide graph-based query sample reasoning. As
shown in Figure 1, if we only utilize sample representations and relationships to perform few-shot classification tasks, we may misclassify query samples q For category 2. However, if we explicitly learn class-level knowledge representation to guide the reasoning process, we can correctly classify q because q is closer to the representation of class 1.

3.1 Problem description

3.2 Explicit knowledge dissemination network

Framework overview:
1) First, leverage support and query samples to build instance-level graphs.
2) Then, utilize the comparison module to update the sample representation based on the pairwise node relationships in the instance-level graph. In this module, the author constructs multi-head relationships to help model fine-grained sample relationships to learn rich sample representations.
3) Next, instance-level graphs are compressed into class-level graphs to explicitly explore class-level visual knowledge.
4) In the calibration module , class-level message passing operations are performed according to the relationship between classes, and class-level knowledge representation is updated. Since the semantic word embedding of classes can provide rich prior knowledge, we combine it with class-level visual knowledge to construct multi-modal class knowledge representation before calibrating module message passing.
5) Finally, combine class-level knowledge representation with instance-level sample representation to guide the reasoning of query samples.

• Compare modules: multi-headed relationships for instance-level messaging
  For image $i$ , a deep CNN model is used as the skeleton to extract its visual features. (We follow the existing literature and combine the visual features of the image with its corresponding one-hot encoding as the initial node feature$v_i^{(0)}\in R^d$ . Since the labels of query samples are not available in inference generation, we set the elements of their one-hot encoding to 1/N, where N is the number of classes. )
  In each episode, we take the support set and query set samples as nodes to build the graph$G=(V^{(0)}，A^{\left(0\right)})$, where$V^{\left(0\right)}$ is the initial node characteristic matrix,$A^{\left(0\right)}$ is the initial adjacency matrix set representing the sample relationship. Existing work has shown thatvisual features always contain some concepts that can be grouped, that is, the feature dimensions of the same group represent similar knowledge. However, existing graph-based small sample learning methods usually directly use global visual features to calculate the similarity of samples to construct adjacency matrices, which cannot well represent fine-grained relationships. In this paper, we divide the visual features into$K$个块(即$V^{(l)}=[V_1^{(l)},V_2^{\left(l\right)}，\dots ，V_K^{\left(l\right)}]\in R^{r\times d})$, calculate the similarity of each block to explore the multi-head relationship of the sample (i.e.$K$ adjacency matrices$V_1^{(l)},V_2^{\left(l\right)}，\dots ，V_K^{\left(l\right)}\in R^{r\times r}$ ), where$r$ is the number of samples in each episode,$[\ast ,\ast ]$ is the splicing operation,$l$ is the llthin the figure$l$ layer generating matrix. Note that each block$V_i^{\left(l\right)}$The dimension of is $d/K$ . Also calculate the global relationship matrix A g ( l ) ∈ R r × r A^{(l)}_g \in R^{r \times r} basedon non-blocked visual features$A_g^{\left(l\right)}\in R^{r\times r}$ .
  We jointly utilize the global ($A_g^{(l)}$)Wadatou( { A i ( l ) } i = 1 K \{A^{(l)}_i \}^K_{i=1}{ Ai(l)​}i=1K​)关系(即$A^{(l)}=\{A_g^{\left(l\right)},A_1^{\left(l\right)}，\dots ，A_K^{\left(l\right)}\}$ ) propagates information across the instance-level graph to update the sample representation. In this way, we can more fully explore the relationships between samples and learn richer sample representations. in the$At\; layer\; l$ , we use the updated sample representation$V^{\left(l\right)}$ Construct a new adjacency matrix$A_g^{\left(l\right)}$Sum $A_i^{\left(l\right)}$, as shown below:
  
  Inspired by the success of TRPN in the few-shot classification task, we use the following matrix to mask the adjacency matrix:
  
  where $m$ and$n$ is$S\cup$Samples in$Q$$y_m$is the sample $m$ label. This ensures that for two samples of different categories, the higher the feature similarity, the lower the commonality in the message passing process. For two samples of the same category, the results are exactly opposite.
  

• Compression module: Class-level visual knowledge learning
  To obtain class-level knowledge representation, we compress the instance-level graph to generate a class-level graph, where nodes represent the visual knowledge of the class. For example, we compress the nodes in the instance-level graph into 5 clusters/nodes to obtain visual knowledge of classes in 5-way classification tasks. Specifically, we first use ground truth to supervise the generation of the allocation matrix, and then compress the samples according to the allocation matrix to obtain the class-level knowledge representation $V_c\in R^{r_1\times d}$ , where$r_1$Indicates the number of classes in each episode. For the sake of simplicity, this article will refer to $V^{\left(L\right)}$ and$A_g^{(L)}$Substitute into the standard graph neural network to calculate the distribution matrix $P\in R^{r\times r_1}$:
  
  其中，$W\in R^{d\times r_1}$Represents a trainable weight matrix to which softmax operations are applied row-wise. Distribution matrix PPEach element P uv P_{uv}in$P$$P_{uv}$Represents the node uu in the original graph$u$ is assigned to node vvin the class-level graph$The\; probability\; of\; v$ . When generating the distribution matrix$After\; P$ , we generate the initial class-level knowledge representation using the following equation:
  
  where$T$ represents the transpose operation. In a class-level graph, each node feature can be viewed as a weighted sum of node features with the same label in the instance-level graph. Through this approach, we obtain a class-level visual knowledge representation, which will help model the relationships between different classes in the calibration module.

• Calibration module: Class-level messaging with multi-modal knowledge
  Since class-word embeddings can provide information that may not be contained in visual content, we combine it with the generated class-level visual knowledge to build multi-modal knowledge representation. Specifically, we first leverage GloVe (pre-trained on a large text corpus with self-supervised constraints) to obtain $d_1$Dimensional semantic embedding. This article uses the Common Crawl version of GloVe, which is trained on 840B tokens. After obtaining the iiWord embedding of type$i$$e_i\in R^{d_1}$After that, use the mapping network $g:R^{d_1}\to R^d$ maps it to a semantic space with the same dimensions as the visual knowledge representation, that is,$z_i=g(e_i)\in R^d$ . Finally, we obtain the following multimodal class representation:
  
  where$Z\in R^{r_1\times d}$ is the semantic word embedding matrix. This results in a richer representation of class-level knowledge.
  Adjacency matrix of class-level graph ($A_c$) represents the relationship represented by the class, and its value represents the connectivity strength of the class pair. In this article, we calculate the adjacency matrix $A_c$and new class-level knowledge representation $v_c^{\prime \prime }$:
  
  其中$w^{\prime }\in R^{2d\times 2d}$ is a trainable weight matrix. In order for each sample to contain the class knowledge learned in (7), we utilize the assignment matrix to map the class knowledge back to the instance-level graph as follows where
  
  V$V_r\in R^{r\times 2d}$ represents the optimized features. Finally, V r V_ris connected in series$V_r$With $V^{\left(L\right)}$ combined to generate sample representation$V_f$。

3.3 Reasoning

To infer the class label of the query sample, we use $V_f$Calculate the corresponding adjacency matrix $A_f$As follows:

where $V_{f;m}$, $V_{f;n}$respectively represent $m$ samples and$n$ samples. $f_l:R^{3d\_}\to R^1$ is a mapping function. For each query example, we leverage the class label of the supporting example to predict its label:

where one-hot denotes one-hot encoder

3.4 Loss function

The overall framework of the proposed ECKPN can be optimized in an end-to-end form through the following loss function:

where λ0, λ1, and λ2 are hyperparameters, and the experimental settings are 1.0, 0.5, and 1.0. ${\mathcal{L}}_0$、$L_1$和$L_2$They are adjacency loss, assignment loss and classification loss. specifically,

• Adjacency Loss
  for each graph network layer l = { 1 , ... , L } l = \{1 , ... , L \}l={ 1，…, L } In the comparison module, we have multiple adjacency matrices$A_g^{(L)}$和 { A i ( L ) } i = 1 K \{A^{(L)}_i \}^K_{i=1} { Ai(L)​}i=1K​Used to support messaging between samples and query samples. Furthermore, in the above, we have the adjacency matrix $A_f$. To ensure that these adjacency matrices capture the correct sample relationships, we use the following loss function
  
  
  where $m$ and$n$ represents a node in the graph.

• Assignment Loss is to ensure the assignment matrix PP
  calculated in the compression module$P$ is able to correctly cluster samples with the same label, we utilize the following cross-entropy loss function:
  

• Classification Loss
  In order to constrain the proposed ECKPN to predict the correct query label, we use the following loss function
  
  where ${\mathcal{L}}_{ce}$Represents the cross-entropy loss function.

4. Some experimental results

• SOTA method performance comparison
  1) miniimagenet library
  
  2) tieredimagenet library
  
  3) cub and cifar-fs library
  

• Semi-supervised small sample classification performance
  

• Parameter analysis
  

• ablation experiment
  

5 Conclusion

1) There are two settings for graph-based meta-learning methods: direct inference and inductive . The direct inference method describes the relationship between the samples of the support set and the query set for joint prediction, and its performance is better than the inductive method. The inductive method can only learn the network based on the relationship between the support set and predict each query sample. Perform separate classification
2) This article belongs to the direct push method
3) This article proposes an end-to-end few-shot learning architecture based on graphs for the first time