2021-IJCA-Masked Label Prediction Unified Message Passing Model for Semi-Supervised Classification

GNNs form feature propagation through neural networks to make predictions, while LPA uses label propagation across graph adjacency matrices to obtain results. However, there is still no efficient way to directly combine these two algorithms.

Model

The author uses Graph Transformer, combined with label embedding to combine the above features and label propagation, to build a UniMP model. Furthermore, a masked label prediction strategy is introduced to train our model to prevent the label leakage problem.

Method
Known fixed node features $H^{(l)}=\left\{h_1^{(l)} , h_2^{(l)}, \ldots, h_n^{(l)}\right\}$ ,计算从 j 到 i 的每条边的多头注意力:
$\begin{gathered} q_{c, i}^{(l)}=W_{c, q}^{(l)} h_i^{(l)}+b_{c, q}^{(l)} \\ k_{c, j}^{(l)}=W_{c, k}^{(l)} h_j^{(l)}+b_{c, k}^{(l)} \\ e_{c, i j}=W_{c, e} e_{i j}+b_{c, e} \\ \alpha_{c, i j}^{(l)}=\frac{\left\langle q_{c, i}^{(l)}, k_{c, j}^{(l)}+e_{c, i j}\right\rangle}{\sum_{u \in N(i)}\left\langle q_{c, i}^{(l)}, k_{c, u}^{(l)}+e_{c, i u}\right\rangle} \end{gathered}$
其中， $\langle q, k\rangle=\exp \left(\frac{q^T k}{\sqrt{d}}\right)$ is a power level dot product function, $d$ is the hidden size of each head. for the $c$ head attention, first source feature $h^{(l)}_i$ and distant features $h^{(l)}_j$ Convert to query vector $q_{c, i}^{(l)} \in \mathrm{R}^{\mathrm{d}}$ and key vector $k_{c, j}^{(l)} \in \mathrm{R}^{\mathrm{d}}$ use different trainable parameters $W_{c, q}^{(l)}, W_{c, k}^{(l)}, b_{c, q}^{(l)}, b_{c, k}^{(l)}$
Provided edge features $e_{c,ij}$ will be encoded and added to the key vector as additional information for each layer. from a distance after getting the attention of the figure's bulls $j$ to source $i$ 进行消息聚合：
$\begin{gathered} v_{c, j}^{(l)}=W_{c, v}^{(l)} h_j^{(l)}+b_{c, v}^{(l)} \\ \hat{h}_i^{(l+1)}=\|_{c=1}^C\left[\sum_{j \in N(i)} \alpha_{c, i j}^{(l)}\left(v_{c, j}^{(l)}+e_{c, i j}\right)\right] \end{gathered}$
where || is the concatenation operation of c head attention. The multi-head attention matrix replaces the original normalized adjacency matrix as the transition matrix for message passing.

然后是残差连接，防止模型过度平滑
$\begin{gathered} r_i^{(l)}=W_r^{(l)} h_i^{(l)}+b_r^{(l)} \\ \beta_i^{(l)}=\operatorname{sigmoid}\left(W_g^{(l)}\left[\hat{h}_i^{(l+1)} ; r_i^{(l)} ; \hat{h}_i^{(l+1)}-r_i^{(l)}\right]\right) \\ h_i^{(l+1)}=\operatorname{RELU}\left(\operatorname{LayerNorm}\left(\left(1-\beta_i^{(l)}\right) \hat{h}_i^{(l+1)}+\beta_i^{(l)} r_i^{(l)}\right)\right) \end{gathered}$

多头平均，删除线性变换
$\begin{gathered} \hat{h}_i^{(l+1)}=\frac{1}{C} \sum_{c=1}^c\left[\sum_{j \in N(i)} \alpha_{c, i j}^{(l)}\left(v_{c, j}^{(l)}+e_{c, i j}^{(l)}\right]\right. \\ h_i^{(l+1)}=\left(1-\beta_i^{(l)}\right) \hat{h}_i^{(l+1)}+\beta_i^{(l)} r_i^{(l)} \end{gathered}$

Label Embedding and Propagation

$H^{(l)}=\left((1-\beta) A^*+\beta I\right)^l\left(X+\widehat{Y} W_d\right) W^{(1)} W^{(2) \ldots W^{(l)}}=\left((1-\beta) A^*+\beta I\right)^l X W+\left((1-\beta) A^*+\beta I\right)^l \widehat{Y} W_d W$
where the model is approximately decomposed into feature propagation $\left((1-\beta) A^*+\beta I\right)^l XW$ 和标签传发 $\left((1-\beta) A^*+\right.\beta I)^l \widehat{Y} W_d W$

Masked label prediction Supervised training of its model parameters $\theta$
given X and A $θ$ :
$\underset{\theta}{\arg\max} \log p_\theta(\width{Y}\mid X, A)=\sum_{i=1}^{\width{V}} \log p_\theta\left(\width{y_i}\mid X, A\right)$

Graph Neural Code

import torch
import torch.nn.functional as F
from torch_geometric.nn import TransformerConv
class Transformer(torch.nn.Module):
    #初始化
    def __init__(self, hidden_channels):
        super(Transformer, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = TransformerConv(10, hidden_channels,dropout=0.5) #GCNConv更换为TransformerConv
        self.conv2 = TransformerConv(hidden_channels, 2,dropout=0.5)#GCNConv更换为TransformerConv
	#前向传播
    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        #注意这里输出的是节点的特征，维度为[节点数,类别数]
        return x
    
x = torch.randn(4, 10)
edge_index = torch.tensor([[0,0,0,1,1,2],[1,2,3,2,3,3]])

model = Transformer(36)
res = model(x, edge_index)
print(res)