【Paper Notes】(DGCN-HN) Deep Graph Convolutional Networks with Hybrid Normalization for Accurate and …

This article was published on KDD 2021

On the basis of LR-GCCF and LightGCN, it combines left normalization (assign equal normalization to different neighbors, PinSAGE) and symmetric normalization (assignment of small weights to neighbors with large degree, LightCGN)

1. Intro

1.1 Not enough layers

The GCN-based model achieves the best performance in the shallow layer, which does not use high-order signals

Add residual connection and overall connection (holistic connection).

The proposed model can deepen the graph convolution to 8 layers.

1.2 Use fixed normalization rules

The GCN-based model uses the same normalization rule for aggregating neighborhood information, making the importance of neighbors equally important

• left normalization (left normalization), assign the same normalization to different neighbors
• Symmetric normalization (symmetric normalization), assigning smaller weights to popular neighbors and larger weights to unpopular neighbors

1. Using fixed normalization rules can lead to suboptimal results. The article explores the effects of left normalization and symmetric normalization from the perspective of accuracy and diversity .
2. Nodes are inherently different, requiring different normalizations to aggregate neighborhood information. A new model is proposed: Hybrid Normalized Deep Convolutional Network (DGCN-HN).
   Hybrid normalization layer and simplified attention network to more flexibly model neighbor importance by adaptively fusing information from different normalization rules

• Shallow GCN cannot use high-order signals, for u 2 u_2 who like electronic products$u_2$Recommending drones requires high-level signals.
• Use left normalization for all nodes, the popularity of IPhone will seriously affect $u_1$of interest, $u_1$Buying an IPhone is just a need, and the interest is not in electronic products, because $u_1$Buy paper stationery frequently.
• Using symmetric normalization for all nodes, the popular IPhone cannot provide the interest of "electronic products", $u_2$和$u_3$interests are difficult to characterize.

2. METHOD

Consists of three main components:

• Design of Deep Graph Convolutional Network Recommendations with Residual and Holistic Connections
• Introducing a hybrid normalization layer for flexible modeling of neighbor importance, combining left normalization and symmetric normalization
• A simplified attention network is proposed to adaptively fuse different normalization methods to represent information

2.1 Deep Graph Convolutional Network for Recommendation

The linear aggregation of each layer can be expressed as a residual connection, the formula is as follows:
$$h_u^{(l+1)}=\sum _{v\in N_u}{\tilde{A}}_{uv}{h}_v^{(l)}+h_u^{\left(l\right)},h_u^{(0)}=e_u$$

$$h_v^{(l+1)}=\sum _{u\in N_v}{\tilde{A}}_{vu}{h}_u^{\left(l\right)}+h_v^l,h_v^{\left(0\right)}=e_v$$

After the aggregation of the L layer, the representations of all layers are integrated using the overall connection, and the model uses the element-based average aggregation as the fusion strategy of the last layer. The formula is as follows:
$$e_u^{\ast }=\frac{1}{L+1}\sum _{i=0}^L{h}_u^{(i)}$$
$$e_v^{\ast }=\frac{1}{L+1}\sum _{j=0}^L{h}_v^{(j)}$$

(The role of this part is to be discussed, see the final experiment)

2.2 Hybrid Normalization for Flexible Modeling of Neighbor Importance

Taking the user node as an example, the formula for symmetric normalization is as follows:

$$h_u^{(l+1)}=\sum _{v\in N_u}\frac{1}{\sqrt{\left|N_u\right|}\sqrt{|N_v|}}{\mathbf{h}}_v^{(l)}+h_u^{\left(l\right)}$$
For example,
$$h_u^{(l+1)}=\sum _{v\in N_u}\frac{1}{|N_u|}{h}_v^{\left(l\right)}+h_u^{\left(l\right)}$$

2.3 Simplified Attention Network for Adaptive Combination

In order to combine the advantages of the two normalizations, the simplest is the average sum, a little more complicated, weighted average, plus Attention

$h_{N_u,LN}^{\left(l\right)}$和$h_{N_u,SN}^{\left(l\right)}$, which means left normalization and symmetric normalization. Taking the user node as an example, the node update formula is as follows:
$$h_{N_u,LN}^{(l+1)}=\sum _{v\in N_u}\frac{1}{\left|N_u\right|}{h}_v^{(l)}$$

$$h_{N_u,SN}^{(l+1)}=\sum _{v\in N_u}\frac{1}{\sqrt{|N_u|}\sqrt{|N_v|}}{h}_v^{\left(l\right)}$$

$${\mathbf{h}}_u^{(l+1)}=h_u^{(l)}+a_{in,LN}^{(l+1)}{h}_{N_u,LN}^{(l+1)}+a_{u,SN}^{(l+1)}{h}_{N_u,SN}^{(l+1)}$$

任何，$a_{u,\ast }^l$Denotes the normalized attention scores of different layers.

When calculating the attention score, two aspects are considered:

• Neighbor's self-information
• Similarity between Center Representation and Neighbor Representation

Let us consider the following:
$$z_{u,\ast }^{(l+1)}={\mathbf{W}}_1^{(l)}p\left({\mathbf{W}}_2^{\left(l\right)}\left({\mathbf{h}}_{N_{u,\ast }}^{(l+1)}+{\mathbf{h}}_{N_u,\ast }^{(l+1)}\odot h_u^{\left(l\right)}\right)\right)$$

Among them, $z_{u,\ast }^{(l+1)}$Is the attention score before normalization, $W_1^{(l)}\in {\mathbb{R}}^{1\times d^t},W_2^{\left(l\right)}\in R^{d^t\times d^l}$ is the feature transformation matrix,$\sigma$ is the activation function,$d^t$ is the dimensionality of the hidden layer of the attention network. $\odot$ represents the Hadamard product, and the elements at the corresponding positions are multiplied.

As a result, the training failed to converge in the experiment, and a simple one was replaced . The feature transformation matrix and activation function were removed, and average aggregation was used to obtain a simplified attention network as follows:
$$z_{u,\ast }^{(l+1)}=\mathrm{ave}\left({\mathbf{h}}_{N_u,\ast }^{(l+1)}+{\mathbf{h}}_{N_u,\ast }^{(l+1)}\odot h_u^{(l)}\right)$$
and then use the Softmax function to get$\alpha$， 公式 as follows：
$$a_{u,\ast }^{\left(l\right)}=\frac{exp(z_{u,\ast }^{\left(l\right)})}{{\sum }_{k\in LN,SN}exp(z_{u,k}^{\left(l\right)})}$$
The loss function is BPR loss

3. EXPERIMENTS

• The simplest definition of coverage is the ratio of the items recommended by the recommendation system to the total items. A higher coverage rate indicates that the model can generate recommendations for more items, thereby promoting the mining of long-tail effects.
  The upper bound of Coverage@k is 1. The bigger the better, the more items are recommended to users.
• Entropy@K focuses on assignment recommendations for different items. It calculates the entropy value based on the number of recommendations in different rounds. The larger the entropy value, the better the diversity, and the higher the heterogeneity of recommended items.

The article also takes into account the diversity of recommended results, the results are as follows:

The effect of different normalization schemes on GCN:

As can be seen

• When using a single normalization, left normalization achieves the best diversity, symmetric normalization achieves the best accuracy, and right normalization performs worst in all respects
• In hybrid normalization, the combination of left normalization and symmetric normalization achieves the best accuracy and diversity

Ablation experiment:

• The introduced simplified attention network improves accuracy and increases diversity
• Hybrid normalization has a significant performance improvement on both datasets, especially for the improvement of Coverage

Summarize:

1. The article explores the effects of left normalization and symmetric normalization from the perspective of accuracy and diversity. The author can propose this idea because he jumped out of the GCN model itself, considered the difference between recommendation and other tasks, and connected it with reality.

2. According to the author's experiment, even if only the residual is added, the effect is greatly improved, but this is contrary to our own previous experiment.