【Paper Notes】KDD2019 | KGAT: Knowledge Graph Attention Network for Recommendation

Abstract

For better recommendation, not only the user-item interaction should be modeled, but also the relationship information should be taken into account

The traditional method factorization machine treats each interaction as an independent instance, but ignores the relationship between items (eg: the director of one movie is also the actor of another movie)

Higher-order relationship: connect two items with one or more link attributes

KG+user-item graph+high order relations—>KGAT

Recursively propagate the embedding of neighboring nodes (maybe users, items, attributes) to update the embedding of its own node, and use the attention mechanism to distinguish the importance of neighboring nodes

Introduction

$u_1$is the target user to whom the recommendation is to be provided. Yellow and gray circles denote important users and items discovered through higher-order relations but ignored by traditional methods.

For example, user $u_1$watched the movie $i_1$, the CF method focuses on also viewing $i_1$The history of similar users of $u_4$和$u_5$, while supervised learning focuses on the relationship with $i_1$have the same attribute $e_1$movie $i_2$, obviously, these two kinds of information are complementary for recommendation, but the existing supervised learning fails to unify the two, for example, here $i_1$Sum $i_2$r $r_2$The attributes are all $e_1$, but it fails to pass $r_3$reach $i_3$，$i_4$, because it treats them as independent parts and cannot take into account the high-order relationship in the data, for example, the users in the yellow circle watched the same director $e_1$Other movies of $i_2$, or the movie in the gray circle is also related to $e_1$There are other relationships. This is also important information for making recommendations.
u 1 ⟶ r 1 i 1 ⟶ − r 2 e 1 ⟶ r 2 i 2 ⟶ − r 1 { u 2 , u 3 } , u 1 ⟶ r 1 i 1 ⟶ − r 2 e 1 ⟶ r 3 { i 3 , i 4 } , \begin{array}{l} u_{1} \stackrel{r_{1}}{\longrightarrow} i_{1} \stackrel{-r_{2}}{\longrightarrow} e_{1} \ stackrel{r_{2}}{\longrightarrow} i_{2} \stackrel{-r_{1}}{\longrightarrow}\left\{u_{2}, u_{3}\right\}, \\ u_{ 1} \stackrel{r_{1}}{\longrightarrow} i_{1} \stackrel{-r_{2}}{\longrightarrow} e_{1} \stackrel{r_{3}}{\longrightarrow}\left\ {i_{3}, i_{4}\right\}, \end{array}u1​⟶r1​​i1​⟶−r2​​e1​⟶r2​​i2​⟶−r1​​{ u2​,u3​},u1​⟶r1​​i1​⟶−r2​​e1​⟶r3​​{ i3​,i4​},​

There is a problem

Harnessing this high-level information is challenging:

1) The number of nodes with high-order relationships with target users increases sharply as the order increases, which brings computational pressure to the model

2) Higher-order relations contribute unevenly to predictions.

To this end, the paper proposes the Knowledge Graph Attention Network (KGAT) model, which updates a node's embedding based on the embedding of its neighbors, and recursively performs this embedding propagation to capture high-order connections in linear time complexity. An attention mechanism is additionally employed to learn the weight of each neighbor during propagation.

GNN->KGAT

1. Recursive embedding propagation, using domain node embedding to update the current node embedding

2. Use the attention mechanism to learn the weight of each neighbor during propagation

advantage:

1. Compared with the path-based method, manual calibration of the path is avoided

2. Incorporate higher-order relationships directly into predictive models compared to rule-based methods

3. Model framework

3.1 Problem Definition

Input: collaborative knowledge graph $\mathcal{G}$，$G$ interacts with user-item data${\mathcal{G}}_1$And knowledge graph $G_2$composition

Output：user$u$ point 击 item$i$ general rate${\hat{y}}_{ui}$

Higher-order connections : Exploiting higher-order connections is crucial to perform high-quality recommendations. We will $L-$ order connections ($L$ - order connectivtiy) is defined as a multi-hop relation path:
$$e_0\stackrel{r_1}{⟶}{e}_1\stackrel{r_2}{⟶}...\stackrel{r_L}{⟶}{e}_L$$

3.2 Embedding Layer

The paper uses the TransR model in knowledge graph embedding. Its main idea is that different entities have different meanings under different relationships, so entities need to be projected into a specific relationship space. If $h$ and$t$ has$r$ relation, then they are in$The\; expression\; of\; the\; r$ relational space should be close, otherwise it should be far away, and the expression of the formula is:
$$e_h^r+e_r\approx e_t^r$$
Here $e_h,e_t\in {\mathbb{R}}^d$,${\mathbf{e}}_r\in R^k$ is$h,t,\; embedding\; ofr$ .

它的得分为：
$$g(h,r,t)=||{\mathbf{W}}_r{e}_h+e_r-{\mathbf{W}}_r{e}_t|{|}_2^2$$
其中$W_r\in R^{k\times d}$ is the relationrrTransformation matrix for$r$$Projection\; of\; d-$ dimensional solid space to$in\; the\; k-$ dimensional relational space. $g(h,r,The\; lower\; the\; value\; of\; t)$ , the higher the probability that the triplet is true.

Finally, use pairwise ranking loss to measure the effect:
$${\mathcal{L}}_{KG}=\sum _{(h,r,t,t^{{}^{\prime }})\in \tau }-ln\sigma (g(h,r,t^{{}^{\prime }})-g(h,r,t))$$
The meaning of this formula is to make the value of the negative sample minus the value of the positive sample as large as possible. The choice of negative samples is to use $t$ is randomly replaced by one of the other.

3.3 Attentive Embedding Propagation Layers

Information dissemination

Consider entity $h$ , we use${\mathcal{N}}_h=\{(h,r,t)|(h,r,t)\in G\}$ for those ending in$h$ is a triplet of head entities. Calculate$ego$ -network:
$${\mathbf{e}}_{N_h}=\sum _{(h,r,t)\in {\mathcal{N}}_h}\pi (h,r,t){\mathbf{e}}_t$$
$\pi (h,r,t)$ means that in relation$r$ from$t$ to$The\; amount\; of\; information\; for\; h$ .

knowledge perception attention

Weight $\pi (h,r,t)$ is achieved through the attention mechanism
$$\pi (h,r,t)=({\mathbf{W}}_r{\mathbf{e}}_t{)}^{T}tanh({\mathbf{W}}_r{e}_h+e_r)$$
Here use $tanh$ as an activation function can make the closer${\mathbf{e}}_h$和$e_t$have higher attention scores. Using $softmax$归一化：
$$\pi (h,r,t)=\frac{exp\left(\pi \right(h,r,t))}{{\sum }_{(h,r^{{}^{\prime }},t^{{}^{\prime }})\in {\mathcal{N}}_h}exp\left(\pi \right(h,r^{{}^{\prime }},t^{{}^{\prime }}))}$$
Finally, by virtue of $\pi (h,r,t)$ We can know which neighbor nodes should be given more attention.

information aggregation

will eventually $The\; representation\; of\; h$ in the solid space${\mathbf{e}}_h$And its ego-network representation $e_{{\mathcal{N}}_h}$aggregated as $h$新这些：
$$e_h^{(1)}=f({\mathbf{e}}_h,e_{N_h})$$
$f(\cdot )$ in the following ways:

1. GCN Aggregator:
   $$f_{GCN}=LeakyReLU\left(W\right(e_h+{\mathbf{e}}_{{\mathcal{N}}_h}))$$
2. GraphSage Aggregator:
   $$f_{GraphSage}=LeakyReLU\left(W\right(e_h||{\mathbf{e}}_{N_h}))$$
3. Bi-Interaction Aggregator：
   $$f_{Bi-Interaction}=LeakyReLU(W_1({\mathbf{e}}_h+{\mathbf{e}}_{{\mathcal{N}}_h}))+LeakyReLU\left(W_2\right(e_h\odot e_{N_h}))$$

Higher order propagation:

We can further stack more propagation layers to explore higher-order connectivity information and collect information propagated from higher-hop neighbors, so in $l$步中：
$${\mathbf{e}}_h^{(l)}=f({\mathbf{e}}_h^{(l-1)},e_{{\mathcal{N}}_h}^{(l-1)})$$
Let $e_{N_h}^{(l-1)}={\sum }_{(h,r,t)\in {\mathcal{N}}_h}\pi (h,r,t){\mathbf{e}}_t^{(l-1)}$, $e_t^{(l-1)}$Also through the above steps from ${\mathbf{e}}_t^0$owned.

3.4 Prediction layer

While executing LLAfter layer$L$$u$的多山运动：{ eu ( 1 ) , . . . , eu ( L ) } \{\mathbf e_u^{(1)},...,\mathbf e_u^{(L)} \}{ eu(1)​,...,eu(L)​} , and item$i$的多山可以：$\{e_i^{\left(1\right)},..,e_i^{\left(L\right)}\}$

电影电影电影，即：
$${\mathbf{e}}_u^{\ast }=e_u^{(0)}||...||{\mathbf{e}}_u^{(L)},e_i^{\ast }=e_i^{\left(0\right)}||...||e_i^{\left(L\right)}$$
Finally, the correlation score is calculated by inner product:
$$\hat{y}(u,i)={e_u^{\ast }}^{That\text{'}s}{it}_i^{\ast }$$

3.5 Loss function

BPR loss:
$${\mathcal{L}}_{CF}=\sum _{(u,i,j)\in O}-ln\sigma (\hat{y}(u,i)-\hat{y}(u,j))$$
其中$O=\{(u,i,j)|(u,i)\in {\mathcal{R}}^{+},(u,j)\in R^{-}\}$，$R^{+}$ Represents a positive sample,$R^{-}$ represents a negative sample.

Definition:
$${\mathcal{L}}_{KGAT}=L_{KG}+L_{CF}+\lambda ||\Theta |{|}_2^2$$