[Study Notes] Intensive Reading of DeepWalk Graph Neural Network Paper

Reference materials: DeepWalk [Intensive reading of graph neural network papers]

word2vec

相关论文：
Efficient Estimation of Word Representations in Vector Space
Distributed Representations of Words and Phrases and their Compositionality

Random walk Ramdom Walk brief introduction

A sequence can be sampled through a random walk.

A sequence is like a sentence, and a node is like a word.

The assumption of random walk is similar to word2vec, assuming that adjacent words should be similar. Therefore, a skip-gram problem can be constructed, the central node is input, and the surrounding neighboring nodes are predicted. This will fully apply word2vec.

Deepwalk official ppt introduction

https://dl.acm.org/doi/10.1145/2623330.2623732

Core idea: random walk = sentence

Advantages of Deepwalk

• Scalable - Online learning algorithm, no need to start training from scratch when new data comes.
• You can treat random walks as sentences and directly apply the language model in NLP
• It performs well on sparsely labeled graph classification tasks.

language model

In the field of NLP, there is a phenomenon called "word frequency": some words appear particularly frequently, while others appear less frequently.
In graphs, especially scale-free graph networks, there is a similar phenomenon: "Vertex frequency": some websites are visited very frequently, and some are not.

process

1. input graph
2. Sample a random walk sequence
3. Training word2vec with random walk sequence
4. Use hierarchical softmax
5. Get the vector representation of each node

2. Random walk

• Each node produces $\gamma$ random walk sequences.
• The maximum length of each random walk is $t$。
• Select the next node of a node with equal probability
• 如：$v_{46}\to v_{45}\to v_{71}\to v_{24}\to v_5\to v_1\to v_{17}$

3. Use random walk sequence to construct skip-gram task and train word2vec

4. Use hierarchical softmax

Learning parameters: node representation, classifier weights
using stochastic gradient descent while optimizing

Evaluate

Attribute prediction (node ​​classification problem)

This is a sparse labeling problem. Compare Deepwalk with spectral clustering, edge cluster, modularity, wvRN.

• BlogCatalog
  
• Flicker
  

deepwalk performs very well, especially when there are very few labels.

Can be parallelized

Parallelism does not affect representation quality.

Outlook

• Streaming: No information about the entire graph is required. Dynamic updates.
• "Non-random" walk: can carry a certain tendency.
• Pictures and language complement each other, and breakthroughs in the two fields can learn from each other.
  

Intensive reading of papers

Dataset: Karate Club

problem definition

$G=(V,E)$
$G_L=(V,E,X,Y)$
$X\in {\mathbb{R}}^{|V|\times S}$ : Each node has S-dimensional features
$Y\in R^{|V|\times |Y|}$ : Each node has$Y$ tags

Task: relational classification (does not satisfy the assumption of independent and identical distribution)
Goal: Learn $X_E\in R_r^{|V|\times d}$:d is the dimension after word embedding

Embedding reflecting connection information + characteristics reflecting the node itself => Machine learning classification (fraud detection)

Features you want to learn

• Adaptability: Online Learning Algorithms
• Reflect community clustering information: similar nodes in the original image are still close after embedding
• Low dimensionality: prevent overfitting
• Continuous: Convenient to fit smooth decision boundaries

3.1 Random walk

Starting point: $v_i$
Random walk: ${\mathcal{W}}_{v_i}$：$W_{v_i}^1,W_{v_i}^2,\dots ,W_{v_i}^k$: The upper right corner represents step k.
Random walk has been used for content recommendation, community detection, and as a method of similarity measurement.
Random walks are also the cornerstone of some output sensitive algorithms (the entire graph must be traversed at least once).
Advantages of random walks:
1. Parallel sampling to generate random walk sequences
2. Online learning: When there are new nodes and new relationships in the network, there is no need to recalculate the entire graph information, only the new nodes and new relationships are sampled. , iterative online incremental training is enough.

3.2 Power law distribution (Power laws)

• Random network
  If it is a random network, then the degree of nodes is generally small, and there is no distribution where the degree of some nodes is much larger than that of other nodes, which
  roughly presents a normal curve.
  

• Scale-free network
  

• Zipf's law:
  The word frequency of a word is inversely proportional to the constant power of the word frequency ranking. That is, only a very small number of words (nodes) are frequently used.

3.3 Language model

The goal of a language model is to estimate the likelihood of occurrence of a specific sequence of words. More formally:
given a word sequence $W_1^n=\left(w_0,w_1,\cdots ,w_n\right)$ , hoping to maximize the probability:
$$\mathrm{Pr}\left(w_n\mid w_0,w_1,\cdots ,w_{n-1}\right)$$

That is, the previous $n$ words, predict$n+Probability\; of\; 1$ word.

Thesis uses i − 1 i-1 before$i-1$ node, predict the$n$ nodes.
Introduce$F:v\in V\mapsto {\mathbb{R}}^{|V|\times d}$ mapping, mapping nodes to vectors through table lookup.
So the problem is transformed into (predict the i-th node using the embedding of the first i-1 nodes):
$$\mathrm{Pr}\left(v_i\mid \left(F\left(v_1\right),Phi\left(v_2\right),\cdots ,Phi\left(v_{i-1}\right)\right)\right)$$
But converting it into conditional probability will get smaller and smaller as the multiplication increases, causing the probability to become very small when you travel very far.

Reference word2vec

word2vec is a self-supervised model, and the order of surrounding words is irrelevant.

skip-gram的损失函数： minimize ⁡ Φ    − log ⁡ Pr ⁡ ( { v i − w , ⋯   , v i + w } \ v i ∣ Φ ( v i ) ) \underset{\Phi}{\operatorname{minimize}}\ \ -\log \operatorname{Pr}\left(\left\{v_{i-w}, \cdots, v_{i+w}\right\} \backslash v_i| \Phi\left(v_i\right)\right) Phiminimize​  −logPr({ vi−w​,⋯,vi+w​}\vi​∣Φ(vi​))

The order of the graph generated by a random walk is meaningless.
The model is smaller, inputs one node at a time, and predicts surrounding nodes.

4.Method

4.2 Algorithm: deepwalk

The algorithm is divided into two parts:
1. Random walk generator
2. Update step

• deepwalk pseudocode

• skipgram pseudo code
  

You can set the probability that the random walk sequence will be transmitted back to the starting node after walking a certain way, but the results of the preliminary experiments have no obvious impact.

4.2.1 skipgram

4.2.2 Hierarchical softmax

It turns out that doing softmax directly is too expensive to calculate the partition function.

Therefore, deepwalk needs to train two sets of weights:
1. Word embedding matrix
2. N-1 logistic regression weights (N leaf nodes have N-1 logistic regression)

• Overall flow chart
  

4.2.3 Optimization

The parameters are the two sets of weights mentioned above, the size is $O(d|V|)$

4.3 Multi-threaded asynchronous parallelism

4.4 Algorithm variants

4.4.1 streaming

Ensure that the learning rate remains unchanged and small.

4.4.2 Non-random walks

User interactions tend to be biased. This not only takes into account the existence of the connection, but also the weight of the connection. Sentences can actually be viewed as biased sampling sequences.

5.Experimental design

data set:

5.2 Comparison algorithm

6. Experiment

6.1 Multi-category node classification

$T_R$: Proportion of labeled nodes
Evaluation index: $Macro-F1$ (average F1 for each category) and$Micro-F1$ (the overall TP, FN, FP, TN calculates the overall F1)
Here, one-vs-rest logistic regression is used to implement the classifier.

You can find reference materials

The effect is shown below.
The results shown in the figure:
1. $T_R$Will affect the optimal dimension d, $T_R$The larger the value, the larger the optimal d is.
2. $The\; larger\; \gamma$ , the better the effect, but there is a margin
3.$T_R$The bigger the value, the better the effect.
4. Pictures of different sizes, different $The\; relative\; influence\; of\; \gamma$ is consistent

7.Related work

1. Embedding is obtained through self-supervised (unsupervised) learning.
2. Only consider the connection information in the graph. Subsequently, embedding and annotation can be used to train a supervised classification model.
3. Online learning.