[GCN] GCN study notes 1

Spectral domain graph convolution

Spectral domain graph convolution

convolution

Convolution definition

Convolution is an important operation in analytical mathematics. Suppose $f (x)$ 和 $g (x)$ 是 $\mathbb{R}$ , the continuous form of convolution is defined as follows:

$\int_{-\infin} ^{\infin} f(\tau) g(x - \tau) d\tau {\tag{ 1}}$

Different functions and different convolution kernels can produce different convolution results.

Discrete space convolution

$y_n = x * w = \sum_{k=1}^k w_k x_{n-k} \tag{2}$
Please add image description

Introduction to graph convolution

Classic convolutional networks cannot handle graph-structured data. Current ideas for implementing graph convolution:

Spectral domain graph convolution
spatial graph convolution

convolution theorem

The Fourier transform of the convolution of two signals in the spatial domain is equal to the dot product of the Fourier transform of the two signals in the frequency domain. The dot multiplication (element-wise multiplication) refers to the dot product of the two matrices, Multiply corresponding elements in a vector or sequence. That is:
$\mathcal{F} [f_1(t) \star f_2(t)] = F_1(w) . F_2(w) \tag{3}$

$f_1(t)$ 和 $f_2(t)$ are two signals in the airspace
$F_1(w)$ Sum $F_2(w)$ This is two signals in the area
$\star$ represents the convolution operation
$\mathcal{F}$ represents Fourier transform
$.$ represents the product operation

can also be written as:
$f_1 (t)\star f_2(t) = \mathcal{F}^{-1} [F_1(w) . F_2(w)] \tag{4}$

$\mathcal{F}^{-1}$ represents the inverse Fourier transform

Implementation ideas of spectral domain graph convolution

General signal $x$ Japanese paper core $w$ Perform Fourier transform to obtain the frequency domain signal $X$ Sum $IN$
General area signal $X$ Sum $W$ Perform product operation to obtain frequency domain signal $AND$
General area signal $Y$ Perform inverse Fourier transform to obtain the image signal $and$
$y$ Immediate delivery signal $x$ Japanese paper core $w$ Paperback result

How to define the Fourier transform on a graph

Based on graph theory, the Fourier transform on the graph is defined using the Graph Fourier Transform (GFT). The definition of GFT is as follows:
$\mathcal{F}_G(x) = U^T x \tag{5}$

$x$ This is the signal
$U$ is the eigenvector matrix of the graph

Photograph component summative formalism:
$\sum_{k=1}^N u_l(i)\hat{x}(\lambda_l) \tag{6}$

$x (i)$ Display signal $x$ Current point $i$ 处的值
$u_l(i)$ Display image's special direction amount square $U$ 目次 $l$ Column number $i$ single element
$\hat{x}(\lambda_l)$ Display signal $x$ A special expedition $\lambda_l$ value at

Laplacian Matrix

The Laplacian matrix is an important concept in graph signal processing. It is the frequency domain representation of the graph signal. The Laplacian matrix is defined as follows:
$\tag{7}$

$D$ 了矩阵， $D_{ii} = \sum_j A_{ij}$
$A$ is the connection, $A_{ij} = 1$ Display point $i$ harmony point $j$ There is, or is not $A_{ij} = 0$
$L$ is the Laplacian matrix, otherwise $L_{ij} = 0$ ，$。

Laplacian matrix calculation example
Insert image description here

$L_{ij} = -1$ Display point $i$ harmony point $j$ 之间边
$L_{ii} = D_{ii}$ The Laplacian diagonal value is equal to the degree matrix diagonal value
$L_{ij} = 0$ Display point $i$ harmony point $j$ There has been a loss

Properties of Laplacian Matrix

The Laplacian matrix is a positive semidefinite matrix, that is, $x^T L x \geq 0$ , inside $x$ 是任意向量。证明过程：
$x^T L x = x^T D x - x^T A x = \sum_{i=1}^N d_i x_i^2 - \sum_{i,j=1}^N a_{ij} x_i x_j = \frac{1}{2} \sum_{i,j=1}^N a_{ij} (x_i - x_j)^2 \geq 0 \tag{8}$

A symmetric matrix of order n must have n linearly independent real eigenvalues
The eigenvectors corresponding to different eigenvalues of the symmetric matrix are orthogonal to each other. The matrix composed of these rescued eigenvectors is an orthogonal matrix.
The eigenvalues of the Laplacian matrix are all non-negative, and at least one eigenvalue is 0
The eigenvector with the eigenvalue 0 of the Laplacian matrix corresponds to the number of connected components of the graph, that is, $The number of zero eigenvalues of L$ is equal to the number of connected components of the graph.

Spectral Decomposition of Laplacian Matrix

Eigen decomposition, also known as spectral decomposition, is a decomposition in linear algebra, which decomposes a matrix into the form of eigenvectors and eigenvalues. The spectral decomposition of the Laplacian matrix is as follows:
$\Lambda U^T \tag{9}$

$U$ is the eigenvector matrix of the Laplacian matrix
$\Lambda$ is the eigenvalue matrix of the Laplacian matrix
$U^T$ 是 $Transpose matrix of U$

After decomposing the Laplace spectrum, an n-order symmetric matrix must have n linearly independent eigenvectors that are orthogonal to each other. The matrix composed of these orthogonal eigenvectors is an orthogonal matrix. Therefore, the eigenvector matrix of the Laplacian matrix $U$ Correctional exchange, immediately $U^T U = I$ , inside $I$ This is the square measure.

Laplacian matrix and Laplacian operator

The Laplacian matrix and the Laplacian operator are two different concepts, but there is a certain connection between them. The definition of the Laplacian operator is as follows:
$\Delta f = \nabla \cdot \ nabla f = \sum_{i=1}^n \frac{\partial^2 f}{\partial x_i^2} \tag{10}$

$\nabla$ is the gradient operator
$\print \cdot$ Koresandanshi
$\Delta$ is the Laplacian operator
$f$ is function
$x_i$ is the first $i$ self-change amount
$n$ is the number of independent variables
$\frac{\partial^2 f}{\partial x_i^2}$ is a function $f$ self-change amount $x_i$ The second-order partial derivative of
$\sum_{i=1}^n \frac{\partial^2 f}{\partial x_i^2}$ is a function $f$ The sum of second-order partial derivatives with respect to all independent variables
$\Delta f$ 是函数 $Laplacian operator of f$

The Laplacian operator of the graph signal is defined as follows:
$\Delta_G f = U \Delta f = U \nabla \cdot \nabla f = U \sum_{i=1}^n \frac{\partial^2 f}{\partial x_i^2} \tag{11}$

$\Delta_G f$ This is the signal $Laplacian operator of f$
$U$ is the eigenvector matrix of the Laplacian matrix
$\Delta f$ 是函数 $Laplacian operator of f$
$\fold \cdot \fold f$ 是函数 $Laplacian operator of f$

How to get the above formula? For the Laplacian operator of a two-dimensional image, we can write it in the following form:
$\ Delta f = \frac{\partial^2 f}{\partial x^2} + \frac{\partial^2 f}{\partial y^2} \tag{12}$
The discrete form of the Laplacian operator can be written in the following form:
$\Delta f = \frac{\partial^2 f}{\partial x^2} + \frac{\partial^2 f}{\partial y^2} = \frac{f(x+1,y) + f(x-1,y) - 2f(x,y)} {h^2} + \frac{f(x,y+1) + f(x,y-1) - 2f(x,y)}{h^2} \tag{13}$
inside $h$ is the step size. Write the above formula into matrix form:
$\Delta f = \frac{1}{h^2} \begin{bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end{bmatrix } \begin{bmatrix} f(x-1,y) \\ f(x,y) \\ f(x+1,y) \end{bmatrix} + \frac{1}{h^2} \begin {bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} f(x,y-1) \\ f (x,y) \\ f(x,y+1) \end{bmatrix} \tag{14}$

We set h = 1 here and write the above equation in matrix convolution form:

$\Delta f(x) = \begin{bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end{bmatrix}\star f(x)\tag{16}$
It can be seen that the Laplacian operator of a two-dimensional image, It is equal to the sum of the differences between surrounding nodes and its own node. For the Laplacian operator of a graph signal, the Laplacian operator of a node can be defined as the sum of the differences between its connected nodes and its own node. We can write it in the following form:
$\ Delta_G f = \sum_{j=1}^N A_{ij} (f_i - f_j) = \sum_{j=1}^N A_{ij} f_i - \sum_{j=1}^N A_{ij} f_j = \sum_{j=1}^N A_{ij} f_i - \sum_{j=1}^N A_{ji} f_j \tag{17}$

$A_{ij}$ Display point $i$ harmony point $The weight of the edge between j$
$f_i - f_j)$ Display point $i$ harmony point $The difference between j$

写成矩阵形式有：
$\begin{aligned} \Delta_G f =& \begin{bmatrix} \Delta f_1 \\ \Delta f_2 \\ \vdots \\ \Delta f_N \end{bmatrix} = & \begin{bmatrix} A_{11}f_1 - \sum_{j=1}^N A_{1i} f_j \\ \vdots \\ A_{nn}f_n - \sum_{j=1}^N A_{Ni} f_j \\ \end{bmatrix} = & Df - Af =Lf\tag{18} \end{aligned}$

$L$ is the Laplacian matrix
True signal $f$ Left multiplied by a Laplacian matrix can get the signal $The Laplacian of f$ .

Graph Fourier Transform

Signal representation on the diagram

The signal on the graph is generally expressed as a vector. Assume there are n nodes. The signal on the picture is recorded as:
$\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{ bmatrix} \tag{19}$

$x_i \in \mathbb{R}$ Display point $i$ Upper signal.

Classical Fourier Transform

The definition of classical Fourier transform is as follows:
The Fourier transform formula is as follows:
$X(\omega) = \int_{-\infin}^{\infin} x(t) e^{-j \omega t} dt \tag{20}$
Official information below:
$X(\omega) = \sum_{n=0}^{N-1} x(n) e ^{-j \omega n} \tag{21}$

The classic Fourier transform represents a function as a linear combination of several orthogonal basis functions. For the signal on the graph, if we want to perform a Fourier transform, it is natural for us to think that we must also find a set of orthogonal bases and express the signal on the graph through the linear combination of this set of orthogonal bases.

Square format:
$\hat{x} \tag{23}$

$x$ represents the signal on the graph
$U$ represents the eigenvector matrix of the graph
$\hat{x}$ Display signal $x$ 's Furiha series

How do you ask $\hat{x}$ ? We can left-multiply both sides of the above equation at the same time $U^T$ ，得到：
$U^T x = U^T U \hat{x} = I \hat{x} = \hat{x} \tag{24}$

$I$ Display square screen
$\hat{x}$ Display signal $x$ 's Furiha series
$U$ represents the eigenvector matrix of the graph
$x$ represents the signal on the graph
$U^T$ 电影 $Transpose matrix of U$

Therefore, we can get:
$\hat{x} = U^T x \tag{25}$

$\hat{x}$ Display signal $x$ 's Furiha series
$U$ represents the eigenvector matrix of the graph
$x$ represents the signal on the graph
$U^T$ 电影 $Transpose matrix of U$
$\hat{x}$ Display signal $x$ 's Furiha series

Properties of Eigenvector Basis

The eigenvalues of the Laplacian matrix occupy positions similar to frequencies.
- The larger the eigenvalue, the higher the frequency of the corresponding eigenvector.
- The smaller the eigenvalue, the lower the frequency of the corresponding eigenvector.
- The eigenvector with eigenvalue 0 corresponds to the number of connected components of the graph, that is, $The number of zero eigenvalues of L$ is equal to the number of connected components of the graph.
The eigenvectors of the Laplacian matrix serve as the positions of the basis functions.
- The 0 eigenvalue corresponds to a constant eigenvector, which is similar to the constant term in the Fourier transform.
- The eigenvectors corresponding to low eigenvalues are relatively smooth, and the eigenvectors corresponding to high eigenvalues transform more violently. The two correspond to low-frequency basis functions and high-frequency basis functions.

How to understand these two conclusions?
We use graph Laplacian quadratic form to define the smoothness of the signal. It represents the sum of the squared differences of the signals of two nodes connected by an edge multiplied by the weight. The smaller the value, the smoother it is.

$x^{\top} L x = \frac{1}{2}\sum_{i,j=1}^N A_{ij} (x_i - x_j)^2 \tag{28}$

又会
$x^{\top} L x = x^{\top} U \Lambda U^{\top} x = \hat{x}^{\top} \Lambda \hat{x} = \sum_{k =1}^N \lambda_k \hat{x}^2(\lambda_k) \tag{29}$
Therefore, the smaller the corresponding eigenvalue, the smoother the corresponding eigenvector.

Summarize

Using graph Fourier transform, the signal defined on the graph node can be transformed into $x\in\mathbb{R}^n$ Convert from spatial domain to spectral domain.

Transformation from spatial domain to spectral domain: $x\rightarrow \hat{x} = U^T x$
Conversion from spectral domain to spatial domain: $\hat{x} \rightarrow x = U \hat{x}$
Naka $\Lambda U^{-1}$

convolution theorem

Convolution theorem in space domain: $x\star h \rightarrow X(\omega) H(\omega) < /span>$

$\star g = \mathcal{F}^{-1} \odot [X(\omega) H(\omega)] = U(U^{\top}x \odot U^{\top}g) \tag{30}$
If you express this formula in matrix multiplication form, drop the harmand product. At the same time, we usually don’t care about what the filter signal g looks like in the spatial domain, but only care about its situation in the frequency domain.
Let
$g_{\theta} = (U^ {\top}g) = \text{diag} (\lambda g) =\text{diag} (\hat(\lambda))$

The formula is equivalent to the following formula:
$\star g =U \text{diag} (\hat{g}) U^{\top} x \tag{31}$

$\text{diag}(\hat{g}(\lambda_i))$ is a component of spectral domain convolution.

Three classic graph convolution models

Feature maps of different layers in a convolutional network:
Insert image description here

SCNN

main idea:

Use a learnable diagonal matrix to replace the convolution kernel in the spectral domain to implement graph convolution operations.

ChebNet

Chebyshev polynomials
Chebyshev polynomials are a class of orthogonal polynomials with important applications. They are defined as follows:
$T_0(x) = 1, T_1(x) = x, T_{ n+1}(x) = 2xT_n(x) - T_{n-1}(x) \tag{32}$

矩阵形式为：
$T_0(L) = I, T_1(L) = L, T_{n+1}(L) = 2LT_n(L) - T_{n-1}(L) \tag{32}$

Polynomial interpolation:
$\sum_{k=0}^{ K-1} \alpha_k x^{k-1} \tag{33}$
Chebyshev interpolation replaces the power function with the Chebyshev polynomial Term:
$\sum_{k=0}^{K-1 } \alpha_k T_k(x) \tag{34}$

The core idea of ChebNet:

Chebyshev polynomials are used to approximate the convolution kernel in the spectral domain to implement graph convolution operations.
Using the recursive properties of Chebyshev polynomials, the approximation degree of the convolution kernel in the spectral domain can be controlled within an arbitrary accuracy range.

$\begin{aligned} x \star g_{\theta} =& Ug_{\theta}U^{\top}x \\ =& U \sum_{k=0}^{K}\beta_k T_k(\Lambda) U^{\top} x \\ =& \sum_{k=0}^{K}\beta_k T_k(L)(U \Lambda U^{\top}) x \\ =& \sum_{k=0}^{K}\beta_k T_k(L) x \\ \end{aligned} \tag{35}$

The convolution kernel only has K+1 learnable parameters. Generally, k is much smaller than n, so the number of parameters is greatly reduced.
After using Chebyshev polynomials to replace the convolution in the spectral domain, eigenvalue decomposition is no longer needed.
The convolution kernel has strict spatial locality. At the same time, k is the receptive field radius of the convolution kernel. That is, the K-order nearest neighbor node of the central vertex is used as the domain node.
In this way, we do not need to calculate the Laplacian decomposition and directly use the Laplacian matrix to perform the convolution operation.

GCN

The core idea of GCN:

First-order Chebyshev polynomials are used to approximate the convolution kernel in the spectral domain to implement graph convolution operations.
By utilizing the recursive properties of first-order Chebyshev polynomials, the approximation degree of the convolution kernel in the spectral domain can be controlled within an arbitrary accuracy range.

$\begin{aligned} x \star g_{\theta} =& Ug_{\theta}U^{\top}x \\ =& \sum_{k=0}^{k}\beta_k T_k(\hat{L})x \\ =& \sum_{k=0}^1 \beta_k T_k(\hat{L})x \\ =& \beta_0 T_0(\hat{L})x + \beta_1 T_1(\hat{L})x \\ =& (\beta_0 + \beta_1 \hat{L})x =& (\beta_0 + \beta_1(L - I_n))x \\ =& (\beta_0 - \beta_1(D^{-1/2WD^{-1/2}}))x \\ =& (\theta(D^{-1/2WD^{-1/2}} + I_n))x \\\end{aligned} \tag{36}$