ChebNet paper reproduction (data preparation part)

1. Data Prepation

1.1 Load the dataset

# load data in folder datasets
mnist = input_data.read_data_sets('datasets', one_hot=False)

train_data = mnist.train.images.astype(np.float32)
val_data = mnist.validation.images.astype(np.float32)
test_data = mnist.test.images.astype(np.float32)
train_labels = mnist.train.labels
val_labels = mnist.validation.labels
test_labels = mnist.test.labels

train_data_shape:55000,784

val_data_shape:5000,784

test_data_shape:1000,784

train_label_shape:55000

Explanation: The training set has a total of 55,000 pictures, each of which is 28*28, and the value of each feature is 01. All array types are ndarray arrays

1.2 Structural diagram

1.2.1 Constructing a grid with size m

def grid(m, dtype=np.float32):
    """Return coordinates of grid points"""
    M = m**2  # 784=28*28
    x = np.linspace(0,1,m, dtype=dtype)  # 生成大小为784的0-1之间的等间距的小数数组x
    y = np.linspace(0,1,m, dtype=dtype)  # 生成大小为784的0-1之间的等间距的小数数组y
    xx, yy = np.meshgrid(x, y)  # shape(28,28)
    z = np.empty((M,2), dtype)  # shape(784,2)
    z[:,0] = xx.reshape(M)
    z[:,1] = yy.reshape(M)
    return z

np.linspace(start, stop, num, endpoint, retstep, dtype)

star and stop are the start and end positions, both scalars
num is the total number of interval points including start and stop, the default is 50
endpoint is a bool value, when it is False, the last point will be removed to calculate the interval
rest is a bool value, and it is True It will return the data list and the interval value at the same time.
dtype defaults to the type of the input variable. After the type is given, the generated array type will be converted to the target type.
This is a function that can generate equally spaced arrays, and it is quite commonly used. The most important are the first three parameters

X, Y = np.meshgrid(x, y) represents the combination of each data in x and each data in y to generate many points, and then put the x coordinates of these points into X, and put the y coordinates into Y , and the corresponding positions correspond to

The np.empty() function syntax is as follows:

empty(shape[, dtype, order])

According to the given shape and data type dtype, return a one-dimensional or multi-dimensional array. The elements of the array are not empty and are randomly generated data.

The parameters are explained as follows:

shape: An integer or tuple of integers defines the shape of the returned array.

dtype: data type, defines the type of the returned array, optional. as dtype = int

order: {'C', 'F'}, specifies the storage order of the returned array elements in memory.

1.2.2 Computing pairwise distances

def distance_sklearn_metrics(z, k=4, metric='euclidean'):
    """Compute pairwise distances"""
    d = sklearn.metrics.pairwise.pairwise_distances(z, metric=metric, n_jobs=1)  # 计算z中两两行之间的距离，shape(784,784)
    # k-NN
    idx = np.argsort(d)[:,1:k+1]  # 将每一行分别排序后获取排序后的下标，获取每一行中与其距离最近的8行，shape(784,8)
    d.sort()  # 将每一行分别排序
    d = d[:,1:k+1]  # 获取每一行与其距离最近的8行的距离值
    return d, idx  # 返回每一行中与其距离最近的8行的下标与距离值

sklearn.metrics.pairwise_distances(X, Y=None, metric=’euclidean’, n_jobs=None, **kwds)

Computes a distance matrix from vector array X and optional Y.
This method takes an array of vectors or a distance matrix and returns a distance matrix. If the input is an array of vectors, the distance is computed. If the input is a distance matrix, it is returned.
If Y is given (the default is None), the returned matrix is ​​the pairwise distances between arrays starting at X and Y.

The argsort(x) function arranges the elements in x from small to large , extracts its corresponding index (index), and then returns

1.2.3 Construct the adjacent sparse weight matrix of the graph

def adjacency(dist, idx):
    """Return adjacency matrix of a kNN graph"""
    M, k = dist.shape
    assert M, k == idx.shape
    assert dist.min() >= 0
    assert dist.max() <= 1

    # Pairwise distances
    sigma2 = np.mean(dist[:,-1])**2
    dist = np.exp(- dist**2 / sigma2)  # 将距离标准化

    # Weight matrix
    I = np.arange(0, M).repeat(k)  #shape(784*8=6272,)
    J = idx.reshape(M*k)  #shape(784*8=6272,)
    V = dist.reshape(M*k)  #shape(784*8=6272,)
    W = scipy.sparse.coo_matrix((V, (I, J)), shape=(M, M))  # 构造I*J的稀疏矩阵，对应非0的值为V

    # No self-connections
    W.setdiag(0)  # 对角线的值设置为0

    # Undirected graph
    bigger = W.T > W
    W = W - W.multiply(bigger) + W.T.multiply(bigger)

    assert W.nnz % 2 == 0  # 判断非零个数是否为偶数
    assert np.abs(W - W.T).mean() < 1e-10
    assert type(W) is scipy.sparse.csr.csr_matrix
    return W

1.2.4 Construction grid graph

def grid_graph(grid_side,number_edges,metric):
    """Generate graph of a grid"""
    z = grid(grid_side)
    dist, idx = distance_sklearn_metrics(z, k=number_edges, metric=metric)
    A = adjacency(dist, idx)   # shape(784, 784)的一个系数稀疏矩阵
    print("nb edges: ",A.nnz)  # 稀疏矩阵中有6396个零
    return A  # 返回邻接稀疏权重矩阵

1.3 Computing the foul language graph

1.3.1 Multiple Edge Matching HEM

def HEM(W, levels, rid=None):
    """
    Coarsen a graph multiple times using the Heavy Edge Matching (HEM).

    Input
    W: symmetric sparse weight (adjacency) matrix
    levels: the number of coarsened graphs

    Output
    graph[0]: original graph of size N_1
    graph[2]: coarser graph of size N_2 < N_1
    graph[levels]: coarsest graph of Size N_levels < ... < N_2 < N_1
    parents[i] is a vector of size N_i with entries ranging from 1 to N_{i+1}
        which indicate the parents in the coarser graph[i+1]
    nd_sz{i} is a vector of size N_i that contains the size of the supernode in the graph{i}

    Note
    if "graph" is a list of length k, then "parents" will be a list of length k-1
    """

    N, N = W.shape
    
    if rid is None:
        rid = np.random.permutation(range(N))
        
    ss = np.array(W.sum(axis=0)).squeeze()
    rid = np.argsort(ss)
        
        
    parents = []
    degree = W.sum(axis=0) - W.diagonal()
    graphs = []
    graphs.append(W)

    print('Heavy Edge Matching coarsening with Xavier version')

    for _ in range(levels):

        weights = degree            # graclus weights
        weights = np.array(weights).squeeze()

        # PAIR THE VERTICES AND CONSTRUCT THE ROOT VECTOR
        idx_row, idx_col, val = scipy.sparse.find(W)
        cc = idx_row
        rr = idx_col
        vv = val

        if not (list(cc)==list(np.sort(cc))):
            tmp=cc
            cc=rr
            rr=tmp

        cluster_id = HEM_one_level(cc,rr,vv,rid,weights)
        parents.append(cluster_id)

        # COMPUTE THE EDGES WEIGHTS FOR THE NEW GRAPH
        nrr = cluster_id[rr]
        ncc = cluster_id[cc]
        nvv = vv
        Nnew = cluster_id.max() + 1
        # CSR is more appropriate: row,val pairs appear multiple times
        W = scipy.sparse.csr_matrix((nvv,(nrr,ncc)), shape=(Nnew,Nnew))
        W.eliminate_zeros()
        
        # Add new graph to the list of all coarsened graphs
        graphs.append(W)
        N, N = W.shape

        # COMPUTE THE DEGREE (OMIT OR NOT SELF LOOPS)
        degree = W.sum(axis=0)

        # CHOOSE THE ORDER IN WHICH VERTICES WILL BE VISTED AT THE NEXT PASS
        ss = np.array(W.sum(axis=0)).squeeze()
        rid = np.argsort(ss)

    return graphs, parents

1.3.2 Constructing a binary tree

def compute_perm(parents):
    """
    Return a list of indices to reorder the adjacency and data matrices so
    that the union of two neighbors from layer to layer forms a binary tree.
    """

    # Order of last layer is random (chosen by the clustering algorithm).
    indices = []
    if len(parents) > 0:
        M_last = max(parents[-1]) + 1
        indices.append(list(range(M_last)))

    for parent in parents[::-1]:

        # Fake nodes go after real ones.
        pool_singeltons = len(parent)

        indices_layer = []
        for i in indices[-1]:
            indices_node = list(np.where(parent == i)[0])
            assert 0 <= len(indices_node) <= 2

            # Add a node to go with a singelton.
            if len(indices_node) is 1:
                indices_node.append(pool_singeltons)
                pool_singeltons += 1

            # Add two nodes as children of a singelton in the parent.
            elif len(indices_node) is 0:
                indices_node.append(pool_singeltons+0)
                indices_node.append(pool_singeltons+1)
                pool_singeltons += 2

            indices_layer.extend(indices_node)
        indices.append(indices_layer)

    # Sanity checks.
    for i,indices_layer in enumerate(indices):
        M = M_last*2**i
        # Reduction by 2 at each layer (binary tree).
        assert len(indices[0] == M)
        # The new ordering does not omit an indice.
        assert sorted(indices_layer) == list(range(M))

    return indices[::-1]

assert (compute_perm([np.array([4,1,1,2,2,3,0,0,3]),np.array([2,1,0,1,0])])
        == [[3,4,0,9,1,2,5,8,6,7,10,11],[2,4,1,3,0,5],[0,1,2]])

1.3.3 Construction of clustering tree

def perm_adjacency(A, indices):
    """
    Permute adjacency matrix, i.e. exchange node ids,
    so that binary unions form the clustering tree.
    """
    if indices is None:
        return A

    M, M = A.shape
    Mnew = len(indices)
    A = A.tocoo()

    # Add Mnew - M isolated vertices.
    rows = scipy.sparse.coo_matrix((Mnew-M,    M), dtype=np.float32)
    cols = scipy.sparse.coo_matrix((Mnew, Mnew-M), dtype=np.float32)
    A = scipy.sparse.vstack([A, rows])
    A = scipy.sparse.hstack([A, cols])

    # Permute the rows and the columns.
    perm = np.argsort(indices)
    A.row = np.array(perm)[A.row]
    A.col = np.array(perm)[A.col]

    assert np.abs(A - A.T).mean() < 1e-8 # 1e-9
    assert type(A) is scipy.sparse.coo.coo_matrix

1.3.4 Constructing the Graph Laplacian Matrix

def laplacian(W, normalized=True):
    """Return graph Laplacian"""

    # Degree matrix.
    d = W.sum(axis=0)

    # Laplacian matrix.
    if not normalized:
        D = scipy.sparse.diags(d.A.squeeze(), 0)
        L = D - W
    else:
        d += np.spacing(np.array(0, W.dtype))
        d = 1 / np.sqrt(d)
        D = scipy.sparse.diags(d.A.squeeze(), 0)  # 构造对角稀疏矩阵
        I = scipy.sparse.identity(d.size, dtype=W.dtype)  # 构造单位矩阵
        L = I - D * W * D  # 构造标准化拉普拉斯矩阵

    assert np.abs(L - L.T).mean() < 1e-9
    assert type(L) is scipy.sparse.csr.csr_matrix
    return L

1.3.5 Using multiple edge matching to construct K coarsened graphs

def coarsen(A, levels):
    
    graphs, parents = HEM(A, levels)
    perms = compute_perm(parents)

    laplacians = []
    for i,A in enumerate(graphs):
        M, M = A.shape
            
        if i < levels:
            A = perm_adjacency(A, perms[i])

        A = A.tocsr()  # 将矩阵转换为压缩形式
        A.eliminate_zeros()  # 删除零
        Mnew, Mnew = A.shape
        print('Layer {0}: M_{0} = |V| = {1} nodes ({2} added), |E| = {3} edges'.format(i, Mnew, Mnew-M, A.nnz//2))

        L = laplacian(A, normalized=True)
        laplacians.append(L)  # 将当前图的拉普拉斯矩阵加入列表
        
    return laplacians, perms[0] if len(perms) > 0 else None

Perms store the binary tree constructed after edge aggregation, and laplacians store the graph Laplace matrix corresponding to each coarsening graph

The first layer represents a total of 976 nodes, 192 of which are neutral false nodes added (for constructing a binary tree), and the real number of nodes in the original image is 976-192=784

1.4 Calculate the maximum eigenvalue of each rough language graph

def lmax_L(L):
    """Compute largest Laplacian eigenvalue"""
    return scipy.sparse.linalg.eigsh(L, k=1, which='LM', return_eigenvectors=False)[0]

scipy.sparse.linalg.eigsh(A, k=6, M=None, sigma=None, which='LM', v0=None, ncv=None, maxiter=None, tol=0, return_eigenvectors=True, Minv= None, OPinv=None, mode='normal')
function: find k eigenvalues, eigenvectors of real symmetric square matrix or complex Hermitian matrix

1.5 Reindex the node index of the data set according to the binary tree node index, and construct the binary tree of the data set

# Reindex nodes to satisfy a binary tree structure
train_data = perm_data(train_data, perm)
val_data = perm_data(val_data, perm)
test_data = perm_data(test_data, perm)

print(train_data.shape)
print(val_data.shape)
print(test_data.shape)

print('Execution time: {:.2f}s'.format(time.time() - t_start))
del perm