深度学习的低秩优化:在紧凑架构和快速训练之间取得平衡（上）

论文出处：[2303.13635] Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training (arxiv.org)

由于篇幅有限，本篇博客仅引出问题的背景、各种张量分解方法及其分解FC/Conv层的方法，欢迎各位讨论！

高度的计算复杂度和存储成本使得深度学习难以在资源受限的设备上使用，并且不环保，功耗高。本文专注于高效深度学习技术的低秩张量优化方法。在空间域，深度神经网络通过对网络参数进行低秩逼近( low rank approximation )进行压缩，以更少的网络参数直接降低了存储需求。在时域中，网络参数可以在几个子空间中进行训练，从而实现快速收敛的高效训练。这两种类型可以通过低秩优化联系起来，发现时域和空间域的冗余是同源的，为了加快神经网络的训练速度，需要在空间效率和时间效率之间取得平衡。
An Overview of Low-Rank Tensor Optimization for Efficient Deep Learning

文章目录

1. Problem background

There are two main challenges in deep learning: high complexity and slow convergence .
- The high complexity means that there are millions of parameters in the deep neural network, and the calculation between a large number of parameters and inputs is very troublesome , which emphasizes the need for compression and acceleration of efficient algorithms, especially for devices with limited resources (such as mobile phones, IoT devices). Moreover, since deeper or wider structures can lead to better performance, deep neural networks are gradually characterized by their overparameterization , which means too much redundancy in DNNs, leading to overfitting.
- The traditional BP algorithm has slow convergence speed and long training time. At the same time, the convergence speed is also sensitive to the setting of the learning rate and the way of initializing the weights.
Reducing the high complexity of deep neural networks falls into two categories: reducing the number of parameters , and reducing the average bit width of data representation .
- Methods to reduce the amount of parameters: low rank approximation (the focus of this article), pruning, weight-sharing, sparsity, knowledge distillation
- Reduce the average bit width of data representation: Quantization and entropy coding
The low-rank optimization of model compression is divided into three categories: pre-training methods, preset methods and compressed sensing methods (the difference lies in the way of training)
- The pre-training method directly decomposes the pre-trained model to obtain hot initialization of the compressed format, and then retrains the compressed model to restore performance.
- Preset methods train a network from scratch with no pre-training in a compact format.
- Different from the above two methods, the compressed sensing method explicitly considers the compression during training by gradually making the network have a low-rank structure.
While a discussion of low-rank optimization can also be found in [2302.09019] Tensor Networks Meet Neural Networks: A Survey and Future Perspectives (arxiv.org) , this paper further examines how it can be combined with other compression techniques in pursuit of lower , and recommend effective rank as the most effective metric to use in low-rank optimization.
Accelerating Convergence of Deep Neural Networks: Subspace Training
- First-order optimization methods have some inherent flaws, such as slow theoretical and empirical convergence.
- Second-order methods can handle this kind of problems well, but pre-DNNs are not suitable due to the large amount of computation. Projecting the parameters onto a tiny subspace represented by many independent variables is an efficient way to solve this problem , with only a few variables to optimize.

2. Low-rank optimization of model compression

Compression methods such as quantization, pruning, and low-rank approximation can reduce the complexity of DNNs without significantly reducing accuracy. Among them, low-rank approximation is widely adopted due to the solid theoretical basis of tensor decomposition .

(1) Various tensor decomposition methods for network compression

Low-rank approximation can provide ultra-high compression ratios for recurrent neural networks with little loss of accuracy. However, when it comes to Convolutional Neural Networks (CNNs), the compression performance is not as satisfying as RNNs. The operation of reconstructing the 4D convolution kernel into a matrix and using singular value decomposition (SVD) to decompose the matrix into two factors leads to distortion of structural information . Therefore, more efficient tensor factorization has attracted interest. Applying CP decomposition to decompose the convolutional layer into four consecutive convolutional layers significantly speeds up the CNN. Tucker decomposition utilizes channel redundancy to decompose the 4D kernel into a 4D compact kernel and two matrices. On the basis of these three classic decompositions, many other flexible methods have emerged, including HT, TT, TR, BTD, GKPD, tensor decomposition based on semi-tensor product (STP), etc., which have significantly improved DNN compression performance. Table 1 presents the performance of common tensor decomposition methods for compressing ResNet32 using the Cifar10 dataset.

The time-space complexity of different methods, for the FC layer, is shown in Table 3. Table 4 is the Conv layer.

Time and space complexity of different methods for FC layer

The basic representation and operation of tensors will not be expanded here, and some learning materials are recommended:

Talking about Tensor Decomposition (2): The Mathematical Basis of Tensor Decomposition (zhihu.com)

Tensor Basics | Tensor Rank and Rank Tensor - Zhihu (zhihu.com)

( Xinyu Chen - Zhihu (zhihu.com)

Tensor Computation for Data Analysis | SpringerLink

The following is an introduction to the nine decomposition methods mentioned in this article:
insert image description here

1) Singular Value Decomposition

For a given matrix $X∈R^{N×N}$ , its SVD can be written as:

$X = U diag(s)V^T$

Let R be the rank of the matrix, $R ≤ min{(M, N)}$ . Note that $U∈R^{M×N}$ and $V ∈ R^{N×R}$ minutes complete $UU^T = I$ 和 $VV^T = I$ 。而 $s∈\Bbb{R}^R$ decrease in order, that is, $s_1≥s_2≥...≥s_R$ 。

Since the weight format of the FC layer is a natural matrix, the singular value decomposition can be directly used. Using singular value decomposition (SVD), the FC layer can be approximated as two consecutive layers. The weights of the first layer and the second layer can be expressed as $diag(\sqrt{s} )V^T$ 和 $Udiag(\sqrt{s})$ . For the Conv layer, the 4D kernel is first reconstructed into a 2D matrix. By exploiting different types of redundancies, there are two decomposition schemes, one is to reshape W into $S - b y - C H W$ matrix, that is,channel decomposition, the other isspace decomposition, get $S H - b y - CW matrix$ . Then, calculate the SVD of the reconstruction matrix. Similar to the process of compressing the FC layer, one can use the factor $B$ and $A$ reshaped tensor representation of the two Conv layers to replace the original layers.

然而，这两种方法都只能利用一种冗余。此外，输入通道之间也存在冗余。可以通过张量分解同时利用各种冗余，帮助我们获得更高的压缩比。

2) CP Decomposition

SVD将一个矩阵分解为秩一矩阵的和，而CP分解将一个张量分解为秩一张量（rank-one tensors）的和。对于n阶张量 $\mathcal{X}∈R^{I_1×I_2×···×I_N}$ , CP分解可表示为:

$\mathcal{X}=\llbracket \lambda ; \mathbf{A}^{(1)}, \mathbf{A}^{(2)}, \cdots, \mathbf{A}^{N} \rrbracket=\sum_{r=1}^{R} \lambda_{r} \mathbf{a}_{r}^{(1)} \circ \mathbf{a}_{r}^{(2)} \circ \cdots \circ \mathbf{a}_{r}^{(N)} .$

Each $\mathbf{a}_{r}^{(n)}$ means $A^{(n)}$ th $r$ 列， $λ∈\Bbb{R}^R$ stands forThe significance of the $r components.$ Tensor $\mathcal{X}$ The rank of $X$ $R$ representation, defined as the minimum number of rank 1 tensors.

When using CP to compress the FC layer, it is first necessary to expand the weight matrix W into a 2d-order tensor W $\mathcal{ W}∈R^{O_1×O_2··×O_d×I_1×I_2×···×I_d}$ . At the same time, the input vector $x ∈ R^I$ should be expressed as a $d$ -order tensor $x∈R^{I_1×I_2×···×I_d}$ . For convolutional kernels, by directly performing CP on the 4-D kernel tensor, the layer will be approximated by four consecutive convolutional layers, whose weights are represented by four factor matrices respectively.

3) Tucker Decomposition

Tucker decomposition can be seen as a higher-order generalization of principal component analysis (PCA). It represents a rank-n tensor and a rank-n core tensor multiplied by a basis matrix along each modality. Therefore, for $\mathcal{X}∈R^{I_1×I_2×···×I_N}$ ,have:

$\mathcal{X}=\mathcal{G} \times_{1} \mathbf{A}^{(1)} \times_{2} \mathbf{A}^{(2)} \times_{3} \cdots \times_{N} \mathbf{A}^{(N)}$

其中 $\mathcal{G} \in \mathbb{R}^{R_{1} \times R_{2} \times \cdots \times R_{N}}$ Called the core tensor. Among them, " $\times{ }_{n}$ "means $n$ modulus product, i.e. tensor with modalityMatrix multiplication of $n .$ On elements, " $\times{ }_{n} $" can be expressed as:

$\left(\mathcal{G} \times{ }_{1} \mathbf{A}^{(1)}\right)_{i_{1}, r_{2}, \cdots, r_{N}}=\sum^{R_{1}}_{r_1 = 1} \mathcal{G}_{r_{1}, r_{2}, \cdots, r_{N}} \mathbf{A}_{i_{1}, r_{1}}^{(1)}$

Factor matrix $A(n)∈R^{I_n×R_n}$ The column of can be seen as the principal components of the nth modality. The core tensor $\mathcal{G}$ can be seen as $\mathcal{X}$ , or coefficients in a low-dimensional subspace. In this case we can say that $\mathcal{X}$ is a $rank-(R_1, R_2,···,R_N)$ tensor

In the case of compressed FC layers, similar to CP, the same compactification is performed on weights and inputs by directly performing Tucker decomposition on second-order tensors. For the Conv layer, since the spatial size of the kernel is too small, we can only use Tucker2 to exploit the redundancy between filters and between input channels to generate 1×1 convolution, H×W convolution and 1×1 convolution.

4) Block Term Decomposition

Block Term Decomposition (Block Term Decomposition, BTD) is introduced in as a more powerful tensor decomposition, which combines CP decomposition and Tucker decomposition. Therefore, BTD is more robust than the original CP and Tucker decomposition . CP approximates a sum of tensors of rank tensors, while BTD is a sum of tensors of low rank Tucker format. Alternatively, BTD can be viewed as an instance of Tucker by concatenating the factor matrices in each modality and arranging all the core tensors of each subtensor into a block-diagonal core tensor . Therefore, consider an n-order tensor $\mathcal{X} ∈ R^{I_1×I_2×···×I_d}$ , its BTD can be expressed as:

$\mathcal{X}=\sum^N_{n=1}\mathcal{ G}_n \times_{1} \mathbf{A}^{(1)}_n \times_{2} \mathbf{A}^{(2)}_n \times_{3} \cdots \times_{d} \ mathbf{A}^{(d)}_n$

In the above formula, N represents the CP rank, that is, the number of block items, $\mathcal{G} \in \mathbb{R}^{R_{1} \times R_{2 } \times \cdots \times R_{d}}$ is the core tensor of the Nth multilinear rank block term with rank $R_1, R_2,···,R_d)$ 。

When BTD is applied to a compressed FC layer, the resulting compact layer is called a Block Term layer (BTL). In BTL, the input tensor $\mathcal{X} ∈ R^{I_1×I_2×···×I_d}$ From the original input vector $X ∈ R^I$ tensorization, the original weight matrix W is reshaped as $W ∈ R^{O_1×I_1×O_2×I_2 ×O_d× I_d}$

Then, we can use the factor tensor { $A^{(d)}_n∈R^{Od×Id×Rd} via BTD$ $A_{n}^{(d)} \in R^{O d \times I d \times R d}$ } ${}^d_{n=1}$ Break down W. By $BTD(\mathcal{W} )$ and $\mathcal{X}$ The tensor contraction operator is performed between $X$ $\mathcal{Y} ∈ R^{O_1×O_2×···×O_d}$ , which can be vectorized as the final output vector.

For the Conv layer, the literature claims that by reconstructing the 4D kernel into a matrix $W ∈ R^{S×CHW}$ layer can be converted into a BTL. Specifically, the matrix should be further reconstructed as $1 × H × 1 × W × S_1 × C_1 × S_2 × C_2 ×···× S_d × C_d$ 。

5) Hierarchical Tucker Decomposition

Hierarchical Tucker Decomposition (HT) is a hierarchical variant of Tucker Decomposition that iteratively represents a high-rank tensor and two low-rank sub-tensors and the resulting transfer matrix using Tucker decomposition. For tensor $\mathcal{X} ∈ R^{I_1×I_2×···×I_N}$ , we can simply divide the index set {1,2,···,N} into two subsets, namely $T = \{t_1, t_2，··，t_k\}， S = \{s_1, s_2，··，s_{N−k}\}$

设 $U_{12···N}∈R^{I_{t_1} I_{t_2}···I_{t_k}I_{s_1} I_{s_2}···I_{s_{N-k}} ×1}$ represents the matrix reconstructed from X, the truncated matrix $U_t∈R^{I_{t_1} I_{t_2}···I_{t_k} ×R_t}$ , $U_s∈R^{I_{s_1} I_{s_2}···I_{s_{N-k}} ×R_s}$ Represents the column basis matrix corresponding to the two subspaces. Then, we can:

$U_{12···N} = (U_t ⊗ U_s)B_{12···N} ,$

Among them, $B_{12··N}∈R^{R_tR_s×1}$ is called the transition matrix, and "⊗" means the Kronecker product between the two matrices. Then, the set $\mathbb{T}$ is divided into two subsets $\mathbb{L}=\{l_1, l_2,···,l_q\}$ 和 $\mathbb {V}=\{v_1，v_2，···，v_{k−q}\}$ 。我们可以将 $U^l∈\mathbb{R}^{I_{l_1} I_{l_2}··I_{l_q}×R_l}$ ， $U^v∈\mathbb{R}^{I_{v_1} I_{v_2}··I_{v_q}×R_v}$ , $U^t ∈\mathbb{R} ^{I_{t_1} I_{t_2}··I_{t_q}×R_t}$ 的 $U_t$ Expressed as:

$U_{t} = (U_l ⊗ U_v)B_{t}$

A similar factoring procedure applies to both $U_s$ . By repeating this process until the index set cannot be partitioned, we can finally obtain the tree-like HT format of the target tensor.

To achieve FC layer compression through HT, the weight matrix needs to be transformed into $\mathcal W∈\mathbb{ R}^{\left(I_1·O_1 \right )×\left(I_2·O_2\right)×··×\left(I_d·O_d\right)}$ , and tensorize the input data as $\mathcal{X} ∈ R^{I_1×I_2×···×I_d}$ . In order to reduce the calculation complexity, the chain calculation shown in the figure is adopted. However, since there is no law relating convolution and shrinkage, the kernel of the Conv layer must be recovered from the HT format. By the way, to maintain balance, the 4D kernel should be tensorized as $\ mathcal W∈R^{\left(H·W \right )×\left(C_1·S_1 \right )×\left(C_2·S_2 \right )×··×\left(C_d·S_d \right )}$ 。

6) Tensor Train Decomposition

A sequence of tensors (TT) is a special case of HT, which is a degenerate HT format. TT decomposes high-rank tensors into collections of third-rank or second-rank tensors. These core tensors are concatenated via the contraction operator. Suppose we have a tensor of order N, $\mathcal X∈R^{I_1×I_2×··×I_N}$ , on elements, we can decompose it into TT format:

$\mathcal X_{i_1,i_2,··· ,i_N} =\sum _{r_1,r_2,··· ,r_N} \mathcal G^1_{i_1,r_1}\mathcal G^2_{r_1,i_2,r_2} · · · \mathcal G^N_{r_{N-1},i_{N}}$

Among them, $\{\mathcal G_n∈\mathbb{R}^{R_{n−1}×I_n×R_n}\}^N_{n =1}$ with $R_0 = 1$ and $R_n = 1$ 's is called a TT core. Rank $\{Rn \}^N_{n = 0}$ The set of is called TT rank.

TT is used to compress the FC layer, where the weight matrix is reshaped into a high-order tensor, $\mathcal W∈\mathbb{R}^{\left(I_1·O_1 \right )×\left(I_2·O_2\right)×··×\left(I_d·O_d\right)}$ . In TT format $\mathcal W$ , the obtained TT kernel $\{\mathcal G_n∈\mathbb{R}^{R_{n−1}×I_n ×O_n×R_n}\}^N_{n=1}$ Can be directly contracted with tensorized input. It was proposed in [141] that TT compresses Conv layers more efficiently than HT, while HT is more suitable for compressing FC layers whose weight matrices prefer to be reshaped into balanced tensors.

Using TT on the Conv layer was introduced in [37], where the 4D kernel tensor should be reshaped as $H W) × (C_1 S_1) × (C_2 S_2) × × (C_d S_d)$ , and the input feature map is reshaped to $X∈\mathbb R^{H×W×C_1×···×C_d}$ 。在前馈阶段，张量化的输入X将与每个TT核心逐一收缩。尽管TT可以显著节省存储成本，但计算复杂度可能高于原始的Conv层。因此，[149]中提出了HODEC（高阶分解卷积），以同时降低计算和存储成本，从而将每个TT核心进一步分解为两个三阶张量。

7) Tensor Ring Decomposition

由于边缘TT核的不统一（即 $R_0=1$ 和 $R_n=1$ ），如何排列张量的维数以找到最优的TT格式仍然是一个悬而未决的问题。为了解决这个问题，张量环（TR）分解提出了在核上进行循环多线性乘积的分解。考虑一个给定的张量， $\mathcal X∈R^{I_1×I_2×··×I_N}$ ，在元素上，我们可以将其TR表示公式化为：

$\mathcal X_{i_1,i_2,··· ,i_N} =\sum _{r_1,r_2,··· ,r_N} \mathcal G^1_{r_1,i_1,r_2}\mathcal G^2_{r_2,i_2,r_3} · · · \mathcal G^N_{r_{N},i_{N},r_1} = tr( \sum _{r_2,··· ,r_N} \mathcal G^1_{:,i_1,r_2}\mathcal G^2_{r_2,i_2,r_3} · · · \mathcal G^N_{r_{N},i_{N},:} )$

where all kernels $\{\mathcal G_n∈\mathbb{R}^{R_{n}×I_n×R_{n+1}}\ }^N_{n=1}$ And $R_{N+1}=R_1$ Called the TR nucleus. This form is equivalent to $R_1$ Sum of the form TT. Thanks to the circular multilinear product obtained by using trace operations, TR treats all cores equally and becomes more powerful and general than TT.

Furthermore, TR corrects for the variability of gradients in TT due to the recurrent strategy. Therefore, TR is also suitable for compressing FC layers. In [137], TR was first used to compress DNNs. Specifically, the weight matrix of the FC layer should be reshaped to be of size $I_1×··×I_d×O_1×·×O_2$ A rank 2 tensor of , and then represent the tensor in TR format. For the feed-forward process, first merge the first d cores and the last d cores to obtain $\mathcal F_1∈ \mathbb R^{R_1× I_1×··×I_d×R_{d+1}}$ 和 $\mathcal F_2∈ \mathbb R^{R_{d+1}×O_1×··×O_d×R_{1}}$ . Then, we can calculate the input $\mathcal X∈R^{I_1×I_2×··×I_d}$ and $\mathcal F_1$ The contraction between, get a can use $\mathcal F_2$ Shrunken matrix. The size of the final output tensor is $O_1×O_2× ··×O_d$ . For the Conv layer, if the kernel tensor is kept at rank 4 and the spatial information is kept, its TR format can be formulated as:

$K_{s,c,h,w} = \sum^{R_1}_{r_1=1}\sum^{R_2}_{r_2=1}\sum^{R_3}_{r_3=1}\mathcal U_{r_1,s,r_2}\mathcal V_{r_2,c,r_3}\mathcal Q_{r_3,h,w,r_1}$

Therefore, the original layer can be represented by three consecutive layers with weight tensors $\mathcal V, \mathcal Q, and \mathcal U$ _ If a higher compression ratio is required, we can further convert $\mathcal U$ V $\mathcal V$ are regarded as fromTensors obtained by merging $d core tensors, thus increasing the computational burden of merging.$

8) Generalized Kronecker P.roduct Decomposition

Kronecker product decomposition (KPD) can decompose a matrix into two smaller factor matrices interconnected by a Kronecker product, which is very efficient when applied to compressed RNNs. To further compress the Conv layer, it is generated as a generalized Kronecker integral decomposition (GKPD), which represents a tensor by the sum of multidimensional Kronecker products between two factor tensors. Formally, $A∈R^{J_1×J_2×··×J_N}$ and $B∈R^{K_1×K_2×···×K_N}$ The multidimensional Kronecker product between is formulated as:

$(A⊗B)_{i_1,i_2,··· ,i_N} = \mathcal A_{j_1,j_2,··· ,j_N} B_{k_1,k_2,··· ,k_N}$

Among them $j_n = bi_n/K_nc$ and $k_n = i_nmodK_n$ . Based on this, for a given N-order tensor $\mathcal X∈R^{J_1K_1×J_2K_2×···×J_NK_N}$ , GKPD can be expressed as:

$\mathcal X= \sum^R_{r=1} \mathcal A_r ⊗\mathcal B_r$

where R is called the Kronecker rank. For a given $To find the best approximation in R$ 's GKPD, we can transform this optimization problem into finding the best rank of the matrix - $R$ approximation, $S V D$ can be achieved by carefully placing $\mathcal X$ is rearranged into a matrix, and $\mathcal A$ sum $\mathcal B$ is rearranged into vectors for easy solution.

In order to decompress the Conv layer using GKPD, the 4D kernel is expressed as:

$W_{s,c,h,w} = \sum^R_{r=1}(\mathcal A_r)_{s_1,c_1,h_1,w_1} ⊗ (\mathcal B_r)_{s_2,c_2,h_2,w_2}$

Where $S_1S_2=S, C_1C_2=C, H_1H_2=H, W_1W_2=W$ _ Each $A_r⊗B_r$ A 2D convolution between and input can be converted to a depth equal to $C_2$ A 3D convolution, followed by multiple 2D convolutions. In addition, $The R$ Kronecker product group can be seen as calculating $R$ parallel channels. Analysis shows that the larger $S_1$ and $C_2$ Helps to get more FLOP reduction.

9) Semi-tensor Product-based Tensor Decomposition

Semi-tensor product (STP) is a generation of traditional matrix product, which extends the calculation of two-dimensional matching matrix to the calculation of two-dimensional arbitrary matrix. Since tensor contraction is based on traditional matrix product, we can further replace STP with tensor contraction, which will lead to a more general and flexible tensor decomposition method. Proposed in [152], half-tensor product-based tensor decomposition aims to enhance the flexibility of Tucker decomposition, TT decomposition, and TR decomposition by replacing the traditional matrix product in tensor shrinkage with STP, which shows that the much higher efficiency. Consider a special case where the number of columns $\mathbb R^{M×NP}$ and $QW∈\mathbb R^{P×Q}$ Proportional to the number of rows in $^{P}$ $^{\times}$ $^{Q , STP can be expressed as:}$
STP

Alternatively, elements are represented as:

$Y_{m,g(n,q) }=\sum^P_{p=1}X_{m,g(n,p)}W_{p,q}$

Equation, $NQY∈\mathbb R^{M×NQ}$ , $g (N, Q) = (Q - 1) N + n and g (N, p) = (p - 1) N + n$ is a heavy index function.

Therefore, taking the STP-based Tucker decomposition as an example, namely STTu, can be formulated as:

insert image description here

其中 $\mathcal G∈\mathbb R^{R_1×R_2×··×R_N}，A^{(n)}∈\mathbb R^{I_n/t × R_n/t}$ . Compared with ordinary Tucker, the number of parameters is changed from $(\prod^ N_{ n=0}R_n+ \sum^N_{n=1}I_nR_n)$ 减少到 $(\prod^ N_{ n=0}R_n+ \sum^N_{n=1}{I_nR_n}/t^2)$ 。

(2) Optimization method for low-rank approximation

We have introduced various tensor decomposition methods, but how to apply these methods to DNN without significant loss of accuracy is an optimization problem to be discussed. Since the lower the tensor rank, we will achieve higher compression ratio , we hope that each layer of DNN can be decomposed by very low rank tensor decomposition. However, as the rank decreases, the approximation error increases, leading to a drastic loss of accuracy . Therefore, there is a tradeoff between accuracy and compression ratio, which is a widely studied problem. There are mainly three low-rank optimization methods to achieve a good trade-off: pre-training methods, preset methods, and compressed sensing methods.