CUR Decomposition

Prerequisite: Matrix Multiplication Sampling

常规矩阵乘法的计算：

设$A\in M_{m\times n},B\in M_{n\times p}$
$$AB=\sum _{k=1}^{n}A(:,k)B(k,:)$$
时间复杂度为$O(n^3)$，太慢了，我们希望通过随机取样的方法计算矩阵乘法$AB$，降低时间复杂度

一次取样：

随机取$A$中的一行与$B$中对应的一列
设取第$k$行与第$k$列的概率为$p_k$，我们有 $p_1+\cdots +p_n=1$
令$X=\frac{1}{p_k}A(:,k)B(k,:)$
则有
$$E(X)=\sum _{k=1}^n{p}_k\cdot \frac{1}{p_k}A(:,k)B(k,:)=AB$$
$$Var(X)\stackrel{def}{=}\sum _{ij}Var(X_{ij})=\sum _{ij}E(X_{ij}^2)-\sum _{ij}{E}^2(X_{ij})$$

注：
在矩阵方差的这个定义下，我们有$Var(X)=E(\Vert X-E(X){\Vert }_F^2)$

前半部分
$$\begin{array}{rl}\sum _{ij}E(X_{ij}^2)=& \sum _{ij}\sum _k{p}_k\cdot \frac{1}{p_k^2}{A}_{ik}^2{B}_{kj}^2\\ =& \sum _k\frac{1}{p_k}\left(\sum _i{A}_{ik}^2\right)\left(\sum _j{B}_{kj}^2\right)\\ =& \sum _k\frac{1}{p_k}\Vert A(:,k){\Vert }^2\Vert B(k,:){\Vert }^2\\ \stackrel{p_k=\frac{\Vert A(:,k){\Vert }^2}{\Vert A{\Vert }_F^2}}{=}& \Vert A{\Vert }_F^2\sum _k\Vert B(k,:){\Vert }^2\\ =& \Vert A{\Vert }_F^2\Vert B{\Vert }_F^2\end{array}$$

注：
对于倒数第二个等号，由柯西不等式，本来当 $p_k\propto \Vert A(:,k)\Vert \Vert B(k,:)\Vert$时取最小值。
但是为了计算方便，这里让 $p_k\propto \Vert A(:,k){\Vert }^2$，也就是 $p_k=\frac{\Vert A(:,k){\Vert }^2}{\Vert A{\Vert }_F^2}$

后半部分
$$\sum _{ij}{E}^2(X_{ij})=\Vert E(X){\Vert }_F^2=\Vert AB{\Vert }_F^2$$
因此
$$Var(X)=\Vert A{\Vert }_F^2\Vert B{\Vert }_F^2-\Vert AB{\Vert }_F^2\le \Vert A{\Vert }_F^2\Vert B{\Vert }_F^2$$

多次取样：

核心思想：通过多次取样降低方差
随机取$A$中的$s$列组成 $C\in M_{m\times s}$和$B$中相应的$s$行组成 $R\in M_{s\times p}$
当然还要乘上相应的系数，具体来说：
$$C=\left[\begin{array}{c}\frac{A(:,k_1)}{\sqrt{s{p}_{k_1}}},\cdots ,\frac{A(:,k_s)}{\sqrt{s{p}_{k_s}}}\end{array}\right]$$
$$R=\left[\begin{array}{c}\frac{B(k_1,:)}{\sqrt{s{p}_{k_1}}}\\ ⋮\\ \frac{B(k_s,:)}{\sqrt{s{p}_{k_s}}}\end{array}\right]$$
其中$k_1,\cdots ,k_s$是从 { 1 , 2 , ⋯   , n } \{1,2,\cdots,n\} { 1,2,⋯,n}中随机取$s$个的结果
这样
$$\begin{array}{rl}CR& =\frac{1}{s}\sum _i\frac{1}{p_{k_i}}A(:,k_i)B(k_i,:)\\ & =\frac{1}{s}\sum _i{X}_i\end{array}$$
因此$CR$就相当于重复取样多次后的$X$的平均值，相应地
$$Var(CR)=Var(\frac{1}{s}\sum _i{X}_i)=\frac{1}{s^2}\sum _{i}Var(X_i)\le \frac{1}{s}\Vert A{\Vert }_F^2\Vert B{\Vert }_F^2$$
最终结论：
$$Var(CR)=E\left(\Vert AB-CR{\Vert }_F^2\right)\le \frac{1}{s}\Vert A{\Vert }_F^2\Vert B{\Vert }_F^2$$

CUR Decomposition

Motivation: Sketch of Matrix

Let $A\in M_{m\times n}$
The time complexity of computing $Ax$ is $O(mn)$
But if we decomposite $A$ into $CUR$, where $C\in M_{n\times s},U\in M_{s\times r},R\in M_{r\times n}$, then the time complexity would be $O(ms+sr+rn)$, which would be $O(m+n)$ if $s$ and $r$ are $O(1)$.

Comparison with SVD

Pros

• faster
• preserve the actual data $\Rightarrow$ easier to interpret than the linear combination of data as in SVD
• preserve some property, like the sparsity

Cons

• less accurate approximation
• stronger error bound

Intuitions

First Try: Identity Matrix

Just let $B=I$ and repeat the matrix multiplication sampling process.
However, $I$ is full rank and each row has the same weight, so sampling will lose too much information, and require the bound to be too strong:
$$E\left(\Vert AI-CR{\Vert }_F^2\right)\le \frac{1}{s}\Vert A{\Vert }_F^2\Vert I{\Vert }_F^2=\frac{n}{s}\Vert A{\Vert }_F^2$$

Projection Matrix

We want a low rank $B$, so that sampling won’t cause information loss.
Consider the projection matrix $P=R^T(R{R}^T{)}^{-1}R$, where $R\in M_{s\times n}$ is a matrix of $r$ rows of $A$ picked according to length squared sampling. $P$ is just like a pseudo indentity.

Note: Properties of projection matrix
$Px=x$ if $x$ is in the row space of $R$
$Px=0$ otherwise

Main Theorem

Let $A\in M_{m\times n},r,s\in {\mathbb{N}}^{+}$
Let $C\in M_{m\times r}$ be a matrix of $s$ columns of $A$ picked according to length squared sampling, $R\in M_{s\times n}$ be a matrix of $r$ rows of $A$ picked according to length squared sampling again.
Then we can find a $U\in M_{r\times s}$, so that
$$E\left(\Vert A-CUR{\Vert }_2^2\right)\le \Vert A{\Vert }_F^2\left(\frac{1}{\sqrt{r}}+\frac{r}{s}\right)$$

Note:
Unlike what we do in matrix multiplication sampling, the rows are not necessarily corresponding to the columns.

And we find $U$ in this way:
Wanting to make the rows of $UR$ be the rows of $P=R^T(R{R}^T{)}^{-1}R$ corresponding to the columns of $A$, we just picked some rows of $R^T$, right-times it by $(R{R}^T{)}^{-1}$, which comes to $U$

Proof

$$\Vert A-CUR{\Vert }_2\le \Vert A-AP{\Vert }_2+\Vert AP-CUR{\Vert }_2$$

Note:
The two norm of matrix $A$ is $\Vert A{\Vert }_2\stackrel{def}{=}\underset{x}{\max }\Vert Ax{\Vert }_2$
satisfying the triangle inequality $\Vert A+B{\Vert }_2\le \Vert A{\Vert }_2+\Vert B{\Vert }_2$

first part:
$$\Vert A-AP{\Vert }_2^2=\underset{x}{\max }\Vert Ax-APx{\Vert }^2$$
For the $x$ in row space of $R$, $\Vert Ax-APx{\Vert }_2=0$, so just consider the $x\perp$ row space of $R$
$$\begin{array}{rl}\Vert Ax-APx{\Vert }^2& =\Vert Ax{\Vert }^2\\ & =x^T{A}^{T}Ax=x^T(A^{T}A-R^{T}R)x\\ & \le \Vert x\Vert \Vert (A^{T}A-R^{T}R)x\Vert \\ & \le \Vert A^{T}A-R^{T}R{\Vert }_2\\ & \le \frac{\Vert A{\Vert }_F^4}{r}\end{array}$$

Note:
The last step comes from $E\left(\Vert AB-CR{\Vert }_F^2\right)\le \frac{1}{s}\Vert A{\Vert }_F^2\Vert B{\Vert }_F^2$. Replace $A$ by $A^T$ and replace $B$ by $A$

second part:
$$\begin{array}{rl}E(\Vert AP-CUR{\Vert }_2)& \le E(\Vert AP-CUR{\Vert }_F)\\ & \le \frac{1}{s}\Vert A{\Vert }_F^2\Vert P{\Vert }_F^2\\ & \le \frac{r}{s}\Vert A{\Vert }_F^2\end{array}$$

Note:
The last step comes from $\Vert P{\Vert }_F^2\le r$
That is because $\Vert P{\Vert }_F^2=$ sum of squared singular value, having $r$ in total, each of which $\le 1$ because $\Vert Pv\Vert \le 1$ for any unit vector $v$ since $P$ is a projection.

Computation Step by Step

1. Pick $s$ columns of $A$ according to length squared sampling, which forms $C$
2. Pick $r$ rows of $A$ according to length squared sampling, which forms $R$
3. Compute $(R{R}^T{)}^{-1}$
4. Pick $s$ rows of $R^T$ corresponding to step 1, and right-times the resulting matrix by $(R{R}^T{)}^{-1}$, which forms $U$

Note:
What if the matrix $R{R}^T$ doesn’t have an inverse?
Use psedo inverse, which can be represented and computed based on some matrix decomposition, such as SVD.
As for why the pseudo inverse work, see this passage to know about the intuition behind pseudo inverse and you will understand.
Whatever, the $R{R}^T$ is a $r\times r$ matrix, so the time complexity is quite low.