EigenDecomposition

For any real symmetric square $d \times d$ matrix A , we can find its eigenvalues $\lambda_1 \ge \lambda_2 \ge ... \ge \lambda_d$ and corresponding orthonormal eigenvectors, such that:

$Ax_1 = \lambda_1x_1$

$Ax_2 = \lambda_1x_2$
…
$Ax_d = \lambda_1x_d$

Suppose there are only m different non-zero eigenvalues, so we have

$Ax_1 = \lambda_1x_1$

$Ax_2 = \lambda_1x_2$
…
$Ax_r = \lambda_1x_r$

which can be written into a compact form such as ( $U^T = U^{-1}$ and ( $UU^T=I$ ):

$A=U\Sigma U^T \tag{1}$

$\Sigma=UAU^T \tag{2}$
Then we use above matrix $A$ to tranform a vector x, then

$\begin{aligned} Ax & =U\Sigma U^Tx \\ & = \left[ \begin{matrix} x_1 & x_2 \cdots & x_r \end{matrix} \right] \left[ \begin{matrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_r \end{matrix} \right] \left[ \begin{matrix} x_1^T \\ x_2^T\\ \cdots\\ x_r^T \end{matrix} \right] x \\ \end{aligned}$

Start the calculation from right to left:

Re-Orient: Apply $U^T$ on x <==> dot proct of x with each rows of $U^T$ . x will be projected on each orthonormal basis (new coordinate) which are the rows of the $U^T$ . The geometric meaning of $U^Tx$ is equivalent to rotate x based on the new cordinate (orthonormal basis) in $U^T$ .
$\Sigma U^T x$ is to use $\Sigma$ to extend or shrink the rotated $x$ along with the new coordinate. (If there is one $\lambda$ =0, then that dimension will be removed. )
Re-Orient(to original):Same meaning with $U^T$ . $U$ will rotate back x to original coordinate.

PCA

Main idea: Find a coordinate transformation matrix P applied on data suc that
(一种直观的看法是：希望投影后的投影值尽可能分散。数据越分散，可分性就越强，可分性越强，概率分布保存的就越完整)

The variability of the transformed data can be explained as much as possible along the new coordinates (降维后的信息损失尽可能小，尽可能保留原始样本的概率分布)
For each new coordinate, they should be orthoganal to each other to avoid the redundancy. (降维后的基之间是完全正交的)
So the final objective function is:
$\argmin_{P \in R^{m,d}, U\in R^{d,m}} \sum^{n}_{i=1} ||x_i - UPx_i||^2_2 \tag{*}$
, where d is the original dimension of data and m is the new dimension of the transoformed data (r=rank(A) <=m ), $x_1,..., x_n \in R^d$ , $P\in R^{d, m}$ is the transformation of new coordinate, $U \in R^{d,m}$ (in the end $U = P^T$ )is the transoformation of original coordinate.
In order to figure out above problems, the essential idea is to maxmize the variances of transfomred data point projected on new coordinates, and at the same time we need to minimize the covariance between two different coordinates(orthogonal<==>covarince is 0). Therefore, equivalently, we can diagolize the symmetric covariance matrix of the transformed data.

Example (Let d =2).

Put data into matrix X:

$X=\left[ \begin{matrix} x_1, x_2, ...,x_n \end{matrix} \right] =\left[ \begin{matrix} a_1 & a_2 & ... &a_n\\ b_1 & b_2 & ... &b_n \end{matrix} \right] \in R^{d=2, n}$

Suppose the data points has been centralized among each features. Therefore, the covariance matrix is the following:

$A=\frac{1}{n}XX^T= \left[ \begin{matrix} \frac{1}{n}\sum_{i=1}^{n} a^2_i & \frac{1}{n}\sum_{i=1}^{n} a_ib_i\\ \frac{1}{n}\sum_{i=1}^{n} a_ib_i\ & \frac{1}{n}\sum_{i=1}^{n} b^2_i \end{matrix} \right]$

Transformed data by matrix P is $Y = PX$ , where $P\in R^{m,d=2}$ . Then the transofrmed covariance matrix is:

$\begin{aligned} \frac{1}{n}YY^T & = \frac{1}{n}(PX)(PX)^T \\ & = \frac{1}{n}PXX^TP^T \\ & = P(\frac{1}{n}XX^T)P^T \\ & = PAP^T \\ \end{aligned}$

Once the minimization (*) is done , $\frac{1}{n}YY^T$ will look like $\Sigma$ . Underline technique used to achieve the goal is the formula (2) , so that
$\begin{aligned} PAP^T & = \Sigma \\ & = \left[ \begin{matrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_m \end{matrix} \right] \end{aligned}$

equivalently, $A=P^T \Sigma P$

where $\lambda_1 =\sigma_1^2 \ge \lambda_2=\sigma_2^2 \ge ... \ge \lambda_m=\sigma_m^2$ , their corresponding eigenvectors construct the transformation matrix $P=\left[ \begin{matrix} u_1^T\\ u_2^T\\ ...\\ u_m^T \end{matrix} \right] \in R^{m, d=2}$ , depends on your problems, we can select different r<=m rows in P.

在这里插入图片描述

SVD

$A$ will not be a symmetric matrix but with $m \times n$ dimension.
注意(SVD)： Original Feature is n (the number of colmns), New Feature is m
注意(PCA above) ： Original Feature is m (the number of rows)

Formula of Singular Value Decomposition
$A_{m,n} = U \Sigma V^T$

The main word of SVD is trying to solve the following two tasks:

First, find a set of orthonormal bais in n dimensional space
Second, once performed on above space, the resulting basis in m dimensional space are still orthonormal.

Suppose we have already got the set of orthonormal bais in n dimensional space $V_{n,n} = \left[ v_1, v_2, ..., v_n \right]$ (note: some vs can be 0 vectors), where $v_i \perp v_j$

Project A on n dimensional bais, we have:
$\left[ Av_1, Av_2, ..., Av_n \right]$

To make the projected basis be orthogonal as well, we need let the following happen:
$\begin{aligned} Av_i \cdot Av_j & = (Av_i)^T Av_j \\ & = v_i^T A^T A v_j \\ & = 0 \end{aligned}$
Therefore, if $v_i s$ are eigen vectors of $A^TA$ , then the resulting projected basis are also orthogonal because:
$\begin{aligned} Av_i \cdot Av_j & = (Av_i)^T Av_j \\ & = v_i^T (A^T A v_j )\\ & = v_i^T\lambda_j v_j \\ & = 0 \end{aligned}$
To scale the projected orthogonal basis into unit length.
We have
$\begin{aligned} u_i & = \frac{ Av_i}{|Av_i|} \\ & = \frac{Av_i}{\sqrt{|Av_i|^2}} \\ & = \frac{Av_i}{\sqrt{(Av_i)^TAv_i)}} \\ & = \frac{Av_i}{\sqrt{v_i^T(A^TAv_i)}} \\ & = \frac{Av_i}{\sqrt{v_i^T\lambda_iv_i}} (note: v_i^Tv_i=1)\\ & = \frac{ Av_i}{\sqrt{\lambda_i}} \\ \end{aligned}$

$u_i \sqrt{\lambda_i}= u_i \sigma_i = Av_i,$ , where $\sigma_i = \sqrt{\lambda_i}$ is called singular value, $0 \le i \le r$ , r=rank(A).

In the end, we expand $\left[ u_1, u_2, ...,u_r\right]$ to $\left[ u_1, u_2, ...,u_r | u_{r+1}, ..., u_m\right]$ and select $\left[ v_{r+1}, v_{r+1}, ...,v_{m}\right]$ from the nullspace of A where $Av_i=0$ , i>r, and $\sigma_i=0$
下图中k就是r