【机器学习】基础之线性代数(超详细总结)

此博文markdown原文链接，猛戳这里！！

标量、向量、矩阵和张量

• 标量$(\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{a}\mathrm{r})$：一个单独的数，一般用斜体表示标量，通常是小写字母。定义标量时，通常会指出其所属类型。比如定义自然数标量时，会这样定义"$令n\in \mathbb{N}$表示元素数目"。
• 向量$(\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r})$：一列数，并且有序排列。通过索引可以定位到每个单独的数。通常用小写的斜粗体字母表示。通常也会指出其所属类型。若每个元素都属于$R$ , and this vector has$n$ elements, then the vector input real number set$R$ 's$A\; set\; composed\; of\; n$ Cartesian products, denoted as${\mathbb{R}}^n$，即：$\mathbf{x}\in R^n$。

$$x=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]$$

​ You can think of a vector as a point in space, and each element is a coordinate on a different coordinate axis.

• matrix $\left(matrix\right)$ : A two-dimensional array, each element is uniquely identified by two indices. Usually use italic bold uppercase to represent the matrix, such as$A.$ _ If the height of the matrix is$m$ with width$n$ . Then say$\mathbf{A}\in {\mathbb{R}}^{m\times n}$。$A_{m,n}$to represent $m$ row$The\; elements\; of\; n$ columns. meansiiA i , : A_{i, :}for$i\; -row\; elements$$A_{i,:}$, means $Use\; A\; for\; j$ column elements$A_{:,j}$。

$$A=\left[\begin{array}{cc}{A}_{1,1}& A_{1,2}\\ A_{2,1}& A_{2,2}\end{array}\right]$$

​ Sometimes it is necessary to operate the matrix element by element, you can use $f(\mathbf{A}{)}_{i,j}$means $f$ acts on matrix elements$A_{i,j}$superior.

• Tensor $\left(tensor\right)$ : Arrays with more than two dimensions. Use$A$ represents a tensor,$A_{i,j,k}$to represent its elements.

import numpy as np
# 标量 
s = 5
print(s)
# 向量
v = np.array([1, 2])
print(v)
# 矩阵
m = np.array([[1, 3], [2, 4]])
print(m)
# 张量
t = np.array([
    [[1,2,3], [2,3,4]],
    [[3,4,5], [4,5,6]],
    [[5,6,7], [6,7,8]]
])
print(t)

Matrix transpose

Matrix transposition is to flip the original matrix along the main diagonal, matrix A \boldsymbol AThe transpose of$A$$\mathbf{A}=A^T$ , defined as follows
$$({\mathbf{A}}^{\mathrm{T}}{)}_{i,j}=A_{j,i}$$
A vector can be viewed as a matrix with only one column, and the transpose of a vector can be viewed as a matrix with only one row. A scalar can be thought of as a matrix with one row and one column whose transpose is equal to itself. $a^{\mathrm{T}}=a$.

A = np.array([[1, 2, 3],[1, 0, 2]])
A_t = A.transpose()

Matrix addition

Addition corresponds to element-wise addition, requiring that the two matrices must have the same shape.
$$\mathbf{C}=A+\mathbf{B},C_{i,j}=A_{i,j}+B_{i,j}$$
标量和矩阵相乘，或者和矩阵相加时，只需要将其与矩阵的每个元素相乘或者相加。
$$\mathbf{D}=a\cdot \mathbf{B}+c,其中D_{i,j}=a\cdot B_{i,j}+c$$

A = np.array([
    [1, 2, 3], [4, 5, 6]
])

B = np.array([
    [1, 2, 3], [4, 5, 6]
])

C = A + B
print(C)
# [[ 2  4  6]
#  [ 8 10 12]]

D = 2 * B + 1
print(D)
# [[ 3  5  7]
#  [ 9 11 13]]

矩阵和向量相乘

向量内积：两个向量的对应元素相乘并相加。
$$\mathbf{a}\cdot \mathbf{b}=\left|\mathbf{a}\right||\mathbf{b}|\cos \theta ,\theta 是两个向量的夹角$$
从上述表达式中我们可以看出，向量内积的物理含义就是向量$\mathbf{a}$在 b \boldsymbol b The projection length in the$b$ direction, if$The\; modulus\; of\; b$ is$1$，则$\mathbf{a}\cdot \mathbf{b}=\left|\mathbf{a}\right|\cos \theta$ , this conclusion can also be applied to matrix multiplication, which we will use in the following matrix linear transformation.

Matrix multiplication : matrix ${\mathbf{A}}_{m\times p}$And matrix ${\mathbf{B}}_{p\times n}$The product of can be expressed as
$${\mathbf{C}}_{m\times n}=A_{m\times p}{B}_{p\times n}$$
Among them, the matrix $\mathbf{A}$的列数和矩阵$\mathbf{B}$的行数必须相同，得到的矩阵$\mathbf{C}$是$m$行$n$列。具体的乘法操作如下
$$C_{i,j}=\sum _k{A}_{i,k}{B}_{k,j}$$
该矩阵乘积不是矩阵对应元素相乘，但是这种操作也确实存在，被称为元素对应乘积或者叫Hadamard乘积，记为$\mathbf{A}⨀\mathbf{B}$。

矩阵相乘服从分配律和结合律，但并不满足交换律。两个向量的点积满足交换律：
$${\mathbf{x}}^{\mathrm{T}}\mathbf{y}={\mathbf{y}}^{T}x$$
where$x$ is$n$ row$1$ column, after the transpose is$1$ row$n$ columns, multiplied by$y$ is$n$ row$1$ column, the final result is$1$ line$1$ column, the same result as the right.
The transpose of the matrix product is
$$(\mathbf{A}\mathbf{B}{)}^{\mathrm{T}}={\mathbf{B}}^T{A}^{\mathrm{T}}$$
两个向量的点积是标量，标量转置的结果是自身，可以通过上述结论得到
$${\mathbf{x}}^{\mathrm{T}}\mathbf{y}=({\mathbf{x}}^{\mathrm{T}}\mathbf{y}{)}^{\mathrm{T}}={\mathbf{y}}^{\mathrm{T}}\mathbf{x}$$
了解矩阵乘积之后，我们就可以用矩阵乘积的方式来表示线性方程组$\mathbf{A}\mathbf{x}=\mathbf{b}$. 其中$\mathbf{A}\in {\mathbb{R}}^{m\times n}$是一个已知矩阵$m$行$n$列，$\mathbf{x}\in {\mathbb{R}}^n$是一个求解$n$ rows of unknown vectors,$\mathbf{b}\in {\mathbb{R}}^m$ is a known vector. This provides a more compact representation.

A = np.array([
    [1, 2, 3], [4, 5, 6]
])

A1 = np.array([
    [1, 2, 3], [2, 3, 4]
])

B = np.array([
    [1, 2, 3], [3, 4, 5], [5, 6, 7]
])

C = A.dot(B)  # 点乘 矩阵乘积
C = np.dot(A, B)
# C = np.dot(B, A)  # 不满足交换律

print("矩阵(点)乘积: \n", C)
# [[22 28 34]
#  [49 64 79]]

print("矩阵逐元素乘积: \n", np.multiply(A, A1))
print("矩阵逐元素乘积: \n", A * A1)
# [[1  4  9]
#  [8 15 24]]

v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])
v = v1.dot(v2)  # 向量内积  结果为标量
v = np.dot(v1, v2)
v = np.dot(v2, v1)  # 满足交换律
print("向量内积: ", v)
# 32

Inverse and Identity Matrix

Identity matrix $\left(identitymatrix\right)$ , all elements along the main diagonal are 1, and all other elements are 0. Any matrix and identity matrix product is unchanged. The identity matrix is ​​written as${\mathbf{I}}_n\in R^{n\times n}$ .
Matrix$The\; matrix\; inverse\; of\; A$ is written as${\mathbf{A}}^{-1}$，满足
$$A^{-1}\mathbf{A}={\mathbf{I}}_n$$
因此，$\mathbf{A}\mathbf{x}=\mathbf{b}$中可求得$\mathbf{x}={\mathbf{A}}^{-1}\mathbf{b}$.

A = [[1.0,2.0],[3.0,4.0]]
A_inv = np.linalg.inv(A)
print(A_inv)
# [[-2.   1.]
#  [1.5 - 0.5]]

线性相关和生成子空间

线性相关：若$\mathbf{a}=[{\mathbf{a}}_1,{\mathbf{a}}_2,\cdots ,{\mathbf{a}}_{\mathbf{n}}]$中每个向量都能由其他向量线性表示，则$\mathbf{a}$是线性相关的。也就是说，存在不全为$0$的数$k_1,k_2,\cdots ,k_n$，使得
$$k_1{\mathbf{a}}_1+k_2{\mathbf{a}}_2+\cdots +k_n{\mathbf{a}}_n=0$$
holds, thena \boldsymbol aThe vectors in$a\; are\; linearly\; related.$Otherwise, linearly independent. If they are orthogonal to each other, then they are also linearly independent.

If the inverse matrix ${\mathbf{A}}^{-1}$ exists, then for each vector$b$ , there must be a solution. But for the system of equations, for the vector$Some\; values\; of\; b$ may have no solution, or infinitely many solutions. It is impossible for there to be more than one solution but less than an infinite number of solutions. Because if x and y are both equation solutions, then
$$\mathbf{z}=\alpha \mathbf{x}+(1-\alpha )y$$
is also a solution of this equation.

The matrix must be a square matrix, and all column vectors are linearly independent . A square matrix whose column vectors are linearly dependent is said to be singular. If matrix A is not a square matrix or is a singular square matrix, the square matrix may still have a solution, but it cannot be solved by matrix inversion.

Norm

Norm $\left(norm\right)$ is used to measure the vector size. Formally$L^{The\; P}$ norm is defined as follows:
$${\Vert \mathbf{x}\Vert }_p={\left(\sum _i|x_i{|}^p\right)}^{\frac{1}{p}}$$
where $p\in \mathbb{R},p\ge 1$。

A norm is a function that maps a vector to nonnegative values. Intuitively, a vector x \boldsymbol xThe norm of$x$$x$ -distance.

$L^2$ norm: also known as Euclidean norm,$||x|{|}_2$is the Euclidean distance. Sometimes $L^{2-}$ norm squared to measure vectors:${\mathbf{x}}^{T}x$. Square$L^{The\; 2}$ norm is more convenient in calculation. For example it is for$Each\; component\; of\; the\; gradient\; of\; x$ depends only on the corresponding components of x, and$L^2$ norm is for$Each\; component\; of\; the\; gradient\; of\; x$ depends on the entire$x$向量。
$${\Vert \mathbf{x}\Vert }_2=\sqrt{\sum _i{x}_i^2}$$
$L^1$ norm:$L^{The\; 2}$ norm is not necessarily applicable to all situations. It grows very slowly near the origin, so it is not suitable for the need to distinguish$0$ and very small but non-$0$ value case. $L^{The\; 1}$ norm is a better choice, andits growth rate is the same in all directions. Defined as follows
$${\Vert x\Vert }_1=\sum _i|x_i|$$
$L^{\infty }$ norm: In the mathematical sense, it is the maximum value of the absolute value of the vector elements, also known as$\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}$。
$${\Vert x\Vert }_{\infty }=\underset{i}{\max }|x_i|Measurement$$
matrix used in machine learning is$F$范数$(\mathrm{F}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{u}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m})$，定义如下：
$${\Vert \mathbf{A}\Vert }_F=\sqrt{\sum _{i,j}{A}_{i,j}^2}$$
其类似于$L^2$范数，两个向量的点积也可以用范数来表示，
$${\mathbf{x}}^{\mathrm{T}}\mathbf{y}={\Vert \mathbf{x}\Vert }_2{\Vert \mathbf{y}\Vert }_2\cos \theta$$

v = np.array([1, 2, 3, 4])
print("向量1范数: ", np.linalg.norm(v, ord=1))
print("向量2范数: ", np.linalg.norm(v, ord=2))
print("向量无穷范数: ", np.linalg.norm(v, ord=np.inf))
m = np.array([
    [1, 2],
    [3, 4]
])
print("矩阵F范数: ", np.linalg.norm(m, ord="fro"))

# 向量1范数:  10.0
# 向量2范数:  5.477225575051661
# 向量无穷范数:  4.0
# 矩阵F范数:  5.477225575051661

线性空间

• 定义：设 $V$ is a non-empty set,$R$ is a real number field, if for any two elements$a,b\in V$ , there is always a unique element$c\in$ $V$ corresponds to it, called$\alpha$ and$The\; sum\; of\; \beta$ , recorded as$c=a+\beta$ ; for any number$l\in R$ and any element$a\in V$ , there is always a unique element$d\in V$ corresponds to it, called$\lambda$ and$\mathbf{\alpha }$ 的积,记为 $\mathbf{\delta }=\lambda \mathbf{\alpha }$. 这两种运算满足下面各条运算规律:

1. 交换律 $\mathbf{\alpha }+\mathbf{\beta }=\mathbf{\beta }+\mathbf{\alpha }$;

2. 结合律 $(\mathbf{\alpha }+\mathbf{\beta })+\mathbf{r}=\mathbf{\alpha }+(\mathbf{\beta }+\mathbf{\gamma })$;

3. 有零元 $0\in V$. 使任意 $\mathbf{\alpha }\in V$ 有 $0+\mathbf{\alpha }=\mathbf{\alpha }$;

4. There are negative elements for any $a\in V$ , all haveα \boldsymbol{\alpha}The negative element of$\alpha$$b\in V$, 使 $a+b=0$;

5. With identity $1\cdot a=a$ ?

6. Derivative $\lambda \left(ma\right)=\left(lm\right)a$ ;

7. Random range $(\lambda +\mu )\mathbf{\alpha }=\lambda \mathbf{\alpha }+\mu \mathbf{\alpha };$

8. 对元素的分配律 $\lambda (\mathbf{\alpha }+\mathbf{\beta })=\lambda \mathbf{\alpha }+\lambda \mathbf{\beta }$.

   则称 $V$ 为线性空间(或向量空间), $V$ 中的元素统称为向量.

• 线性空间的基：若线性空间$V$中存在$n$个元素$[{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\cdots ,{\mathbf{\alpha }}_n]$ , they arelinearly independentandVVAny element in$V$$\alpha$可由$[a_1,a_2,\cdots ,a_n]$ linear representation, it is called$[a_1,a_2,\cdots ,a_n]$ is the linear spaceVVA basis of$V$$n$ represents the dimension of the space.

• Coordinates : Suppose $[a_1,a_2,\cdots ,a_n]$ is the linear spaceVVA basis of$V$ , for VVAny element in$V$$\alpha$ , you can use a set of ordered numbers$(x_1,x_2,x_3,\cdots ,x_n)$，使得
  $$a=x_1{a}_1+x_2{a}_2+\cdots +x_n{a}_n$$
  则$(x_1,x_2,x_3,\cdots ,x_n{)}^T$ becomes the element$\alpha$在基$[a_1,a_2,\cdots ,a_n]$ under the coordinates. Write
  $$a={\left(x_1,x_2,\cdots ,x_n\right)}^{\mathrm{T}}$$

• Basis transformation : Let $[a_1,a_2,\cdots ,a_n]$和$[b_1,b_2,\cdots ,b_n]$ are all bases in linear spaces, and
  $$\{\begin{array}{l}{b}_1=p_{11}{a}_1+p_{21}{a}_2+\cdots +p_n{a}_n\\ b_2=p_{12}{a}_1+p_{22}{a}_2+\cdots +p_n{a}_n\\ ⋮\\ b_n=p_{1n}{a}_1+p_2{a}_2+\cdots +p_{nn}{a}_n\end{array}$$
  等价于
  $$\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]={\mathbf{P}}^{\mathrm{T}}\left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]$$
  或者
  $$\left(b_1,b_2,\cdots ,b_n\right)=\left(a_1,a_2,\cdots ,a_n\right)The\; above\; formula\; of\; P$$
  is called the base coordinate transformation formula,$P$ is calledtransition matrix, because$\left(b_1,b_2,\cdots ,b_n\right)$ is linearly independent, so the matrix$P$ is reversible.

• Coordinate transformation : assume linear space VVVector a \boldsymbol ain$V$$\mathbf{a}$在基$[{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\cdots ,{\mathbf{\alpha }}_n]$下的坐标为$(x_1,x_2,x_3,\cdots ,x_n{)}^{\mathrm{T}}$，在基$[{\mathbf{\beta }}_1,{\mathbf{\beta }}_2,\cdots ,{\mathbf{\beta }}_n]$下的坐标为$(x_1^{\prime },x_2^{\prime },x_3^{\prime },\cdots ,x_n^{\prime }{)}^{\mathrm{T}}$,且两个基满足上述基变换关系，则有
  $$\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right)=\mathbf{P}\left(\begin{array}{c}{x}_1^{\prime }\\ x_2^{\prime }\\ ⋮\\ x_n^{\prime }\end{array}\right),\text{ 或 }\left(\begin{array}{c}{x}_1^{\prime }\\ x_2^{\prime }\\ ⋮\\ x_n^{\prime }\end{array}\right)={\mathbf{P}}^{-1}\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right)$$
  证明：令$\mathbf{x}=(x_1,x_2,\cdots ,x_n{)}^{\mathrm{T}},{\mathbf{x}}^{\prime }=(x_1^{\prime },x_2^{\prime },\cdots x_n^{\prime }{)}^{\mathrm{T}},\mathrm{A}=[{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\cdots ,{\mathbf{\alpha }}_n],\mathrm{B}=[{\mathbf{\beta }}_1,{\mathbf{\beta }}_2,\cdots ,{\mathbf{\beta }}_n]$.$\left(Aiscapital\alpha notEnglishletterA\right)\_$

  则
  $$\begin{gathered} \mathrm{A}\mathbf{x}=a=\mathrm{B}{\mathbf{x}}^{\prime }=\mathrm{A}\mathbf{P}{x}^{\prime } \\ \mathbf{x}=\mathbf{P}{\mathbf{x}}^{\text{'}} \end{gathered}$$
  The above formula is established.

• Linear mapping (transformation) : Let $V_n,U_m$are two linear spaces respectively, $T$ is a$V_n$到$U_m$mapping, if the mapping $T$ satisfies:

  对$\forall {\mathbf{\alpha }}_1,a_2\in V_n$，有$T(a_1+a_2)=T\left(a_1\right)+T(a_2)$；

  对$\forall \alpha \in V_n,l\in \mathbb{R}$，有$T\left(la\right)=\lambda T\left(a\right)$ .

  then $T$ is referred to from$V_n$到$U_m$The linear map or linear transformation of .

  The geometric meaning of matrix transformation : To accurately describe a vector, we must clarify the basis of the vector and find out the projection length of the vector on each component of the basis . Assuming that the research is based on a two-dimensional Cartesian coordinate system, $\mathbf{x}=(1,0{)}^{\mathrm{T}},\mathbf{y}=(0,1{)}^T$ as a set of basis in the space. The form expressed as a matrix is
  $$\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$ According to the physical meaning of the vector inner product, the vector ( 4 , 2 ) (4, 2)
  in the two-dimensional Cartesian coordinate system$(4,2)$ is expressed as:
  $$\begin{gathered} (4,2)\mathbf{x}=4 \\ (4,2)\mathbf{y}=2 \\ {\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)}^{\mathrm{T}}\left(\begin{array}{c}4\\ 2\end{array}\right)=\left(\begin{array}{c}4\\ 2\end{array}\right) \end{gathered}$$
  It can be seen that bycalculating the projection value of the vector on each component of the base, its coordinates in the space defined by the base can be obtained. Generally we want the base to have a modulus of$1.$ From the geometric meaning of the vector inner product, we can directly find the coordinates of the vector on the base.
  Similarly, suppose we want$m$ nn$n-$ dimensional vector, mapped to a space represented by a new basis, where the basis is$k$ nn_$n$维向量，可以表示如下：${\mathbf{X}}_{n\times m}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_m]$，${\mathbf{x}}_i$是一个$n$维列向量，${\mathbf{P}}_{n\times k}=[{\mathbf{p}}_1,{\mathbf{p}}_2,\cdots ,{\mathbf{p}}_k]$，${\mathbf{p}}_i$是一个$n$维列向量，每个列向量是一个基向量。
  $${\mathbf{P}}_{n\times k}^{\mathrm{T}}{\mathbf{X}}_{n\times m}=\left[\begin{array}{c}{\mathbf{p}}_1^{\mathrm{T}}\\ {\mathbf{p}}_2^{\mathrm{T}}\\ ⋮\\ {\mathbf{p}}_k^{\mathrm{T}}\end{array}\right]\left[\begin{array}{c}{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_m\end{array}\right]={\left[\begin{array}{cccc}{\mathbf{p}}_1^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{p}}_1^{\mathrm{T}}{\mathbf{x}}_2& \cdots & {\mathbf{p}}_1^{\mathrm{T}}{\mathbf{x}}_m\\ {\mathbf{p}}_2^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{p}}_2^{\mathrm{T}}{\mathbf{x}}_2& \cdots & {\mathbf{p}}_2^{\mathrm{T}}{\mathbf{x}}_m\\ ⋮& ⋮& \ddots & ⋮\\ {\mathbf{p}}_k^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{p}}_k^{\mathrm{T}}{\mathbf{x}}_2& \cdots & {\mathbf{p}}_k^{\mathrm{T}}{\mathbf{x}}_m\end{array}\right]}_{k\times m}={\mathbf{Z}}_{k\times m}$$
  由上述表达式可知，最终计算出的${\mathbf{Z}}_{k\times m}$矩阵就是矩阵$\mathbf{X}$经过基$\mathbf{P}$变换之后的结果，其中${\mathbf{p}}_i^{\mathrm{T}}{\mathbf{x}}_j$是标量 。矩阵相乘实质就是将右边矩阵中的每一列向量变换到以左边矩阵行向量为基表示的空间中去。通过观察变换后的矩阵，我们发现该矩阵的维度可能发生改变，因此可以实现矩阵降维。当然，要想将该矩阵变换到原先基表示的空间中，只需要在变换之后的矩阵左乘${\mathbf{P}}^{-1}$即可。$\mathbf{P}$通常被假设为一组标准正交基，这样一来更加方便计算。比如可以利用性质${\mathbf{P}}^{-1}\mathbf{P}=\mathbf{I},{\mathbf{P}}^{\mathrm{T}}\mathbf{P}=\mathbf{I},{\mathbf{P}}^{-1}={\mathbf{P}}^{\mathrm{T}}$。但只要矩阵$\mathbf{P}$是的列向量线性无关即可进行坐标变换。

特殊矩阵和向量

• 对角矩阵$\left(diagonalmatrix\right)$ : only contain non-zero elements on the main diagonal, and all other positions are zero$.$An identity matrix is ​​a diagonal matrix. use$diag\left(v\right)$ represents a diagonal element consisting of vectorv \boldsymbol vThe diagonal square matrix given by the elements in$v\; .$

• The multiplication of diagonal matrices is very convenient , to calculate $diag\left(v\right)x$ , just putx \boldsymbol xEach element xi x_iin$x$$x_i$big $v_i$times.

• Not all diagonal matrices are square; rectangular matrices can be diagonal as well. Diagonal matrices that are not square do not have an inverse, but matrix multiplication can still be computed efficiently.

• Symmetric matrix $\left(symmetricmatrix\right)$ is a matrix whose transpose is equal to itself.

• Unit vector $\left(unitvector\right)$ is a vector with unit norm, that is, the 2 norm is$1$。

• Orthogonal : if the dot product of the vector is $0$ , then the two vectorsare orthogonal. If both vectors have non-zero norm, then the angle between the two vectors is$90^{\circ }$。在${\mathbb{R}}^n$中，至多有$n$个范数非零的向量互相正交，若这些向量不仅互相正交，而且范数都为$1$，则为标准正交。

• 正交矩阵$(\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{g}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x})$：指行向量和列向量分别标准正交的方阵，也就是
  $$\begin{gathered} {\mathbf{A}}^{\mathrm{T}}\mathbf{A}=\mathbf{A}{\mathbf{A}}^{\mathrm{T}}=\mathbf{I} \\ {\mathbf{A}}^{-1}={\mathbf{A}}^T \end{gathered}$$
  orthogonal matrix inversion calculation cost is small. $|\mathbf{A}|=1$

  Suppose the matrix $A$ is a column vector matrix$\mathbf{A}=\left[{\mathbf{x}}_1,{\mathbf{x}}_2,x_3,\cdots ,x_n\right]$。

  then there is
  $$\begin{array}{rl}{\mathbf{A}}^{\mathrm{T}}\mathbf{A}& =\left[\begin{array}{c}{\mathbf{x}}_1^{\mathrm{T}}\\ {\mathbf{x}}_2^{\mathrm{T}}\\ {\mathbf{x}}_3^{\mathrm{T}}\\ ⋮\\ {\mathbf{x}}_n^{\mathrm{T}}\end{array}\right]\left[\begin{array}{c}{\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3,\cdots ,{\mathbf{x}}_n\end{array}\right]=\left[\begin{array}{ccccc}{\mathbf{x}}_1^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{x}}_1^{\mathrm{T}}{\mathbf{x}}_2& {\mathbf{x}}_1^{\mathrm{T}}{\mathbf{x}}_3& \cdots & {\mathbf{x}}_1^{\mathrm{T}}{\mathbf{x}}_n\\ {\mathbf{x}}_2^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{x}}_2^{\mathrm{T}}{\mathbf{x}}_2& {\mathbf{x}}_2^{\mathrm{T}}{\mathbf{x}}_3& \cdots & {\mathbf{x}}_2^{\mathrm{T}}{\mathbf{x}}_n\\ {\mathbf{x}}_3^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{x}}_3^{\mathrm{T}}{\mathbf{x}}_2& {\mathbf{x}}_3^{\mathrm{T}}{\mathbf{x}}_3& \cdots & {\mathbf{x}}_3^{\mathrm{T}}{\mathbf{x}}_n\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ {\mathbf{x}}_n^{\mathrm{T}}{\mathbf{x}}_1& {\mathbf{x}}_n^{\mathrm{T}}{\mathbf{x}}_2& {\mathbf{x}}_n^{\mathrm{T}}{\mathbf{x}}_3& \cdots & {\mathbf{x}}_n^{\mathrm{T}}{\mathbf{x}}_n\end{array}\right]\\ & =\mathbf{I}=\left[\begin{array}{ccccc}1& 0& 0& \cdots & 0\\ 0& 1& 0& \cdots & 0\\ 0& 0& 1& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& \cdots & 1\end{array}\right]\end{array}$$
  由上述推到可以看出，对于$\forall i\ne j,{\mathbf{x}}_i^{\mathrm{T}}{\mathbf{x}}_j=0$且对于$\forall i=j,{\mathbf{x}}_i^{\mathrm{T}}{\mathbf{x}}_j=1$，说明如果矩阵的各列向量都是单位向量，并且两两正交，那么该矩阵是正交矩阵。对于行向量来说，也是如此。
  $n$阶正交矩阵的$n$个行向量或列向量构成向量空间${\mathbb{R}}^n$的一个标准正交基。

    v = np.array([1, 2, 3])
    print("以v为对角元素的对角阵: \n", np.diag(v))
    # [[1 0 0]
    #  [0 2 0]
    #  [0 0 3]]
  
    v2 = np.arange(1, 10).reshape(3, 3)
    print(v2)
    # [[1 2 3]
    #  [4 5 6]
    #  [7 8 9]]
    print("二维矩阵的对角元素: \n", np.diag(v2))
    # 二维矩阵的对角元素:
    #  [1 5 9]
  

• Similarity matrix : Let $\mathbf{A},B$ are both$Matrix\; of\; order\; n$ , if there is an invertible matrix$P$ , such that${\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}=B$ , then call$B$ is$The\; similarity\; matrix\; of\; A$ , or the two are similar. In layman's terms, the two matrices are similar, and the essence is thatthe matrices of the same linear transformation under different bases are similar.
  Proof: Suppose there is a linear transformation$T$,存在两组不同的基$\mathbf{\alpha }=[{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\cdots ,{\mathbf{\alpha }}_n]，\mathbf{\beta }=[{\mathbf{\beta }}_1,{\mathbf{\beta }}_2,\cdots ,{\mathbf{\beta }}_n]$，两者之间的关系通过过渡矩阵来关联，也就是之前在"基变换中提到的$\mathbf{P}$"，则有
  $$\begin{gathered} b=aP \\ T\left(a\right)=\mathbf{\alpha }\mathbf{A} \\ T\left(b\right)=\mathbf{\beta }\mathbf{B}\Rightarrow T(\mathbf{\alpha }\mathbf{P})=\beta B\Rightarrow T\left(a\right)P=\mathbf{\alpha }\mathbf{P}\mathbf{B} \\ \Rightarrow \mathbf{\alpha }\mathbf{A}\mathbf{P}{\mathbf{P}}^{-1}=\mathbf{\alpha }\mathbf{P}\mathbf{B}{\mathbf{P}}^{-1}\Rightarrow \alpha A=\alpha PB{P}^{-1}\Rightarrow \mathbf{A}=\mathbf{P}\mathbf{B}{\mathbf{P}}^{-1} \\ \Rightarrow {\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}=B \end{gathered}$$
  is certified.

  The same linear transformation, the matrices under different bases are called similarity matrices.

  There is also a more popular way of proof,

  Proof : Assume a linear space VVVector a \boldsymbol ain$V$$\mathbf{a}$在基$\mathbf{\alpha }=[{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2,\cdots ,{\mathbf{\alpha }}_n]$下的坐标为$\mathbf{x}=(x_1,x_2,x_3,\cdots ,x_n{)}^{\mathrm{T}}$，在基$\mathbf{\beta }=[{\mathbf{\beta }}_1,{\mathbf{\beta }}_2,\cdots ,{\mathbf{\beta }}_n]$下的坐标为${\mathbf{x}}^{\prime }=(x_1^{\prime },x_2^{\prime },x_3^{\prime },\cdots ,x_n^{\prime }{)}^T$ , two bases satisfy$b=aP$ .

  • In $\beta$ ,$x\text{'}$ by$P$ becomesα \boldsymbol \alphaThe vector under$\alpha$$\mathbf{P}{\mathbf{x}}^{\prime }$；
  • In $\alpha$ , through the matrix$A$ completes the linear transformation —$\mathbf{A}\mathbf{P}{\mathbf{x}}^{\prime }$；
  • By ${\mathbf{P}}^{-1}$ turns back intoβ \boldsymbol \betaVector under$\beta$${\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}{\mathbf{x}}^{\prime }$;
  • Therefore in $\mathbf{\beta }$下，$\mathbf{B}{\mathbf{x}}^{\prime }=P^{-1}AP{x}^{\prime }$
  • Get $\mathbf{B}=P^{-1}\mathbf{A}\mathbf{P}$

  Proven.
  Or
  according to the above assumptions,
  $$\begin{gathered} \mathbf{A}\mathbf{x}=\mathbf{P}\mathbf{B}{\mathbf{x}}^{\prime } \\ \mathbf{x}=\mathbf{P}{\mathbf{x}}^{\prime } \\ \Rightarrow \mathbf{A}\mathbf{P}{\mathbf{x}}^{\prime }=PB{x}^{\prime }\Rightarrow \mathbf{B}={\mathbf{P}}^{-1}AP \end{gathered}$$
  Understand geometrically
  

  As can be seen from the figure above, the vector $x$ and$x^{\prime }$ The same vector$a$ has different coordinates in different bases, but the two are essentially the same. Things are seen differently from different angles. $Ax$ is$x$ in baseα \boldsymbol \alphaThe coordinates after linear transformation through the matrix under$\alpha$$\mathbf{B}{\mathbf{x}}^{\prime }$ is$x^{\prime }$ in base$The\; coordinates\; after\; linear\; transformation\; through\; the\; matrix\; under\; \beta$ , the coordinate space defined by the two bases can pass the transition matrix$P$ is transformed so that$\mathbf{A}\mathbf{x}=\mathbf{P}\mathbf{B}{\mathbf{x}}^{\prime }$。

Eigenvectors and Eigenvalues

The essence of matrix multiplication is space transformation.

Eigenvector: In a geometric sense, it is a vector whose direction does not change before and after space transformation . Indicates the direction of the transformation, such as stretching transformation refers to the direction of stretching. From this we can see that,

Eigenvalue: In a geometric sense, it is the degree of space transformation . Indicates the degree of transformation, such as how many times to scale in a certain direction.

Eigendecomposition: Decomposes a matrix into a set of eigenvectors and eigenvalues. Suppose $A$ is a$n$ -order matrix, number$\lambda$ and$n-$ dimensional non-zero vector$x$ satisfies
$$Ax=\lambda \mathbf{x}(\mathbf{x}\ne 0)$$
is called$\lambda$ is the matrixA \boldsymbol AEigenvalues ​​of$A$$x$ is called matrixA \boldsymbol AThe eigenvectors of$A.$

To understand the formula intuitively , it is equivalent to the vector $x$ passes through the matrixA \boldsymbol AAfter the spatial linear transformation of$A$ , only the expansion and contraction in the direction has been carried out, and no other changes have been made. This is the true meaning of the eigenvectors and eigenvalues. Solving the eigenvalues ​​and eigenvectors of the matrix is ​​to see which vectors the matrix can only change in direction (eigenvectors), and how much the change is (eigenvalues). The meaning is that we can discuss and research based on the extracted feature vectors.

Understand geometrically:

As can be seen from the above figure, from the square on the left through a matrix $After\; A$ is linearly transformed, it becomes the rectangle on the right, but we can easily find that the vector$i$ and$j$ is just transformed in direction, obviously, they are the matrixA \boldsymbol AThe eigenvectors of$A.$This transformation is expressed as follows:
$$\mathbf{A}=\left[\begin{array}{cc}2& 0\\ 0& 4\end{array}\right]$$

$$A\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}0\\ 4\end{array}\right]=4\left[\begin{array}{c}0\\ 1\end{array}\right]\Rightarrow {\mathbf{j}}^{\prime }=4j$$

$$A\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{c}2\\ 0\end{array}\right]=2\left[\begin{array}{c}1\\ 0\end{array}\right]\Rightarrow {\mathbf{i}}^{\prime }=2i\_$$

then $2$ and$4$ is their corresponding eigenvalues.

特征分解的另一种定义：假设矩阵$\mathbf{A}$有$n$个线性无关的特征向量$\mathbf{V}=[{\mathbf{v}}^{(1)},{\mathbf{v}}^{(2)},\cdots ,{\mathbf{v}}^{(n)}]$，对应的特征值分别为$\mathbf{\lambda }=[{\lambda }^{(1)},{\lambda }^{(2)},\cdots ,{\lambda }^{(n)}{]}^{\mathrm{T}}$,则矩阵$\mathbf{A}$的特征分解可以记为
$$\mathbf{A}=\mathbf{V}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\mathbf{\lambda }){\mathbf{V}}^{-1}$$
并不是所有的矩阵都可以分解特征值；在某些情况下，实矩阵的特征值分解可能会出现复矩阵。

m = np.array(
    [[1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]])

print("特征值: \n", np.linalg.eigvals(m))
eigvals, eigvecs = np.linalg.eig(m)
print("特征值: \n", eigvals)
print("特征向量: \n", eigvecs)

# 特征值: 
#  [ 1.61168440e+01 -1.11684397e+00 -1.30367773e-15]
# 特征值: 
#  [ 1.61168440e+01 -1.11684397e+00 -1.30367773e-15]
# 特征向量: 
#  [[-0.23197069 -0.78583024  0.40824829]
#  [-0.52532209 -0.08675134 -0.81649658]
#  [-0.8186735   0.61232756  0.40824829]]

奇异值分解

矩阵的奇异值分解$(\mathrm{S}\mathrm{V}\mathrm{D})$，将矩阵分解为奇异向量$(\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r})$和奇异值$(\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e})$。奇异值分解会得到一些与特征分解相同类型的信息。但是奇异值分解应用更为广泛。每个实数矩阵都有一个奇异值分解，但不一定都有特征分解。比如，非方阵的矩阵没有特征分解，此时只能使用奇异值分解。

奇异值分解的形式为
$$\mathbf{A}={\mathbf{U}\mathbf{\Sigma }\mathbf{V}}^{\mathrm{T}}$$
若$\mathbf{A}$是$m\times n$的矩阵，那么$\mathbf{U}$是$m\times m$的，其列向量成为左奇异向量，而$\mathbf{V}$是$n\times n$ , its column vector is calledright singular vector, and$\Sigma$ is$m\times Diagonal\; matrix\; of\; n$ whose diagonal elements arematrix A \boldsymbol AThesingular value of$A.$

Singular values ​​often correspond to important information hidden in the matrix, and the importance is positively correlated with the size of the singular values .

In the field of image processing, singular values ​​can not only be applied to data compression, but also to denoise images. If an image contains noise, we have reason to believe that those small singular values ​​are due to noise. When we force these small singular values ​​to 0, we can remove the noise in the picture.

The main application areas include:

• Latent Semantic Analysis (LSA) or Latent Semantic Indexing (LSI);
• Recommender system can be said to be the most valuable application point;
• Compression of matrix form data (mainly image data).

A singular matrix must be a square matrix. If the determinant of the matrix is ​​0, the matrix is ​​singular, otherwise it is non-singular . A non-singular matrix is ​​invertible, and a reversible matrix is ​​a non-singular matrix. If the matrix $A$ is a non-singular matrix, then$\mathbf{A}\mathbf{x}=0$有唯一零解，$\mathbf{A}\mathbf{x}=\mathbf{b}$有唯一解。

m = np.array([
    [1, 2, 3, 4],
    [4, 5, 6, 8]
])
U, Sigma, V = np.linalg.svd(m)

print("左奇异向量矩阵U: \n", U)
print("奇异值对角矩阵Sigma: \n", Sigma)
print("右奇异向量矩阵V: \n", V)

# 左奇异向量矩阵U: 
#  [[-0.41523775 -0.90971293]
#  [-0.90971293  0.41523775]]

# 奇异值对角矩阵Sigma: 
#  [13.04656085  0.88727114]

# 右奇异向量矩阵V: 
#  [[-0.31074009 -0.41229564 -0.51385119 -0.68513492]
#  [ 0.84668379  0.28938495 -0.26791389 -0.35721851]
#  [-0.25916053  0.51832106  0.54670221 -0.60439705]
#  [-0.34554737  0.69109474 -0.60439705  0.19413726]]

线性代数应用之主成分分析$(\mathrm{P}\mathrm{C}\mathrm{A})$

数据降维：就是将高维度特征数据降到低维度，保留一些比较重要的维度特征数据，去除噪声和不重要的信息，达到提升数据处理速度的目的。虽然降维以损失一些信息为代价，但是它尽可能地保留了原始数据的重要信息，这样就可以节省计算的时间成本、数据集更易使用、更易理解。

主成分分析$(\mathrm{P}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{i}\mathrm{p}\mathrm{a}\mathrm{l}\mathrm{C}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{A}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{y}\mathrm{s}\mathrm{i}\mathrm{s})$：该算法的主要思想就是将原始数据的$n$维特征降到$k$维。这$k$维正交特征也被称为主成分，是在原始$n$维特征上重构出来的$k$维特征。$\mathrm{P}\mathrm{C}\mathrm{A}$的主要工作就是在空间中找出一组两两互相正交的$k$个基向量，使得原始数据经过该新基的映射(变换)之后变成$k$维特征。新基的选取取决于原始数据的分布。

提出问题：如何选取$k$个正交基使得在变换之后能最大程度地保留原始数据的信息呢？如何才能使信息的损失度最小呢？通俗一点说，就是如何选取原始数据的投影方向，才能最大程度保留原始信息呢？

初步想法：原始数据投影之后，在投影方向上应该使得数据尽可能分散，其分散程度用方差表示。当然这只是将二维降到一维的情况。考虑到多维数据的情况，比如三维降到两维，首先选择的第一个投影方向肯定是投影后方差最大的方向，但是第二个方向必然不可能是方差最大的方向，否则就会使得投影方向重叠，数据无法区分了，为了表示更多的信息，我们希望在该方向投影后方差尽可能的大，而且投影方向不能和第一个投影方向线性相关(要是互相垂直正交，那就不存在相关性了，这也是我们寻求的最佳的投影方向)。

$\star \star \star$对应到线性代数的知识，就是计算原始数据矩阵的协方差矩阵(对角线元素就是每个维度的方差)，并进行协方差矩阵的特征分解，从大到小选择$k$个特征值(方差)及对应的特征向量，这$k$个特征向量就是选取的新基，原始数据在该基上投影之后就能从$n$维特征降到$k$维。

实现原理：在定义优化目标之前需要明确在正交属性空间中表达样本点的"两个性"，使用超平面对样本进行恰当表达。

• 最近重构性：样本点到超平面的距离都足够近；
• 最大可分性：样本点在这个超平面上的投影尽可能分散。

因此$\mathrm{P}\mathrm{C}\mathrm{A}$有两种等价推导，从最近重构性来看：

假设有$m$个$n$维列向量$\mathbf{X}=[{\mathbf{x}}^{(1)},{\mathbf{x}}^{(2)},\cdots ,{\mathbf{x}}^{\left(m\right)}]$, has been centered, that is,${\sum }_{i=1}^m{x}^{(i)}=0$ . If the data from$n$ dimension down to$K$ dimension, then assume that the projection transformation matrix is$\mathbf{P}=[{\mathbf{p}}_1,p_2,\cdots ,p_k]$，$P$ is an orthonormal basis, where${\Vert {\mathbf{p}}_i\Vert }_2=1,p_i^T{p}_j=0$。

由之前的坐标投影变换公式可知，${\mathbf{z}}^{(i)}={\mathbf{P}}^{\mathrm{T}}{\mathbf{x}}^{(i)}$，即可得到原始数据${\mathbf{x}}^{(i)}$在新基$\mathbf{P}$下的新空间坐标${\mathbf{z}}^{(i)}=[z_1^{(i)},z_2^{(i)},\cdots ,z_k^{(i)}{]}^{\mathrm{T}}$，重构成用基向量表示的形式就是${\hat{\mathbf{x}}}^{(i)}=\mathbf{P}{\mathbf{z}}^{(i)}$。
$$\left[\begin{array}{cccc}{p}_1& p_2& \cdots & p_k\end{array}\right]\left[\begin{array}{c}{z}_1^{(i)}\\ z_2^{\left(i\right)}\\ ⋮\\ z_k^{\left(i\right)}\end{array}\right]=z_1^{\left(i\right)}{p}_1+z_2^{\left(i\right)}{p}_2+\cdots +z_k^{\left(i\right)}{p}_k$$

Putting it in a two-dimensional space Cartesian coordinate system can actually be well understood, assuming we know the vector $x=(3,2{)}^{}$Coordinates of${}^T$$P=[{\mathbf{p}}_1,p_2]$ , then the known coordinates, represented by the basis vector, are$x=3{\mathbf{p}}_1+2{\mathbf{p}}_2$。$p_1=[1,0{]}^{\mathrm{T}},p_2=[0,1{]}^{\mathrm{T}}$.

So our optimization goal is to minimize ${\sum }_{i=1}^m{\Vert {\hat{x}}^{(i)}-x^{\left(i\right)}\Vert }_2$.
$$\min \sum _{i=1}^m{\Vert {\hat{\mathbf{x}}}^{(i)}-{\mathbf{x}}^{(i)}\Vert }_2⟺\min \sum _{i=1}^m{\Vert {\hat{\mathbf{x}}}^{(i)}-{\mathbf{x}}^{(i)}\Vert }_2^2$$

$$\begin{array}{rl}\sum _{i=1}^m{\Vert {\hat{\mathbf{x}}}^{(i)}-{\mathbf{x}}^{(i)}\Vert }_2^2& =\sum _{i=1}^m({\hat{\mathbf{x}}}^{(i)}{)}^{\mathrm{T}}({\hat{\mathbf{x}}}^{(i)})-2({\hat{\mathbf{x}}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})+({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})\\ & =\sum _{i=1}^m(\mathbf{P}{\mathbf{z}}^{(i)}{)}^{\mathrm{T}}(\mathbf{P}{\mathbf{z}}^{(i)})-2(\mathbf{P}{\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})+({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})\\ & =\sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{P}}^{\mathrm{T}}\mathbf{P})(z^{(i)})-2({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}{\mathbf{P}}^{\mathrm{T}}({\mathbf{x}}^{(i)})+({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})\\ & =\sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{(i)})-2({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{(i)})+({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})\\ & =-\sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{(i)})+\sum _{i=1}^m({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})\\ & =-\mathrm{t}\mathrm{r}\left({\mathbf{P}}^{\mathrm{T}}\left(\sum _{\mathrm{i}=1}^{\mathrm{m}}({\mathbf{x}}^{(\mathrm{i})})({\mathbf{x}}^{(\mathrm{i})}{)}^{\mathrm{T}}\right)\mathbf{P}\right)+\sum _{\mathrm{i}=1}^{\mathrm{m}}({\mathbf{x}}^{(\mathrm{i})}{)}^{\mathrm{T}}({\mathbf{x}}^{(\mathrm{i})})\\ & =-\mathrm{t}\mathrm{r}\left({\mathbf{P}}^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}\right)+\sum _{\mathrm{i}=1}^{\mathrm{m}}({\mathbf{x}}^{(\mathrm{i})}{)}^{\mathrm{T}}({\mathbf{x}}^{(\mathrm{i})})\end{array}$$

The sixth to seventh steps of the above formula are due to ${\sum }_{i=1}^m(x^{(i)})({\mathbf{x}}^{(i)}{)}^T$ is to find$\mathbf{X}{\mathbf{X}}^{}$The sum of the elements on the diagonal of the${}^T$ matrix, so it can be expressed by trace operations.

By observing the above formula, we find that ${\sum }_{i=1}^m(x^{\left(i\right)}{)}^{\mathrm{T}}({\mathbf{x}}^{(i)})$是常量，${\sum }_{i=1}^m(x^{(i)})({\mathbf{x}}^{(i)}{)}^T$ is the covariance matrixof the original data set, so minimizing the target expression is equivalent to
$$\arg \underset{P}{\min }-\mathrm{t}\mathrm{r}({\mathbf{P}}^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P})\mathrm{s}.\mathrm{t}.{\mathbf{P}}^{\mathrm{T}}\mathbf{P}=\mathbf{I}$$

常用迹运算及求导公式：
$$\begin{array}{rl}\mathrm{t}\mathrm{r}(\mathbf{A})& =\sum _i{A}_{i,i}\\ \mathrm{t}\mathrm{r}(\mathbf{A}\mathbf{B})& =\mathrm{t}\mathrm{r}(\mathbf{B}\mathbf{A})\\ \mathrm{t}\mathrm{r}(\mathbf{A})& =\mathrm{t}\mathrm{r}({\mathbf{A}}^{\mathrm{T}})\\ \mathrm{t}\mathrm{r}(\mathbf{A}\mathbf{B}\mathbf{C})& =\mathrm{t}\mathrm{r}(\mathbf{C}\mathbf{A}\mathbf{B})=\mathrm{t}\mathrm{r}(\mathbf{B}\mathbf{C}\mathbf{A})\\ \frac{\partial \mathrm{t}\mathrm{r}(\mathbf{A}\mathbf{B})}{\mathbf{A}}& =\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{B}\mathbf{A})}{\mathbf{A}}={\mathbf{B}}^{\mathrm{T}}\end{array}$$

Using the Lagrangian multiplier algorithm to optimize the above formula, we can get:
$$\begin{array}{r}J(\mathbf{P})=-\mathrm{t}\mathrm{r}\left({\mathbf{P}}^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}+\lambda (P^{T}P-\mathbf{I})\right)\\ Takethe\; partialderivative\; withrespect\; toP,\frac{\partial J}{\partial \mathbf{P}}=-{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}+\lambda P\_=0\\ ⟺{\mathbf{X}\mathbf{X}}^{T}P=\lambda P\end{array}$$
From the above final expression, according to the eigendecomposition of the matrix, we can conclude that $P$ is${XX}^{}$kkof${}^{T\; matrix}$$A\; matrix\; composed\; of\; k$ eigenvectors,$\lambda$ is its eigenvalue. So the final problem is transformed into finding${XX}^T$ 's ex$K$ large eigenvalues ​​and corresponding eigenvectors, where${XX}^T$ is the matrixX \boldsymbol XThe covariance matrix of$X.$

Finally, it is only necessary to map the raw data to P \boldsymbol PJust within the space defined by$P$
$$\mathbf{Z}=P^{\mathrm{T}}\mathbf{X}$$
$Z$ is the data after dimensionality reduction.

Next, let's look at the second derivation - based on maximum separability, that is, the variance of the data distribution after projection is as large as possible,

Based on the same assumptions of the previous derivation method, for any data $x^{\left(i\right)}$ , it is in the projection transformation matrixP \boldsymbol PThe coordinates under$P$$z^{(i)}=P^{\mathrm{T}}{\mathbf{x}}^{\left(i\right)}$ , the variance at the new coordinates is$({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{\left(i\right)})$, the goal is to maximize the sum of the variances of all data, that is,
$$\max \sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{\left(i\right)})$$

$$\begin{array}{rl}\sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{\left(i\right)})& =\sum _{i=1}^m({\mathbf{P}}^{\mathrm{T}}{\mathbf{x}}^{\left(i\right)}{)}^{\mathrm{T}}({\mathbf{P}}^T{x}^{\left(i\right)})\\ & =\sum _{i=1}^m(x^{\left(i\right)}{)}^{\mathrm{T}}\mathbf{P}{\mathbf{P}}^{\mathrm{T}}({\mathbf{x}}^{\left(i\right)})\\ & =\mathrm{t}\mathrm{r}\left(P^T\left(\sum _{i=1}^m(x^{(\mathrm{i})})({\mathbf{x}}^{\left(i\right)}{)}^T\right)\mathbf{P}\right)\\ & =tr\left(P^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}\right)\end{array}$$

It is known that the expression in the objective optimization formula is the sum of variances, that is $\left(z^{\left(i\right)}{)}^T\right(z^{\left(i\right)})$is a scalar, in fact it is the matrix$(z^{(i)})({\mathbf{z}}^{(i)}{)}^{}$The sum of the elements on the diagonal of${}^{T\; ,\; so\; that\; it\; can\; be\; related\; to\; the\; trace\; of\; the\; matrix.}$That is
$$({\mathbf{z}}^{\left(i\right)}{)}^{\mathrm{T}}({\mathbf{z}}^{\left(i\right)})=\mathrm{t}\mathrm{r}\left(({\mathbf{z}}^{\left(i\right)})(z^{\left(i\right)}{)}^{\mathrm{T}}\right)$$
那么
$$\begin{array}{rl}\sum _{i=1}^m({\mathbf{z}}^{(i)}{)}^{\mathrm{T}}({\mathbf{z}}^{\left(i\right)})& =\mathrm{t}\mathrm{r}\left({\mathbf{Z}\mathbf{Z}}^T\right)\\ & =tr\left({\mathbf{P}}^{T}X{XP}^{\mathrm{T}}\right)\end{array}$$
This seems to be a better understanding of the above derivation process $\left(4\right)$ The origin of the step.

Therefore, the formula for maximizing the objective is to maximize $tr\left(P^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}\right)$，即
$$\arg \underset{P}{\max }\mathrm{t}\mathrm{r}\left({\mathbf{P}}^{\mathrm{T}}{\mathbf{X}\mathbf{X}}^{\mathrm{T}}\mathbf{P}\right)\mathrm{s}.\mathrm{t}.{\mathbf{P}}^{\mathrm{T}}\mathbf{P}=It$$
can be seen that this expression is equivalent to the objective optimization formula in the previous derivation method.

References

[1] "Deep Learning" flower book
[2] related github
[3] Tongji University "Linear Algebra Sixth Edition"