上面 PCA 从数学方面说明了数据矩阵压缩的可能性，现在从物理模型上说明这点。
数据矩阵由于测量噪声，观测到的数据矩阵理论上为
A
=
A
ˉ
+
N
A = \bar A + N
A = A ˉ + N
A
ˉ
\bar A
A ˉ
N
N
N
N
N
N
r
a
n
k
N
=
m
i
n
(
m
,
n
)
rank N = min(m,n)
r a n k N = m i n ( m , n )
A
A
A
A
ˉ
\bar A
A ˉ
r
=
r
a
n
k
A
ˉ
≪
m
i
n
(
m
,
n
)
r = rank \bar A \ll min(m,n)
r = r a n k A ˉ ≪ m i n ( m , n )
对真实矩阵进行奇异值分解得
A
ˉ
=
σ
1
u
1
v
1
T
+
⋯
+
σ
r
u
r
v
r
T
\bar A = \sigma_1\mathbf{u}_1\mathbf{v}^T_1+\cdots+\sigma_r\mathbf{u}_r\mathbf{v}^T_r
A ˉ = σ 1 u 1 v 1 T + ⋯ + σ r u r v r T
u
i
\mathbf{u}_i
u i
r
≪
m
i
n
(
m
,
n
)
r \ll min(m,n)
r ≪ m i n ( m , n )
我们对存储矩阵
A
ˉ
\bar A
A ˉ
A
ˉ
\bar A
A ˉ
m
n
mn
m n
r
r
r
σ
1
,
⋯
,
σ
r
\sigma_1,\cdots,\sigma_r
σ 1 , ⋯ , σ r
r
r
r
r
r
r
v
1
,
⋯
,
v
r
\mathbf{v}_1,\cdots,\mathbf{v}_r
v 1 , ⋯ , v r
r
n
rn
r n
r
r
r
u
1
,
⋯
,
u
r
\mathbf{u}_1,\cdots,\mathbf{u}_r
u 1 , ⋯ , u r
r
m
rm
r m
r
+
r
n
+
r
m
=
r
(
m
+
n
+
1
)
r+rn+rm=r(m+n+1)
r + r n + r m = r ( m + n + 1 )
r
≪
m
i
n
(
m
,
n
)
r \ll min(m,n)
r ≪ m i n ( m , n )
r
(
m
+
n
+
1
)
≪
m
n
r(m+n+1) \ll mn
r ( m + n + 1 ) ≪ m n
A
ˉ
=
σ
1
u
1
v
1
T
+
⋯
+
σ
r
u
r
v
r
T
\bar A = \sigma_1\mathbf{u}_1\mathbf{v}^T_1+\cdots+\sigma_r\mathbf{u}_r\mathbf{v}^T_r
A ˉ = σ 1 u 1 v 1 T + ⋯ + σ r u r v r T
但由于矩阵
A
A
A
r
=
r
a
n
k
A
≈
m
i
n
(
m
,
n
)
r = rank A \thickapprox min(m,n)
r = r a n k A ≈ m i n ( m , n )
r
(
m
+
n
+
1
)
≈
m
i
n
(
m
,
n
)
(
m
+
n
+
1
)
≈
2
m
n
r(m+n+1) \thickapprox min(m,n)(m+n+1) \thickapprox 2mn
r ( m + n + 1 ) ≈ m i n ( m , n ) ( m + n + 1 ) ≈ 2 m n
由于噪声随机不相关，所以噪声引起的奇异值几乎都相等即
σ
1
N
≈
σ
2
N
≈
⋯
≈
σ
m
i
n
(
m
,
n
)
N
\sigma^N_1 \thickapprox \sigma^N_2 \thickapprox \cdots \thickapprox \sigma^N_{min(m,n)}
σ 1 N ≈ σ 2 N ≈ ⋯ ≈ σ m i n ( m , n ) N
σ
1
N
≪
σ
r
A
ˉ
\sigma^N_1 \ll \sigma^{\bar A}_r
σ 1 N ≪ σ r A ˉ 所以如果我们对观测矩阵
A
A
A
r
=
r
a
n
k
A
ˉ
r = rank \bar A
r = r a n k A ˉ
A
A
A 。注意矩阵
A
A
A
A
ˉ
\bar A
A ˉ
N
N
N
A
A
A
r
a
n
k
A
ˉ
rank \bar A
r a n k A ˉ
实际情况下，由于无法得到真实矩阵
A
ˉ
\bar A
A ˉ
r
a
n
k
A
ˉ
rank \bar A
r a n k A ˉ
r
=
r
a
n
k
A
ˉ
r = rank \bar A
r = r a n k A ˉ 故实际中，只能近似取矩阵
A
A
A
R
R
R
σ
1
2
+
⋯
+
σ
R
2
\sigma^2_1+\cdots+\sigma^2_R
σ 1 2 + ⋯ + σ R 2
99
%
99\%
9 9 %
99.9
%
99.9\%
9 9 . 9 %
σ
1
2
+
⋯
+
σ
n
2
,
n
=
r
a
n
k
A
\sigma^2_1+\cdots+\sigma^2_n,n=rank A
σ 1 2 + ⋯ + σ n 2 , n = r a n k A
数据压缩本质上就是PCA，根据PCA重构公式
A
C
=
U
k
U
k
T
A
A^C = U_kU^T_kA
A C = U k U k T A
σ
1
u
1
v
1
T
+
⋯
+
σ
k
u
k
v
k
T
\sigma_1\mathbf{u}_1\mathbf{v}^T_1+\cdots+\sigma_k\mathbf{u}_k\mathbf{v}^T_k
σ 1 u 1 v 1 T + ⋯ + σ k u k v k T
A
=
U
Σ
V
T
A=U\Sigma V^T
A = U Σ V T
A
C
=
U
k
U
k
T
U
Σ
V
T
=
U
k
U
k
T
[
U
k
,
U
′
]
Σ
V
T
=
U
k
[
U
k
T
U
k
,
U
k
T
U
′
]
Σ
V
T
=
U
k
[
E
k
,
O
]
Σ
V
T
=
[
U
k
E
k
Σ
V
T
,
O
]
=
[
U
k
Σ
k
V
k
T
,
O
]
=
σ
1
u
1
v
1
T
+
⋯
+
σ
k
u
k
v
k
T
A^C = U_kU^T_kU\Sigma V^T = U_kU^T_k[U_k,U']\Sigma V^T \\ = U_k[U^T_kU_k,U^T_kU']\Sigma V^T=U_k[E_k,\mathbf{O}]\Sigma V^T\\ =[U_kE_k\Sigma V^T,\mathbf{O}]=[U_k\Sigma_k V_k^T,\mathbf{O}]\\ =\sigma_1\mathbf{u}_1\mathbf{v}^T_1+\cdots+\sigma_k\mathbf{u}_k\mathbf{v}^T_k
A C = U k U k T U Σ V T = U k U k T [ U k , U ′ ] Σ V T = U k [ U k T U k , U k T U ′ ] Σ V T = U k [ E k , O ] Σ V T = [ U k E k Σ V T , O ] = [ U k Σ k V k T , O ] = σ 1 u 1 v 1 T + ⋯ + σ k u k v k T