f
(
x
)
=
ω
T
∗
x
+
b
f(\boldsymbol{x}) = \boldsymbol{\omega}^T*\boldsymbol{x}+b
f ( x ) = ω T ∗ x + b
非线性模型可由线性模型通过引入层级结构和高维映射得到
具有很好的解释性(哪一个参数更为重要)
根据给定数据集，是给定一个线性模型尽可能准确的预测实值的输出标记
(
ω
∗
,
b
∗
)
=
argmin
(
ω
,
b
)
∑
i
=
1
m
(
y
i
−
ω
∗
x
i
−
b
)
(\boldsymbol{\omega}^*, b^*) = \underset{(\boldsymbol{\omega},b)}{\text{argmin}}\sum_{i=1}^{m}(y_i-\boldsymbol{\omega}*\boldsymbol{x_i}-b)
( ω ∗ , b ∗ ) = ( ω , b ) argmin i = 1 ∑ m ( y i − ω ∗ x i − b )
x
i
\boldsymbol{x_i}
x i
y
=
g
−
1
(
ω
T
x
+
b
)
y = g^{-1}(\boldsymbol{\omega}^T\boldsymbol{x}+b)
y = g − 1 ( ω T x + b )
g
(
)
g()
g ( )
g
(
)
g()
g ( )
寻找一个单调可谓的函数将分类任务的真实标记
y
y
y
f
(
x
)
=
1
1
+
e
−
(
ω
x
+
b
)
f(x) = \frac{1}{1+e^{-(\omega x + b)}}
f ( x ) = 1 + e − ( ω x + b ) 1
ω
\boldsymbol{\omega}
ω
b
b
b 对数几率函数可变化为
l
n
y
1
−
y
=
ω
T
x
+
b
ln\frac{y}{1-y} = \boldsymbol{\omega}^T\boldsymbol{x} + b
l n 1 − y y = ω T x + b
y
=
p
(
y
=
1
  
∣
  
x
)
y = p(y=1\;|\;\boldsymbol{x})
y = p ( y = 1 ∣ x )
1
−
y
=
p
(
y
=
0
  
∣
  
x
)
1-y = p(y=0\;|\;\boldsymbol{x})
1 − y = p ( y = 0 ∣ x )
p
(
y
=
1
  
∣
  
x
)
=
e
ω
T
x
+
b
1
+
e
ω
T
x
+
b
p
(
y
=
0
  
∣
  
x
)
=
1
1
+
e
ω
T
x
+
b
p(y=1\;|\;\boldsymbol{x}) = \frac{e^{\boldsymbol{\omega}^T\boldsymbol{x} + b}}{1+e^{\boldsymbol{\omega}^T\boldsymbol{x}+b}} p(y=0\;|\;\boldsymbol{x}) = \frac{1}{1+e^{\boldsymbol{\omega}^T\boldsymbol{x}+b}}
p ( y = 1 ∣ x ) = 1 + e ω T x + b e ω T x + b p ( y = 0 ∣ x ) = 1 + e ω T x + b 1
l
(
ω
,
b
)
=
∑
i
=
1
m
ln
p
(
y
i
  
∣
  
x
;
ω
,
b
)
l(\boldsymbol{\omega}, b) = \sum_{i=1}^{m}\ln p(y_i\;|\;\boldsymbol{x};\boldsymbol{\omega},b)
l ( ω , b ) = i = 1 ∑ m ln p ( y i ∣ x ; ω , b )
p
p
p
p
(
y
i
  
∣
  
x
i
;
ω
,
b
)
=
y
i
p
1
(
x
,
ω
,
b
)
+
(
1
−
y
i
)
p
0
(
x
,
ω
,
b
)
p(y_i\;|\;\boldsymbol{x}_i;\boldsymbol{\omega},b) = y_ip_1(\boldsymbol{x,\omega},b) + (1-y_i)p_0(\boldsymbol{x,\omega},b)
p ( y i ∣ x i ; ω , b ) = y i p 1 ( x , ω , b ) + ( 1 − y i ) p 0 ( x , ω , b )
p
1
p_1
p 1
p
2
p_2
p 2
p
1
=
1
−
p
0
p_1 = 1 - p_0
p 1 = 1 − p 0
p
(
y
i
  
∣
  
x
i
;
ω
,
b
)
=
y
i
  
p
(
x
,
ω
,
b
)
+
(
1
−
y
i
)
  
(
1
−
p
(
x
,
ω
,
b
)
p(y_i\;|\;\boldsymbol{x}_i;\boldsymbol{\omega},b) = y_i\;p(\boldsymbol{x,\omega},b) + (1-y_i)\;(1-p(\boldsymbol{x,\omega},b)
p ( y i ∣ x i ; ω , b ) = y i p ( x , ω , b ) + ( 1 − y i ) ( 1 − p ( x , ω , b )
给定训练样集，设法将样例投影到一条直线上，使得同类样例尽量接近，异样样例尽量远离
设投影直线的方向为
ω
\boldsymbol{\omega}
ω
X
i
,
μ
i
,
Σ
i
\boldsymbol{X_i}, \boldsymbol{\mu_i}, \boldsymbol{\Sigma_i}
X i , μ i , Σ i
i
i
i
ω
T
μ
i
\boldsymbol{\omega}^T\boldsymbol{\mu_i}
ω T μ i
μ
i
T
Σ
i
μ
i
\boldsymbol{\mu_i^T\Sigma_i\mu_i}
μ i T Σ i μ i
J
=
∣
∣
ω
T
μ
0
−
ω
T
μ
1
∣
∣
2
2
ω
T
Σ
0
ω
+
ω
T
Σ
1
ω
=
ω
T
(
μ
0
−
μ
1
)
(
μ
0
−
μ
1
)
T
ω
ω
T
(
Σ
0
+
Σ
1
)
ω
J = \frac{||\boldsymbol{\omega^T\mu_0-\omega^T\mu_1}||_2^2}{\boldsymbol{\omega^T\Sigma_0\omega + \omega^T\Sigma_1\omega}} = \frac{\boldsymbol{\omega^T(\mu_0-\mu_1)(\mu_0-\mu_1)^T\omega}}{\boldsymbol{\omega^T(\Sigma_0+\Sigma_1)\omega}}
J = ω T Σ 0 ω + ω T Σ 1 ω ∣ ∣ ω T μ 0 − ω T μ 1 ∣ ∣ 2 2 = ω T ( Σ 0 + Σ 1 ) ω ω T ( μ 0 − μ 1 ) ( μ 0 − μ 1 ) T ω
S
ω
=
Σ
0
+
Σ
1
\boldsymbol{S_\omega = \Sigma_0 + \Sigma_1}
S ω = Σ 0 + Σ 1
S
b
=
(
μ
0
−
μ
1
)
(
μ
0
−
μ
1
)
T
\boldsymbol{S_b = (\mu_0-\mu_1)(\mu_0-\mu_1)^T}
S b = ( μ 0 − μ 1 ) ( μ 0 − μ 1 ) T
S
ω
,
S
b
S_\omega, S_b
S ω , S b
设有
N
N
N
i
i
i
m
i
m_i
m i
S
t
=
∑
i
=
1
m
(
(
x
i
−
μ
)
(
x
i
−
μ
)
T
\boldsymbol{S_t} = \sum_{i=1}^{m}(\boldsymbol{(x_i-\mu)(x_i - \mu)^T}
S t = i = 1 ∑ m ( ( x i − μ ) ( x i − μ ) T
μ
\boldsymbol{\mu}
μ
S
ω
\boldsymbol{S_\omega}
S ω
S
ω
=
∑
i
=
1
m
S
ω
i
\boldsymbol{S_\omega} = \sum_{i=1}^{m} \boldsymbol{S_{\omega_i}}
S ω = ∑ i = 1 m S ω i
S
ω
i
=
∑
j
=
1
m
i
(
(
x
j
−
μ
i
)
(
x
j
−
μ
i
)
T
S_{\omega_i} = \sum_{j=1}^{m_i}(\boldsymbol{(x_j-\mu_i)(x_j-\mu_i)^T}
S ω i = ∑ j = 1 m i ( ( x j − μ i ) ( x j − μ i ) T
S
b
=
S
t
−
S
ω
=
∑
i
=
1
N
m
i
(
μ
i
−
μ
)
(
μ
i
−
μ
)
T
\boldsymbol{S_b = S_t - S_\omega} = \sum_{i=1}^{N}m_i\boldsymbol{(\mu_i -\mu)(\mu_i - \mu)^T}
S b = S t − S ω = i = 1 ∑ N m i ( μ i − μ ) ( μ i − μ ) T
S
ω
,
S
t
,
S
b
\boldsymbol{S_\omega, S_t, S_b}
S ω , S t , S b
W
\boldsymbol{W}
W
d
′
d'
d ′
d
′
≪
d
d'\ll d
d ′ ≪ d
多分类问题的基本思路是拆解法，将多分类任务柴蔚若干个二分类任务一对一(OvO) : 将N个类别两两配对，形成
N
(
N
−
1
)
/
2
N(N-1)/2
N ( N − 1 ) / 2 一对多(OvR) : 将一个类别视为正，其余所有均视为负。若只有一个类别为真，那么该类为预测结果；若多个类别都判断为真，根据每个判断结果的置信度预测最终结果。多对多(MvM) :
N
N
N
M
M
M
M
M
M
