机器学习(二)线性模型---LR

机器学习(二)线性模型—LR

2.2 LR
2.2.1 基础
LR是一种二分类模型，属于线性模型的一种，是广义线性分类模型，采用极大似然估计，具有概率可解释性
条件概率：

\begin{matrix} (14) & P (y = 1 | x) = σ (w \cdot x) \end{matrix}

$P(y=1|x)=\sigma(w\cdot x)\tag{14}$

\begin{matrix} (15) & P (y = 0 | x) = 1 - σ (w \cdot x) \end{matrix}

$P(y=0|x)=1-\sigma(w\cdot x)\tag{15}$
Logits/log odds，一件事发生与不发生的比率：

\begin{matrix} (16) & l o g \frac{p}{1 - p} \end{matrix}

$log\frac{p}{1-p}\tag{16}$
2.2.2 推导：
似然函数：

\begin{matrix} (17) & \prod_{i = 1}^{N} σ (w \cdot x_{i})^{y_{i}} (1 - σ (w \cdot x_{i}))^{1 - y_{i}} \end{matrix}

$\prod_{i=1}^N\sigma(w\cdot x_i)^{y_i}(1-\sigma(w\cdot x_i))^{1-y_i}\tag{17}$
对数似然函数：

\begin{matrix} (18) & \sum_{i = 1}^{N} y_{i} l o g σ (w \cdot x_{i}) + (1 - y_{i}) l o g (1 - σ (w \cdot x_{i})) \end{matrix}

$\sum_{i=1}^Ny_ilog\sigma(w\cdot x_i)+(1-y_i)log(1-\sigma(w\cdot x_i))\tag{18}$
首先令

w \cdot x_{i} + b = a

$w\cdot x_i+b=a$

\begin{matrix} (19) & d l o s s = \sum_{i = 1}^{N} y_{i} \frac{σ' (a)}{σ (a)} d a - (1 - y_{i}) \frac{σ' (a)}{1 - σ (a)} d a \end{matrix}

$dloss=\sum_{i=1}^Ny_i\frac{\sigma\prime(a)}{\sigma(a)}da-(1-y_i)\frac{\sigma\prime(a)}{1-\sigma(a)}da\tag{19}$

\begin{matrix} (20) & d a = d w^{T} x_{i} = t r (x_{i}^{T} d w) \end{matrix}

$da=dw^Tx_i=tr(x_i^Tdw)\tag{20}$
将

(20)

$(20)$ 与

σ' (a) = σ (a) (1 - σ (a))

$\sigma\prime(a)=\sigma(a)(1-\sigma(a))$ 带入

(19)

$(19)$

\begin{aligned} d l o s s & = t r (\sum_{i = 1}^{N} y_{i} (1 - σ (a)) x_{i}^{T} d w - (1 - y_{i}) σ (a) x_{i}^{T} d w) \\ (21) & = t r (\sum_{i = 1}^{N} (y_{i} - σ (a)) x_{i}^{T} d w) \end{aligned}

$\begin{align} dloss&=tr(\sum_{i=1}^{N}y_i(1-\sigma(a))x_i^Tdw-(1-y_i)\sigma(a)x_i^Tdw)\notag\\ &=tr(\sum_{i=1}^N(y_i-\sigma(a))x_i^Tdw)\tag{21} \end{align}$
故：

\begin{matrix} (22) & \frac{\partial l o s s}{\partial w} = \sum_{i = 1}^{N} x_{i} (y_{i} - σ (a)) = \sum_{i = 1}^{N} (y_{i} - σ (a)) x_{i} \end{matrix}

$\frac{\partial loss}{\partial w}=\sum_{i=1}^Nx_i(y_i-\sigma(a))=\sum_{i=1}^N(y_i-\sigma(a))x_i\tag{22}$

2.2.3 softmax多分类推导：
损失函数：

\begin{matrix} (23) & l o s s = - \sum_{i = 1}^{N} {y_{i}}^{T} l o g s o f t m a x (W x_{i}) \end{matrix}

$loss=-\sum_{i=1}^N\mathbf{y_i}^Tlogsoftmax(\mathbf{Wx_i})\tag{23}$
其中：

y_{i}

$y_i$ 为

k \times 1

$k\times 1$ 向量，只有一个位置为1表示类别

W

$W$ 为

k \times d

$k\times d$ 权矩阵，d为特征的维度

s o f t m a x (a)

$softmax(a)$ 为逐元素函数
令：

\begin{matrix} (24) & a_{i} = W x_{i} \end{matrix}

$a_i=\textbf{W}x_i\tag{24}$
将

(24) 带 入 (23)

$(24)带入(23)$ :

\begin{aligned} d l o s s & = - \sum_{i = 1}^{N} y_{i}^{T} d l o g s o f t m a x (a_{i}) \\ = - \sum_{i = 1}^{N} y_{i}^{T} d l o g (\frac{e^{a_{i}}}{1^{T} \cdot e^{a_{i}}}) \\ = - \sum_{i = 1}^{N} y_{i}^{T} (d a_{i} - 1 d l o g (1^{T} e^{a_{i}})) \\ = - \sum_{i = 1}^{N} (y_{i}^{T} d a_{i} - d l o g (1^{T} e^{a_{i}})) \\ = - \sum_{i = 1}^{N} (y_{i}^{T} d a_{i} - \frac{1^{T}}{1^{T} e^{a_{i}}} d e^{a_{i}}) \\ = - \sum_{i = 1}^{N} (y_{i}^{T} d a_{i} - \frac{1^{T} (e^{a_{i}} ⊙ d a_{i})}{1^{T} e^{a_{i}}}) \\ = - \sum_{i = 1}^{N} (y_{i}^{T} d a_{i} - \frac{(e^{a_{i}})^{T} d a_{i}}{1^{T} e^{a_{i}}}) \\ = t r (\sum_{i = 1}^{N} \frac{(e^{a_{i}})^{T} d a_{i}}{1^{T} e^{a_{i}}} - y_{i}^{T} d a_{i}) \\ = t r (\sum_{i = 1}^{N} s o f t m a x (a_{i})^{T} d a_{i} - y_{i}^{T} d a_{i}) \\ = t r (\sum_{i = 1}^{N} (s o f t m a x (a_{i})^{T} - y_{i}^{T}) d W x_{i}) \\ (25) & = t r (\sum_{i = 1}^{N} x_{i} (s o f t m a x (a_{i}) - y_{i}) d W) \end{aligned}

$\begin{align} dloss&=-\sum_{i=1}^Ny_i^Tdlogsoftmax(a_i)\notag\\ &=-\sum_{i=1}^Ny_i^Tdlog(\frac{e^{a_i}}{1^T\cdot e^{a_i}})\notag\\ &=-\sum_{i=1}^Ny_i^T(da_i-1dlog(1^Te^{a_i}))\notag\\ &=-\sum_{i=1}^N(y_i^Tda_i-dlog(1^Te^{a_i}))\notag\\ &=-\sum_{i=1}^N(y_i^Tda_i-\frac{1^T}{1^Te^{a_i}}de^{a_i})\notag\\ &=-\sum_{i=1}^N(y_i^Tda_i-\frac{1^T(e^{a_i}\odot da_i)}{1^Te^{a_i}})\notag\\ &=-\sum_{i=1}^N(y_i^Tda_i-\frac{(e^{a_i})^T da_i}{1^Te^{a_i}})\notag\\ &=tr(\sum_{i=1}^N\frac{(e^{a_i})^T da_i}{1^Te^{a_i}}-y_i^Tda_i)\notag\\ &=tr(\sum_{i=1}^Nsoftmax(a_i)^T\ da_i-y_i^Tda_i)\notag\\ &=tr(\sum_{i=1}^N(softmax(a_i)^T\ -y_i^T)dWx_i)\notag\\ &=tr(\sum_{i=1}^Nx_i(softmax(a_i)\ -y_i)dW)\tag{25}\\ \end{align}$
固:

\begin{matrix} (26) & \frac{\partial l o s s}{\partial W} = \sum_{i = 1}^{N} (s o f t m a x (W x_{i}) - y_{i}) x_{i}^{T} \end{matrix}

$\frac {\partial loss}{\partial W}=\sum_{i=1}^N(softmax(Wx_i)-y_i)x_i^T\tag{26}$

2.2.4 相关问题总结

Sigmoid变化的理解？
a) sigmoid函数光滑，处处可导，导数还能用自己表示
b) sigmoid能把数据从负无穷到正无穷压缩到0，1之间，压缩掉了长尾，扩展了核心分辨率。
c) sigmoid在有观测误差的情况下最优的保证了输入信号的信息。
LR有什么特点？
简单容易欠拟合对几率w*x+b 线性的实际问题不能完全用线性来拟合
各feature之间不需要满足条件独立假设，但各个feature的贡献独立计算
如何用LR建立一个广告点击的模型？
特征提取—>特征处理（离散化、归一化、onehot特征交叉、特征选择等）—>找出候选集—->模型训练，得到结果

机器学习(二)线性模型---LR

机器学习(二)线性模型—LR

猜你喜欢