逻辑回归学习笔记

逻辑回归学习笔记

本文仅为了个人学习理解使用

逻辑回归

逻辑回归是用来解决二分类问题
回归模型的输出是连续的；分类模型的输出是离散的

• 逻辑回归=线性回归+sigmoid函数
• 线性回归：$z=w\ast x+b$
• sigmoid函数：$$y=\frac{1}{1+e^{-z}}=\frac{1}{1+e^{-(w\ast x+b)}}$$
• 逻辑回归损失函数（损失函数越小模型越好，训练过程即使得损失函数最小的优化过程）：
  $$C=-[y\ln a+(1-y)\ln (1-a)]$$

损失函数

$$\text{ cost }=\{\begin{array}{ccc}-\log (\hat{p})& \text{ if }& y=1\\ -\log (1-\hat{p})& \text{ if }& y=0\end{array}$$
单个样本损失函数则表示为：
$$\text{ cost }=-y\log (\hat{p})-(1-y)\log (1-\hat{p})$$
所有样本损失函数（求和）：
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log \left({\hat{p}}^{(i)}\right)+\left(1-y^{(i)}\right)\log \left(1-{\hat{p}}^{(i)}\right)$$
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log \left(\sigma \left(X_b^{(i)}\theta \right)\right)+\left(1-y^{(i)}\right)\log \left(1-\sigma \left(X_b^{(i)}\theta \right)\right)$$
上式无法做数学解析解，但是该解析函数为凸函数（没有全局最优解，只有局部最优解），可以用梯度下降法求解。

梯度下降法

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log \left(\sigma \left(X_b^{(i)}\theta \right)\right)+\left(1-y^{(i)}\right)\log \left(1-\sigma \left(X_b^{(i)}\theta \right)\right)$$
$$\nabla J(\theta )=\left(\begin{array}{c}\frac{\partial J(\theta )}{\partial {\theta }_{\partial }}\\ \frac{\partial J(\theta )}{\partial {\theta }_1}\\ \cdots \\ \frac{\partial J(\theta )}{\partial {\theta }_n}\end{array}\right)$$
先看sigmoid函数求导
$$\sigma (t)=\frac{1}{1+e^{-t}}={\left(1+e^{-t}\right)}^{-1}$$
$$\sigma (t{)}^{\prime }=-{\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}\cdot (-1)={\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}\cdot$$
再扩一层
$$\begin{array}{rl}(\log \sigma (t){)}^{\prime }& =\frac{1}{\sigma (t)}\cdot \sigma (t{)}^{\prime }=\frac{1}{\sigma (t)}\cdot {\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}\\ & =\frac{1}{{\left(1+e^{-t}\right)}^{-1}}\cdot {\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}={\left(1+e^{-t}\right)}^{-1}\cdot e^{-t}\end{array}$$
$$\begin{array}{rl}(\log \sigma (t){)}^{\prime }& ={\left(1+e^{-t}\right)}^{-1}\cdot e^{-t}\\ & =\frac{e^{-t}}{1+e^{-t}}=\frac{1+e^{-t}-1}{1+e^{-t}}=1-\frac{1}{1+e^{-t}}\\ & =1-\sigma (t)\end{array}$$
$$\frac{d\left(y^{(i)}\log \sigma \left(X_b^{(i)}\theta \right)\right)}{d{\theta }_j}=y^{(i)}\left(1-\sigma \left(X_b^{(i)}\theta \right)\right)\cdot X_j^{(i)}$$
$$\begin{array}{rl}(\log (1-\sigma (t)){)}^{\prime }=& \frac{1}{1-,\sigma (t)}\cdot (-1)\cdot \sigma (t{)}^{\prime }=-\frac{1}{1-\sigma (t)}\cdot {\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}\\ & =-\frac{1+e^{-t}}{e^{-t}}\cdot {\left(1+e^{-t}\right)}^{-2}\cdot e^{-t}\\ & =-{\left(1+e^{-t}\right)}^{-1}=-\sigma (t)\end{array}$$
$$\frac{d\left(\left(1-y^{(i)}\right)\log \left(1-\sigma \left(X_b^{(i)}\theta \right)\right)\right)}{d{\theta }_j}=\left(1-y^{(i)}\right)\cdot \left(-\sigma \left(X_b^{(i)}\theta \right)\right)\cdot X_j^{(i)}$$
$$\begin{array}{rl}\frac{J(\theta )}{{\theta }_j}=& \frac{1}{m}\sum _{i=1}^m\left(\sigma \left(X_b^{(i)}\theta \right)-y^{(i)}\right)X_j^{(i)}\\ & =\frac{1}{m}\sum _{i=1}^m\left({\hat{y}}^{(i)}-y^{(i)}\right)X_j^{(i)}\end{array}$$
其中${\hat{y}}^{(i)}$就是那个预测值
$$\nabla J(\theta )=\left(\begin{array}{c}\partial J/\partial {\theta }_0\\ \partial J/\partial {\theta }_1\\ \partial J/\partial {\theta }_2\\ \dots \\ \partial J/\partial {\theta }_n\end{array}\right)=\frac{1}{m}\cdot \left(\begin{array}{c}\sum _{i=1}^m\left(\sigma \left(X_b^{(i)}\theta \right)-y^{(i)}\right)\\ \sum _{i=1}^m\left(\sigma \left(X_b^{(i)}\theta \right)-y^{(i)}\right)\cdot X_1^{(i)}\\ \sum _{i=1}^m\left(\sigma \left(X_b^{(i)}\theta \right)-y^{(i)}\right)\cdot X_2^{(i)}\\ \dots \\ \sum _{i=1}^m\left(\sigma \left(X_b^{(i)}\theta \right)-y^{(i)}\right)\cdot X_n^{(i)}\end{array}\right)=\frac{1}{m}\cdot \left(\begin{array}{c}\sum _{i=1}^m\left({\hat{y}}^{(i)}-y^{(i)}\right)\\ \sum _{i=1}^m\left({\hat{y}}^{(i)}-y^{(i)}\right)\cdot X_1^{(i)}\\ \sum _{i=1}^m\left({\hat{y}}^{(i)}-y^{(i)}\right)\cdot X_2^{(i)}\\ \dots \\ \sum _{i=1}^m\left({\hat{y}}^{(i)}-y^{(i)}\right)\cdot X_n^{(i)}\end{array}\right)$$
对上面式子向量化（没看懂，不会推）
$$\nabla J(\theta )=\frac{1}{m}\cdot X_b^T\cdot \left(\sigma \left(X_b\theta \right)-y\right)$$

梯度下降法

最优化算法，适用于有唯一极值点的函数。对于没有唯一极值点的函数，可以多次运行，随机化初始点。
学习率$\eta$取值会影响最优解的速度，若取值不合适则得不到最优解。$\eta$为梯度下降法的一个超参数

目标：使得$J(\theta )=\mathrm{MSE}(y,\hat{y})$尽可能小
$$\frac{1}{m}\sum _{i=1}^m{\left(y^{(i)}-{\hat{y}}^{(i)}\right)}^2=\frac{1}{m}\sum _{i=1}^m{\left(y^{(i)}-{\theta }_0-{\theta }_1{X}_1^{(i)}-{\theta }_2{X}_{2k}^{(i)}-\dots -{\theta }_n{X}_n^{(i)}\right)}^2$$
$$\nabla J(\theta )=\left(\begin{array}{c}\partial J/\partial {\theta }_0\\ \partial J/\partial {\theta }_1\\ \partial J/\partial {\theta }_2\\ \dots \\ \partial J/\partial {\theta }_n\end{array}\right)=\left(\begin{array}{c}{\sum }_{i=1}^{m}2\left(y^{(i)}-X_b^{(i)}\theta \right)\cdot (-1)\\ {\sum }_{i=1}^{m}2\left(y^{(i)}-X_b^{(i)}\theta \right)\cdot \left(-X_1^{(i)}\right)\\ {\sum }_{i=1}^{m}2\left(y^{(i)}-X_b^{(i)}\theta \right)\cdot \left(-X_2^{(i)}\right)\\ \dots \\ {\sum }_{i=1}^{m}2\left(y^{(i)}-X_b^{(i)}\theta \right)\cdot \left(-X_n^{(i)}\right)\end{array}\right)=\frac{2}{m}\cdot \left(\begin{array}{c}{\sum }_{i=1}^m\left(X_b^{(i)}\theta -y^{(i)}\right)\\ {\sum }_{i=1}^m\left(X_b^{(i)}\theta -y^{(i)}\right)\cdot X_1^{(i)}\\ {\sum }_{i=1}^m\left(X_b^{(i)}\theta -y^{(i)}\right)\cdot X_2^{(i)}\\ \dots \\ {\sum }_{i=1}^m\left(X_b^{(i)}\theta -y^{(i)}\right)\cdot X_n^{(i)}\end{array}\right)$$
${\theta }_i={\theta }_i-\eta \frac{\partial J\left({\theta }_0,{\theta }_1,\cdots ,{\theta }_n\right)}{\partial {\theta }_i}$

梯度下降法的小算例（含PYTHON 程序）
梯度下降法1
梯度下降法2

附录

##参考学习
Sigmoid函数解析
逻辑回归十分钟学会，通俗易懂（内含spark求解过程）【B站】