梯度下降求解

梯度下降求解过程

线性回归

机器学习按目标函数进行迭代，使目标函数结果接近最小值
梯度下降，目标函数：
$$J({\theta }_0,{\theta }_1)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i)$$

---

批量梯度下降：$$\frac{\partial J(\theta )}{\partial {\theta }_j}=-\frac{1}{m}\sum _{i=1}^m(y_i-h_{\theta }(x_i))x_{ij}$$

$${\theta }_j^{\prime }={\theta }_j+\frac{1}{m}\sum _{i=1}^m(y_i-h_{\theta }(x_i))x_{ij}$$
批量梯度下降使用所有样本，速度很慢，容易得到最优解

---

随机梯度下降：$${\theta }_j^{\prime }={\theta }_j+(y_i-h_{\theta }(x_i))x_{ij}$$
每次找一个样本，迭代速度快，但不一定每次都朝着收敛的方向

---

小批量梯度下降法：$${\theta }_j^{\prime }=\theta -\alpha \frac{1}{10}\sum _{k=i}^{i+9}(h_{\theta }(x_k)-y_k)x_{kj}$$
每次更新选择一小部分数据迭代，以上两种方式结合

---

逻辑回归

分类数据的回归分析Logistic regression
Sigmoid函数：$$g(z)=\frac{1}{1+e^{-z}}$$
自变量取值为任意实数，值域[0，1]
将实数域内的值，映射到了0-1区间，完成了由值到概率的转换

预测函数：$$h_{\theta }(x)=g({\theta }^{T}x)=\frac{1}{1+e^{-{\theta }^{T}x}}$$

其中函数：
$${\theta }_0+{\theta }_1{x}_1+,\cdots ,+{\theta }_n{x}_n=\sum _{i=1}^n{\theta }_i{x}_i={\theta }^{T}x$$

分类任务：$$\begin{array}{rlr}p(y=1|x;\theta )& =h_{\theta }(x)& \cdots ①\\ p(y=0|x;\theta )& =1-h_{\theta }(x)& \cdots ②\\ p(y|x;\theta )& =(h_{\theta }(x){)}^y(1-h_{\theta }(x){)}^{1-y}& \cdots ③\end{array}$$
二分类任务1，2 整合为3

似然函数：$$L(\theta )=\prod _{i=1}^{m}p(y_i|x_i;\theta )=\prod _{i=1}^m(h_{\theta }(x_i){)}^{y_i}(1-h_{\theta }(x_i){)}^{1-y_i}$$

对数似然： $$l(\theta )=logL(\theta )=\sum _{i=1}^m(y_{i}log{h}_{\theta }(x_i)+(1-y_i)log(1-h_{\theta }(x_i)))$$

引入函数：$$J(\theta )=-\frac{1}{m}l(\theta )$$求解$J(\theta )$的最小值

求解过程：
$$\begin{array}{rl}l(\theta )=logL(\theta )& =\sum _{i=1}^m(y_{i}log{h}_{\theta }(x_i)+log(1-h_{\theta }(x_i)))\\ \frac{\partial J(\theta )}{{\partial }_{{\theta }_j}}& =-\frac{1}{m}\sum _{i=1}^m(y_i\frac{1}{h_{\theta }(x_i)}\frac{\partial }{{\partial }_{{\theta }_j}}{h}_{\theta }(x_i)-(1-y_i)\frac{1}{1-h_{\theta }(x_i)}\frac{\partial }{{\partial }_{{\theta }_j}}{h}_{\theta }(x_i))\\ & =-\frac{1}{m}\sum _{i=1}^m(y_i\frac{1}{g({\theta }^T{x}_i)}-(1-y_i)\frac{1}{1-g({\theta }_i^x)})\frac{\partial }{{\partial }_{{\theta }_j}}g({\theta }^T{x}_i)\\ & =-\frac{1}{m}\sum _{i=1}^m(y_i\frac{1}{g({\theta }^T{x}_i)}-(1-y_i)\frac{1}{1-g({\theta }^T{x}_i)})g({\theta }^T{x}_i)(1-g({\theta }^T{x}_i))\frac{\partial }{\partial {\theta }_j}{\theta }^T{x}_i\\ & =-\frac{1}{m}\sum _{i=1}^m(y_i(1-g({\theta }^T{x}_i))-(1-y_i)g({\theta }^T{x}_i))x_i^j\\ & =-\frac{1}{m}\sum _{i=1}^m(y_i-g({\theta }^T{x}_i))x_i^j\\ & =\frac{1}{m}\sum _{i=1}m(h_{\theta }(x_i)-y_i)x_i^j\end{array}$$

逻辑回归参数更新：
$${\theta }_j={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i)x_i^j$$