【机器学习】1 线性回归

1 Linear Regression with One Variable / Univariate Linear Regression

1.1 model

• Hypothesis：$$h_{\theta }(x)={\theta }_0+{\theta }_{1}x$$
• Parameters：$${\theta }_0,{\theta }_1$$
• Cost Function：square error function / square error cost function$$J({\theta }_0,{\theta }_1)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }{(x^{(i)})-y^{(i)})}^2$$
• Goal（Object Function）：$$\underset{{\theta }_0,{\theta }_1}{\mathrm{minimize}}J({\theta }_0,{\theta }_1)$$

1.2 ‘Batch’ Gradient Descent Algorithm

to solve $$\underset{{\theta }_0,{\theta }_1}{\mathrm{minimize}}J({\theta }_0,{\theta }_1)$$

1.2.1 algorithm

• repeat until convergence{
  $${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1)\text{（for j=0 and j=1）}$$
          $\alpha$ —— learning rate
  }
• Correct:$$\begin{array}{rl}{temp}_0& :={\theta }_0-\alpha \frac{\partial }{\partial {\theta }_0}J({\theta }_0,{\theta }_1)\\ {temp}_1& :={\theta }_1-\alpha \frac{\partial }{\partial {\theta }_1}J({\theta }_0,{\theta }_1)\\ {\theta }_0& :={temp}_0\\ {\theta }_1& :={temp}_1\end{array}$$
• Notice: need to simultaneously update ${\theta }_0$ and ${\theta }_1$

1.2.2 use for univariate linear regression

• repeat{
    $$\begin{array}{rl}{\theta }_0& :={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\\ {\theta }_1& :={\theta }_1-\alpha \frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})\cdot x^{(i)})\end{array}$$
  }

1.2.3 features

• 在梯度下降的每一步中，我们都用到了所有的训练样本

2 Linear Regression with Multiple Variables / Multivariate Linear Regression

2.1 model

• Hypothesis：$x_0=1$$$h_{\theta }(x)={\theta }^{T}x={\theta }_0{x}_0+{\theta }_1{x}_1+\cdot \cdot \cdot +{\theta }_n{x}_n$$
• Parameters：$n+1$-demensional vector$$\theta ={\theta }_0,{\theta }_1,\cdot \cdot \cdot ,{\theta }_n$$
• Cost Function：square error function / square error cost function$$J(\theta )=J({\theta }_0,{\theta }_1,\cdot \cdot \cdot ,{\theta }_n)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }{(x^{(i)})-y^{(i)})}^2$$
• Goal（Object Function）：$$\underset{{\theta }_0,{\theta }_1,\cdot \cdot \cdot ,{\theta }_n}{\mathrm{minimize}}J({\theta }_0,{\theta }_1,{\theta }_n)$$

2.2 Gradient Descent Algorithm

2.2.1 algorithm

• repeat until convergence{
  $${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1,\cdot \cdot \cdot ,{\theta }_n)\text{（simultaneously update θj for j=0,1,⋅⋅⋅,n）}$$
  }

2.2.2 use for multiple linear regression

• repeat{
  $x_0^{(i)}=1$
    $$\begin{array}{rl}{\theta }_0& :={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_0^{(i)})\\ {\theta }_1& :={\theta }_1-\alpha \frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})\cdot x_1^{(i)})\\ \cdot \cdot \cdot & \\ {\theta }_n& :={\theta }_n-\alpha \frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})\cdot x_n^{(i)})\end{array}$$
  }

3 Gradient Descent in Practice

3.1 Feature Scaling（特征缩放）

• Goal: make sure features are on a similar scale
• Advantages:
  (1) make gradient desecent run much faster（多快）
  (2) converge in a lot fewer iterations（少收敛）
• Methods:
  (1) dividing by the maximum value
  (2) mean normalization（均值归一化）

3.1.1 mean normaliztion

• Theory: replace $x_i$ with $x_i-{\mu }_i$ to make features have approximately zero mean
  Do not apply to $x_0=1$
  $$x_i=\frac{x_i-{\mu }_i}{s_i}$$
• ${\mu }_i$ —— average value of $x_i$ in the training sets
• $s_i$ —— the range of values of that feature
  (1) $\text{maximum value}-\text{minimum value}$
  (2) $\text{standard deviation of the variable}$（标准差）

3.2 Learning Rate

• Debugging: How to make sure gradient descent in working correctly

3.2.1 good method: plot

• plot the cost function as we increase the number of iterations（画出代价函数值$J(\theta )$随迭代次数变化的曲线）
• Advantages:
  (1) can show gradient descent if working correctly —— $J(\theta )$ should decrease after every iteration（梯度下降法是否正常运行）
  (2) judge whether or not gradient descent has converged（梯度下降是否收敛）

3.2.1 other method

• declare convergence if $J(\theta )$ decreases by less than $\epsilon$ in one iteration
• Advantage: judge if converging automatically
• Disadvantage: choose $\epsilon$ can be difficult

3.2.3 choose $\alpha$

• consider: $\alpha =0.01,0.03,0.1,0.3,1,3,10,\cdot \cdot \cdot$

3.2.4 plot problem

• problem: $J(\theta )$ does not decrease after every iteration
• cause: $\alpha$ is too large that $J(\theta )$ may not decrease on every iteration; may not converge and slow converge is also possible
• Solve: choose sufficiently smaller $\alpha$
• Problem caused by solution above: $\alpha$ is too small that gradient descent can be slow to converge

3.3 Features and Polynomial Regression

• linear regression does not adapt to all datas
• polynomial regression can translate into linear regression
• we need to observe training sets so that we can choose a sufficient model
• Notice: feature scaling is necessary if choosing polynomial regresssion

4 Normal Equation

a method to solve for $\theta$ analytically one step to get to the optimal value right

4.1 model

• $m$ examples $(x^{(1)},y^{(1)}),\cdot \cdot \cdot ,(x^{(m)},y^{(m)})$ ; $n$ features
• $n+1$ dimensional vector: $$x^{(i)}=\left[\begin{array}{c}{x}_0^{(i)}\\ \cdot \cdot \cdot \\ x_n^{(i)}\end{array}\right]$$
• design matrix ($m\cdot (n+1)$ dimensional vector): $$X=\left[\begin{array}{c}{(x^{(1)})}^T\\ \cdot \cdot \cdot \\ {(x^{(m)})}^T\end{array}\right]$$
• $$y=\left[\begin{array}{c}{y}^{(1)}\\ \cdot \cdot \cdot \\ y^{(m)}\end{array}\right]$$
• $$\theta ={(X^{T}X)}^{-1}{X}^{T}y$$

4.2 writing

• Octave

 pinv(X'*X)*X'*y

• Python

import numpy as np
def notmalEqn(X,y):
	theta = np.linalg.inv(X.T@X)@X.T@y	# X.T@X 等价于 X.T.dot(X)
	return theta

problem: if $X^{T}X$ is non-invertible ( singular or degenerate matrices)?

• cause:
  (1) redundant features (liearly dependent)
  (2) too many features —— delete some features or use regularation
• 出现不可逆矩阵的情况极少发生
• 伪逆pesudo-inverse：pinv()
• 逆：inv()

5 梯度下降与正规方程的比较

Gradient Descent	Normal Euqation
need to choose $\alpha$	no need to choose $\alpha$
need many iterations	no need to iterate
works well even when $n$ is large	slow if $n$ is very large because we need to compute $(X^{T}X{)}^{-1}$【$n$ is hard to get】
适用于各种类型的模型	只适用于 linear regression，不适合 logistic regression algorithm

6 参考

吴恩达 机器学习 coursera machine learning
黄海广 机器学习笔记