Newton's method
Newton's method to solve the problem function value when the variable value of zero, specifically, when the required solution f (θ) = 0, if f derivable, then by iteration formula.
When θ is the vector of Newton, can be expressed using the following equation:
Wherein H is called the Hessian matrix, H (-1) represents the inverse matrix of the Hessian matrix, in fact, the second derivative of the objective function of the parameters θ.
Advantage of Newton's method:
Newton's method in comparison gradient descent method convergence rate quickly, and since the inverse Hessian matrix is decreasing in the iteration, the step size plays diminishing effect.
Disadvantages:
the shortcomings of Newton's method is to calculate the inverse Hessian matrix is more difficult, time-consuming and computing resources. So with quasi-Newton method.
-
An example of online solving Newton's method:
Hessian matrix definition:
- Code implements:
Solution f = 100 * (x2-x1 ** 2) ** 2 + (1-x1) ** 2 Newton Method
import numpy as np
import matplotlib.pyplot as plt
#梯度的公式
def tidu(x):
return np.array([-400*x[0]*(x[1]-x[0]**2)-2*(1-x[0]),200*(x[1]-x[0]**2)])
#海森矩阵的公式
def hessian(x):
return np.array([[-400*(x[1]-3*x[0]**2)+2,-400*x[0]],[-400*x[0],200]])
#牛顿法
def newton(x):
print("初始点为 : ",x)
res=[]
res.append(x)
i = 1
imax = 1000
delta = 1
#迭代的条件是小于imax，或者是更新的距离小于一个很小的数值
while i<imax and delta>10**(-5):
p = -np.dot(np.linalg.inv(hessian(x)),tidu(x))
x_new = x + p
res.append(x_new)
delta = sum((x-x_new)**2) # 更新的距离
print("初始点为 : ",x_new)
i=i+1
x=x_new # 更新x
return np.array(res)
if __name__ =="__main__":
# 用牛顿法求解 f=100*(x2-x1**2)**2+(1-x1)**2
X1=np.arange(-1.5,1.5+0.05,0.05)
X2=np.arange(-3.5,2+0.05,0.05)
[x1,x2]=np.meshgrid(X1,X2)
f=100*(x2-x1**2)**2+(1-x1)**2; # 给定的函数
plt.contour(x1,x2,f,20) # 画出函数的20条轮廓线
x0 = np.array([-1.2,1])
res=newton(x0)
res_x=res[:,0]
res_y=res[:,1]
plt.plot(res_x,res_y)
plt.show()
Author: zhaozhengcoder
link: https: //www.jianshu.com/p/0f864a4c3b38
Source: Jane books
are copyrighted by the author. Commercial reprint please contact the author authorized, non-commercial reprint please indicate the source.
Since Newton's method is used to determine the next location based on the current position of the tangent, so Newton's method has been very aptly called the "tangent method." Search Path Newton method (two-dimensional case) as shown below:
