This article from the scene of a downhill start, first put forward the basic idea of gradient descent algorithm, and then explain the principles of the gradient descent algorithm mathematically, the final realization of a simple example of gradient descent algorithm!
The scenario assumes that the gradient descent
The basic idea of gradient descent method is analogous to a downhill course. Assume that such a scenario: a man trapped in the mountains, you need to come down the mountain (ie find the lowest point of the mountain, which is the valley). But this time a big hill fog, resulting in very low visibility. Thus, down the path can not be determined, he must use the information around them to find the path down the mountain. This time, he can make use of gradient descent algorithm to help them down the mountain. Specifically, that is, to his current position as a reference in which to find the steepest place in this position, then drop towards the local height of the mountain go the same way, if our goal is the mountain, which is the climb to the top , then the time should be going up toward the steepest direction. Then every walk some distance, have repeatedly using the same method, and finally be able to successfully reach the valley.
image.png
We also can assume the steepest mountain where the naked eye can not be observed immediately out, but rather a sophisticated tool to measure by, at the same time, this person just at this time have the ability to measure the steepest direction. So, every person walking some distance, take some time to measure the direction of the steepest location, which is more time-consuming. So in order to reach the bottom of the hill before the sun goes down, it is necessary to reduce as much as possible the number of measurement direction. This is a dilemma of choice, if the measured frequently, you can ensure downhill direction is absolutely right, but very time-consuming, too little if measured, there is risk of deviation of the track. So it is necessary to find a suitable frequency measurement direction, down to ensure that the direction is not wrong, and, without consuming too much!
Gradient descent
The basic process of gradient descent down the mountain and on the scene very similar.
First of all, we have a differential function. This function represents the mountain. Our goal is to find the minimum of this function, which is the base of the mountain. The scenario assumes that before, the fastest way is to find the most down a steep current position and direction, and then go down along this direction corresponds to the function in a given point is to find a gradient , and then toward the opposite direction of the gradient, you can make the fastest decline in the value of the function! Because the direction of the gradient is a function of changes in the fastest direction (will explain later)
so we reuse this method, repeatedly strike a gradient, and finally able to reach a local minimum, which is similar to the process we are down. The strike will determine the steepest gradient direction, which is the scene in the direction of the means of measurement. Why then the direction of the gradient is steepest direction? Next, we start with the differential
differential
Viewing the Significance of differential, you can have a different point of view, the two most common are:
- Image function, the slope of the tangent at a point
-
The rate of change of a function
a few examples of differentiation:
image.png
The above examples are single variable differential, when a function of a plurality of variables, there is a multi-variable differential, i.e. separately for each variable differentiating
gradient
Multivariable generalized gradient is actually a derivative.
The following example:
We can see that the gradient is separately for each variable differential, then separated by a comma, the gradient is <> include them, in fact, a gradient vector instructions.
Calculus gradient is a very important concept, previously mentioned meaning gradient
- In univariate function, the gradient is actually a differential function, representing the slope of the tangent function at a given point of
- In multivariable function, the gradient is a vector, the vector has direction, the direction of the gradient pointed to the rise of fixed-point function in the fastest direction
这也就说明了为什么我们需要千方百计的求取梯度！我们需要到达山底，就需要在每一步观测到此时最陡峭的地方，梯度就恰巧告诉了我们这个方向。梯度的方向是函数在给定点上升最快的方向，那么梯度的反方向就是函数在给定点下降最快的方向，这正是我们所需要的。所以我们只要沿着梯度的方向一直走，就能走到局部的最低点！
梯度下降算法的数学解释
上面我们花了大量的篇幅介绍梯度下降算法的基本思想和场景假设，以及梯度的概念和思想。下面我们就开始从数学上解释梯度下降算法的计算过程和思想！
此公式的意义是：J是关于Θ的一个函数，我们当前所处的位置为Θ0点，要从这个点走到J的最小值点，也就是山底。首先我们先确定前进的方向，也就是梯度的反向，然后走一段距离的步长，也就是α，走完这个段步长，就到达了Θ1这个点！
下面就这个公式的几个常见的疑问：
- α是什么含义？
α在梯度下降算法中被称作为学习率或者步长，意味着我们可以通过α来控制每一步走的距离，以保证不要步子跨的太大扯着蛋，哈哈，其实就是不要走太快，错过了最低点。同时也要保证不要走的太慢，导致太阳下山了，还没有走到山下。所以α的选择在梯度下降法中往往是很重要的！α不能太大也不能太小，太小的话，可能导致迟迟走不到最低点，太大的话，会导致错过最低点！
- 为什么要梯度要乘以一个负号？
梯度前加一个负号，就意味着朝着梯度相反的方向前进！我们在前文提到，梯度的方向实际就是函数在此点上升最快的方向！而我们需要朝着下降最快的方向走，自然就是负的梯度的方向，所以此处需要加上负号 -
梯度下降算法的实例
我们已经基本了解了梯度下降算法的计算过程，那么我们就来看几个梯度下降算法的小实例，首先从单变量的函数开始
单变量函数的梯度下降
我们假设有一个单变量的函数
image.png
函数的微分
image.png
初始化，起点为
image.png
学习率为
image.png
根据梯度下降的计算公式
image.png
我们开始进行梯度下降的迭代计算过程：
image.png
如图，经过四次的运算，也就是走了四步，基本就抵达了函数的最低点，也就是山底
image.png多变量函数的梯度下降
我们假设有一个目标函数
image.png
现在要通过梯度下降法计算这个函数的最小值。我们通过观察就能发现最小值其实就是 (0，0)点。但是接下来，我们会从梯度下降算法开始一步步计算到这个最小值！
我们假设初始的起点为：
image.png
初始的学习率为：
image.png
函数的梯度为：
image.png
进行多次迭代：
image.png
我们发现，已经基本靠近函数的最小值点点image.png梯度下降算法的实现
下面我们将用python实现一个简单的梯度下降算法。场景是一个简单的线性回归的例子：假设现在我们有一系列的点，如下图所示image.png我们将用梯度下降法来拟合出这条直线！
首先，我们需要定义一个代价函数，在此我们选用均方误差代价函数image.png此公示中m是数据集中点的个数
½是一个常量，这样是为了在求梯度的时候，二次方乘下来就和这里的½抵消了，自然就没有多余的常数系数，方便后续的计算，同时对结果不会有影响
y 是数据集中每个点的真实y坐标的值h 是我们的预测函数，根据每一个输入x，根据Θ 计算得到预测的y值，即
image.png我们可以根据代价函数看到，代价函数中的变量有两个，所以是一个多变量的梯度下降问题，求解出代价函数的梯度，也就是分别对两个变量进行微分
image.png明确了代价函数和梯度，以及预测的函数形式。我们就可以开始编写代码了。但在这之前，需要说明一点，就是为了方便代码的编写，我们会将所有的公式都转换为矩阵的形式，python中计算矩阵是非常方便的，同时代码也会变得非常的简洁。
为了转换为矩阵的计算，我们观察到预测函数的形式
image.png我们有两个变量，为了对这个公式进行矩阵化，我们可以给每一个点x增加一维，这一维的值固定为1，这一维将会乘到Θ0上。这样就方便我们统一矩阵化的计算
image.png然后我们将代价函数和梯度转化为矩阵向量相乘的形式coding time
首先，我们需要定义数据集和学习率
import numpy as np # Size of the points dataset. m = 20 # Points x-coordinate and dummy value (x0, x1). X0 = np.ones((m, 1)) X1 = np.arange(1, m+1).reshape(m, 1) X = np.hstack((X0, X1)) # Points y-coordinate y = np.array([ 3, 4, 5, 5, 2, 4, 7, 8, 11, 8, 12, 11, 13, 13, 16, 17, 18, 17, 19, 21 ]).reshape(m, 1) # The Learning Rate alpha. alpha = 0.01接下来我们以矩阵向量的形式定义代价函数和代价函数的梯度
def error_function(theta, X, y): '''Error function J definition.''' diff = np.dot(X, theta) - y return (1./2*m) * np.dot(np.transpose(diff), diff) def gradient_function(theta, X, y): '''Gradient of the function J definition.''' diff = np.dot(X, theta) - y return (1./m) * np.dot(np.transpose(X), diff)最后就是算法的核心部分，梯度下降迭代计算
def gradient_descent(X, y, alpha): '''Perform gradient descent.''' theta = np.array([1, 1]).reshape(2, 1) gradient = gradient_function(theta, X, y) while not np.all(np.absolute(gradient) <= 1e-5): theta = theta - alpha * gradient gradient = gradient_function(theta, X, y) return theta当梯度小于1e-5时，说明已经进入了比较平滑的状态，类似于山谷的状态，这时候再继续迭代效果也不大了，所以这个时候可以退出循环！
完整的代码如下
import numpy as np # Size of the points dataset. m = 20 # Points x-coordinate and dummy value (x0, x1). X0 = np.ones((m, 1)) X1 = np.arange(1, m+1).reshape(m, 1) X = np.hstack((X0, X1)) # Points y-coordinate y = np.array([ 3, 4, 5, 5, 2, 4, 7, 8, 11, 8, 12, 11, 13, 13, 16, 17, 18, 17, 19, 21 ]).reshape(m, 1) # The Learning Rate alpha. alpha = 0.01 def error_function(theta, X, y): '''Error function J definition.''' diff = np.dot(X, theta) - y return (1./2*m) * np.dot(np.transpose(diff), diff) def gradient_function(theta, X, y): '''Gradient of the function J definition.''' diff = np.dot(X, theta) - y return (1./m) * np.dot(np.transpose(X), diff) def gradient_descent(X, y, alpha): '''Perform gradient descent.''' theta = np.array([1, 1]).reshape(2, 1) gradient = gradient_function(theta, X, y) while not np.all(np.absolute(gradient) <= 1e-5): theta = theta - alpha * gradient gradient = gradient_function(theta, X, y) return theta optimal = gradient_descent(X, y, alpha) print('optimal:', optimal) print('error function:', error_function(optimal, X, y)[0,0])运行代码，计算得到的结果如下
image.png所拟合出的直线如下
image.png小结
至此，我们就基本介绍完了梯度下降法的基本思想和算法流程，并且用python实现了一个简单的梯度下降算法拟合直线的案例！
最后，我们回到文章开头所提出的场景假设:
这个下山的人实际上就代表了反向传播算法，下山的路径其实就代表着算法中一直在寻找的参数Θ，山上当前点的最陡峭的方向实际上就是代价函数在这一点的梯度方向，场景中观测最陡峭方向所用的工具就是微分 。在下一次观测之前的时间就是有我们算法中的学习率α所定义的。
可以看到场景假设和梯度下降算法很好的完成了对应！
作者：六尺帐篷
链接：https://www.jianshu.com/p/c7e642877b0e
来源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。
