Introduction to SGD

SGD (Stochastic Gradient Descent), translated as stochastic gradient descent, is a commonly used function optimization method in deep learning.

1.Citation

Introducing $Before\; SGD,$ let me first introduce an example. There are three people on the top of the mountain who are thinking about how to go down the mountain quickly. The boss, the second child and the third child respectively put forward three different opinions.

• The boss said: Starting from the top of the mountain, every time you walk a certain distance, you will look for all the nearby mountain roads and choose the steepest mountain road to continue moving forward. As the name suggests, the boss always chooses the steepest mountain road to walk.

• The second child said: Starting from the top of the mountain, every time you walk a certain distance, you will randomly search for some nearby mountain roads, and choose the steepest mountain road to continue moving forward. As the name suggests, the second child will randomly search for some mountain roads, and then take the steepest one.

• The third child said: Starting from the top of the mountain, just randomly choose the mountain road and walk until you reach the bottom of the mountain.

Although the boss 's walking method is optimal for every road, it will consume a lot of time in the process of finding the steepest mountain road.

Although the second child ’s walking method does not guarantee that the path will be optimal every time, it can ensure that the path will be better every time, and it does not need to spend a lot of time looking for the steepest mountain road.

The third child 's moves are more random, and the path he takes each time may be the best or the worst.

So who do you think reaches the bottom of the mountain first? After studying $After\; SGD$ , you will get your answer.

2. Introduction to SGD

2.1 Introducing problems

Give you $xy$ coordinate system, with some points on it, gives you a straight line through the origin$y=wx$ , how to fit these points in the fastest way?

In order to solve this problem, we need to define a goal for the problem, which is to minimize the deviation of all points from the straight line. Our commonly used error function is the mean square error , for a point $p_1$In other words, its mean square error with the straight line can be defined as $e_1$：

$e_1=(y_1-w{x}_1{)}^2=(w{x}_1-y_1{)}^2$
Complete square expansion:
$$e_1=w^{2x}{{\_}_1}^2-2(w{x}_1{y}_1)+{y_1}^2$$$$e_1={x_1}^{2w}{\_}^2-2(x_1{y}_1)w+{y_1}^{2Similarly}$$
, point$p2$，$p3$，.$\_$$...$， p n pn This is true for$p$$n$
$$e_2={x_2}^{2w}{\_}^2-2(x_2{y}_2)w+{y_2}^2$$$$e_3={x_3}^{2w}{\_}^2-2\left(x_3{y}_3\right)w+{y_3}^2$$$$e_n={x_n}^{2w}{\_}^2-2\left(x_n{y}_n\right)w+{y_n}^{2And}$$
our final error$$e=(\sum e_1+e_2+...+e_n)/nBy$$
merging similar items:

finally get:

Because $a={x_1}^2+...+{x_n}^2$ , so$a>0$ , so$e$ is an upward parabola.

After defining the error function, we can start to calculate the gradient.

Obviously, when$ewis$ the lowest point in the image,$e$ is the smallest, wwat this time$w\; is$ optimal.

How to quickly move from the yellow point on the right to the lowest point? This is stochastic gradient descent. Starting from its own position, it explores every other distance and randomly selects a direction with the largest gradient to move until it reaches the lowest point.
How often should we explore? This is the learning rate$learning$ $rate$了，当$learning$ $rate=0.1$时

当$learning$ $rate=$At$\mathrm{0.2,}$

a good learning rate can quickly minimize the points

2.2 Calculation steps of SGD

Going back to the problem of climbing the mountain just now, we learned through a large amount of data experiments that the second child’s $The\; SGD$ method can reach the bottom of the mountain the fastest.

3. Code implementation of SGD

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
X_scaler = StandardScaler()
y_scaler = StandardScaler()
X = [[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300],[50],[100],[150],[200],[250],[300]]
y = [[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330],[150],[200],[250],[280],[310],[330]]
#plt.show()
X = X_scaler.fit_transform(X) #用什么方法标准化数据？
y = y_scaler.fit_transform(y)
X_test = [[40],[400]] # 用来做最终效果测试
X_test = X_scaler.transform(X_test) 
model = SGDRegressor()
model.fit(X, y.ravel())
y_result = model.predict(X_test)
plt.title('single variable')
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.plot(X, y, 'k.')
plt.plot(X_test, y_result, 'g-')
plt.show()

result: