Reinforcement Learning: Stochastic Approximation and Stochastic Gradient Descent

mean estimation

  Through the previous study, we know that expectations can be obtained through many samples. And to find $\stackrel{There\; are\; two\; methods\; of\; ˉ}{x}$ , one is to directly add the sampled data and then divide by the number, but this method is less efficient.

The second method is iterative calculation, that is, several data are counted as several data. The specific calculation is as follows:

Random approximation: Robbins-Monro (RM)

  Suppose we now need to solve the equation:
$$g(w)=0$$

Then there are two cases, one is the function expression we know, and the other is the expression we don't know. I don't know the expression, how to solve it? In fact, the RM algorithm can solve. Next, take solving the equation as an example, let's learn the RM algorithm, which is an iterative algorithm.

Among them, $a_k$is a positive coefficient. { wk } {\{w_k}\}{ wk​} is the input sequence,$\tilde{g}\{w_k,{the}_k\}$ is the output sequence. For the convenience of understanding, we give a concrete example:
$$g\left(w\right)=English(w\_-1)$$
We know thatg(w) g(w)The real solutionw ∗ w^*$of\; g$$($$w$$)$$w^{\ast }$ =1, given the initial value$w_1$=3,$a_k=1/k$，${the}_k=0$ , the calculated result$w^{\ast }=1$ as follows:

RM Algorithm: Convergence Analysis

  The above analysis is intuitive and not strict, now we give the following mathematically strict convergence conditions.

  Condition 1: Indicates that the required function $g$ is an increasing function whose gradient is bounded.

  Condition 2: Regarding the coefficient $a_k$
   $a_k$The sum of should be equal to infinity, why? We write out the MR algorithm separately, and the following formula can be obtained after adding:

It shows that if $a_k$The sum of is less than infinity, then $a_k\tilde{g}(w_k,{the}_k)$ is bounded, then$w_1$Can't be set arbitrarily, but $a_k$The sum of is equal to infinity, so that we can safely choose the initial value $w_1$ 。

   $a_k^2$The sum of should be less than infinity, indicating that $a_k$It will definitely converge to 0, why?

It can be seen that if $a_k$tends to 0, then there is $a_k\tilde{g}(w_k,{the}_k)$ tends to 0, that is,$w_{k+1}-w_k$tends to 0.

  We will find that in many reinforcement learning algorithms, $a_k$As a small enough constant, the efficiency of the algorithm is low because 1/k will become smaller and smaller. Even though the second condition is not satisfied in this case, the algorithm can still work efficiently because the actual number of iterations is limited.

  Condition 3: Regarding the coefficient $the$ , statement$The\; expectation\; of\; \eta$ is 0 and the variance is bounded.

Stochastic Gradient Descent: SGD

  The stochastic gradient descent (SGD) algorithm is widely used in the field of machine learning and RL. SGD is a special RM algorithm, and the mean estimation algorithm is a special SGD algorithm.

  Suppose our purpose is to solve an optimization problem, there is an equation:

$w$ is the parameter to be optimized. $X$ is a random variable. Expectations are about X. where$w$ and$X$ can be either a scalar or a vector. function$f(.)$ is a scalar.

  Now, our goal is to find the optimal $w$ minimizes the objective function. There are three ways to solve it:

Gradient Descent (GD)

The idea of ​​the GD algorithm is to descend along the direction of the gradient to find the minimum value, so that the obtained solution is close to the real value $w^{\ast }$ ，$a_k$is the step size, which controls the descending speed. Its characteristic is to know the gradient of the function. If you don't know the gradient function, how to solve it?

Batch Gradient Descent (BGD)

  Batch gradient descent uses data to avoid solving the gradient function. Through a large number of samples, the average value is used to approximate the expectation of the gradient function, as follows:

But the problem is that for each $w_k$Both require sampling many samples.

Stochastic Gradient Descent (SGD)

  The stochastic gradient descent method is compared with the gradient descent method: use ${▽}_{k}f(w_k,x_k)$ to replace the true gradient$E[{▽}_{k}f(w_k,x_k)]$ , compared with the batch gradient descent method, n=1, that is, only sample once.

.We
  consider such an optimization example:

   We know that the optimized solution $w^{\ast }=E\left[X\right]$ , we know that$A\; necessary\; condition\; for\; J\left(m\right)$ to reach the minimum value is$▽J(w)=0$ , we can get$w^{\ast }=E\left[X\right]$

   Solve with gradient descent (GD):

   Solve with gradient descent (GD):

SGD Algorithm: Convergence Analysis

  One of the basic ideas of SGD is from GD to SGD, because E is unknown, so simply remove E, and then use a sample to approximately replace this E, and replace the real gradient with a random gradient. This is SGD . We can easily know that there must be errors between them, as follows:

In the case of errors, can SGD be used to find the optimal solution? The answer is yes, so why? Actually, SG is a special RM algorithm. The proof is as follows:

  The problem to be solved by SGD is to optimize and solve such an equation. We can convert this optimization problem into a solution equation $g\left(w\right)$ , and then use the RM algorithm to solve it. So the SGD algorithm actually solves the equationg ( w ) g(w)An RM algorithm for the special problem of roots of$g$$($$w$$)\; .$

  Because SGD is a special RM algorithm, the convergence of the previous RM algorithm can be applied to the convergence analysis of SGD.

Properties of SGD Algorithm

  SGD uses stochastic gradients instead of real gradients, and stochastic gradients are random. Will it cause the randomness of SGD convergence to be relatively large? To answer this question, the relative error is used to analyze the difference between random and batch gradients. Using Lagrange's median value theorem, the following formula is obtained:

We assume $The\; second-order\; gradient\; of\; f$ is a positive number

to get the following deduction:

According to the above equation, we can get an interesting convergence property of SGD. Relative error$d_k$Given $|w_k-w^{\ast }|$is inversely proportional; when$|w_k-w^{\ast }|$δ k δ_kobtained$d_k$Smaller, SGD is similar to GD. when $w_k$close to $w^{\ast }$ , the relative error$d_k$may be larger, and at $w^{Convergence\; near\; \ast }$ exhibits more randomness.
.
  Now, we use an example to understand this property. $X\in R^2$ represents a random position on the plane. Its distribution is evenly distributed in a square area with the origin as the center and a side length of 20. Now randomly sample 100 samples{ xi } i = 0 100 {\{x_i}\}_{i=0}^{100}{ xi​}i=0100​, its true mean value is E[X] = 0, now we use the above algorithm to estimate the mean value, the result is as follows:

It can be found that when the initial guess of the mean is far from the true value, the SGD estimate can quickly approach the neighborhood of the true value, although when the estimate is close to the true value, it shows some randomness, but it still gradually approaches the true value.
.We
  may often encounter a deterministic SGD formula that does not involve any random variables. How to solve such a problem? As shown in the figure below:

$f(w,x_i)$ is a parameterized function,$w$ is the parameter to be optimized.
{ xi } i = 0 n {\{x_i}\}_{i=0}^{n}{ xi​}i=0n​is a set of real numbers, where $x_i$is not a sample of any random variable.

.So
  , can we use the SGD algorithm to solve it? We use its mean $x_k$Instead of the averaging process, the resulting formula is very similar to SGD, but the difference is that no random variables are involved. In order to use the SGD algorithm, we manually introduce random variables, we introduce a random variable $X$ , $X$ is defined in the set{ xi } i = 0 n {\{x_i}\}_{i=0}^{n}{ xi​}i=0n​on, $x_i$Obey the uniform distribution, the probability of each being taken is $1/n$ . So we transform into$E\left[f\right(w,X)]$ , solving this problem is naturally the SGD algorithm. as follows:

Comparison: BGD, MBGD, SGD

  Suppose we want to minimize $J(w)=E\left[f\right(w,X)]$ , given a set of random samples$\{x_i{\}}_{i=0}^n$. Solve this problem with batch gradient descent (BGD), stochastic gradient descent (SGD) and mini-batch gradient descent (MBGD), respectively.

  In the BGD algorithm, all samples are used in each iteration. When $n$ 很大时，$(1/n)\ast S_{i=1}^n▽wf(w_k,x_i)$ close to the real gradient$E[▽wf(w_k,x_i)]$。

  In the MBGD algorithm, $I_k$is $1,..,$A subset of$n$$|I_k|=m$ , set$I_k$by $m$ times of independent sampling.

  In the SGD algorithm, $x_k$is from { xi } i = 0 n {\{x_i\}_{i=0}^n}{ xi​}i=0n​obtained by random sampling.

Summary:
  To a certain extent, MGBD includes BGD and SGD. when $m=1$ becomes SGD, when$When\; m$ is large, it does not completely become BGD because MBGD uses n samples taken randomly, while BGD uses all n samples, where MBGD may use the same value multiple times, while BGD only uses each number once. Compared with SGD, MBGD has less randomness because it uses more samples than just one sample in SGD; compared with BGD, MBGD does not need to use all samples in each iteration, making it More flexible and efficient.

case
   at a given $x_i$, our goal is to calculate the mean $(1/n)\ast S_{i=1}^n||w-x_i||$ . This problem can be equivalently formulated as the following optimization problem: