Reinforcement Learning: Policy Gradients

The idea of ​​strategy gradient method

  We used to express policies in the form of tables before, but now we can also express policies in functions. All the methods learned before are called value-based, and the next one is called policy-based. Next, let's take a look at the idea of ​​the strategy gradient method. The strategies learned before are all expressed in tables, as follows:

  Now, we change the table into a function, then $The\; writing\; of\; \pi$ will also change, as follows:

Among them,$\theta$ is a vector that can be used to representπ πThe parameters in the function$\pi \; .$

  The difference between tables and functions is the probability of obtaining an action. The table form directly looks up the table through the index, but using the function will be a little troublesome, you can’t directly go to the index, you need to calculate the corresponding $\pi (a|s,i)$

  The difference between table and function representations is also in the way the policy is updated. You can directly modify the values ​​in the table in the table. When expressed as a parameterized function, the policy $\pi$ can only be modified by modifying the parameter$\theta$ to update the policy.

.The
idea of ​​the strategy gradient method:
  when represented by a function, we will establish some scalar objective function $J\left(\theta \right)$ , by optimizing the objective function so that the strategy$\pi$ reaches the optimum, as follows:

Selection of scalar objective function

  Above we know that we need to establish a scalar objective function, so what is the scalar objective function? In general, we commonly use two types of scalar objective functions.

  The first one is the average value of the state value , or simply called the average value, which is actually a weighted average of the state value, as follows:

$\bar{v}$ is the weighted average of state value
$d\left(s\right)$ represents the state$The\; probability\; that\; s$ is selected

The above form can also be written in a more concise form, which is the inner product of two vectors:

.

  So, how do we choose $What\; about\; d\left(s\right)$ ? We have two situations, one is$d$和$\pi$ doesn't matter; it's$d$和$\pi$ is related.

  when $d$和$When\; \pi$ has no relationship, we use d 0 d_0respectively$d_0$和${\overline{Pi}}_0$means that we can also take the uniform distribution $d_0(s)=1/|S|=1/n$ , if a certain state is more important, then we can increase its weight.

  when $d$和$When\; \pi$ has a relationship, the commonly used method is to choose a stable distribution, as follows:

.
  The second is the average value of instant rewards, which is a weighted average of instant rewards, as follows:

The above is the first form of reward, and we often see another form of reward, as follows:

Among them, we assume that we follow a given strategy and generate a trajectory, and get a series of rewards $(R_{t+1},R_{t+2},...)$ ; after running an infinite number of steps,$s_0$It doesn't matter anymore, so put $s_0$removed
.

  Above we introduced the selection methods of the two scalar objective functions, and then we will further summarize these two scalars:
  1. They are all strategies π πFunction of$\pi$
  2. Strategy$\pi$ is in the form of a function whose parameter is$\theta$ , different$\theta$ will get different values
  ​​3, you can find the optimal$\theta$ to maximize the value of the scalar
   4,${\bar{r}}_{}$与${\bar{v}}_{}$are equivalent, and when one of them is optimized, the other is also optimized. In the discount factor $c<1$是，有${\bar{r}}_{}=(1-c){\bar{v}}_{}$

Policy Gradient Solving

   After getting a policy scalar, calculate its gradient. Then, gradient-based methods are applied for optimization, where gradient computation is one of the most complex parts. That's because, first of all, we need to distinguish between different ${\bar{v}}_{}$，${\bar{r}}_{}$，${\bar{v}}_{Pi}^0$; Second, we need to distinguish between discounted and undiscounted. The calculation of the gradient, here we will give a brief introduction.

$J\left(\theta \right)$ can be${\bar{v}}_{}$，${\bar{r}}_{}$，${\bar{v}}_{Pi}^0$;
$\eta$ is the distribution probability or weight
"=" can represent strict equality, approximation or proportional to

${\bar{v}}_{}$，${\bar{r}}_{}$，${\bar{v}}_{Pi}^0$The corresponding gradient formula is as follows:

.
Gradient formula analysis:
   we can write the above formula as follows:

$Sobeysthe\; \eta distribution$ ;$Aobeys\pi (A|S,\theta )distribution$

Why do we need such a formula? This is because the real gradient contains the expected $E$ , hope$E$ is not known, so we can approximate it by sampling for optimization, as follows:

Supplementary note:
   because it is necessary to calculate $ln\pi (a|s,\theta )$，so the requirementsπ$\pi (a|s,i)>0$ , how to ensure that all$\pi$ for all$a$ is all greater than 0? Use the softmax function for normalization, as follows:

So, p pThe expression of$\pi$

$h(s,a,\theta )$ is another function, usually obtained by a neural network.

Gradient Ascent and REINFORCE

   The basic idea of ​​the gradient ascent algorithm is that the real gradient has the expected $E$ , so the real gradient is replaced by a random gradient, but there is also a$q_{}(s,a)$ Immediate strategy$The\; real\; action\; value\; corresponding\; to\; \pi$ is unknown. Similarly, we use a method to approximate or to$q_{}$For sampling, the method is to combine with MC——reinforce, as follows:

.We
use random gradients instead of real gradients, so how do we make random variables $(S,A)$ What about sampling? First to$S$ is sampled because$Sobeysthe\; \eta distribution$ , which requires a large amount of data, and it is difficult to achieve a stable state in reality, so it is generally not considered in practice. What about$What\; about\; A$ sampling? Because$Aobeys\pi (A|S,\theta )distribution$ , therefore, in$s_t$Should be according to strategy $\pi \left(\theta \right)$对$a_t$Take a sample. All policy gradients here belong to the on-policy algorithm.

Algorithm understanding

requires $a{b}_t$is smaller, it can be found that $b_t$Able to balance algorithm discovery and data utilization. Because $b_t$ 与 $q_t(s_t,a_t)$ is proportional, therefore, when$q_t(s_t,a_t)$ is larger$b_t$will also be relatively large, meaning ${Pi}_t(s_t,a_t)$ has a higher probability of being selected. $b_t$ 与 $\pi (a_t|s_t,i_t)$ is inversely proportional, therefore, when$b_t$ 较大时 $\pi (a_t|s_t,i_t)$ will be relatively small, which means that if I choose${Pi}_t(s_t,a_t)$ is relatively small, give it a greater probability to choose it at the next moment.

当 $b_t>0$时，this is$\pi (a_t|s_t,\theta )$ Gradient rising algorithm, there are:

when$b_t<0$时，this is$\pi (a_t|s_t,\theta )$ Gradient descent algorithm, there are:

.

reinforcement algorithm

   用 $q_t(s_t,a_t)$ to approximately replace$q_{}(s_t,a_t)$ ， $q_t(s_t,a_t)$ is obtained by Monte Carlo method, that is, from$(s_t,a_t)$ start to get an episode, and then assign the return of this episode to$q_t$, this is the reinforcement algorithm.

.
Reinforce algorithm, its pseudo code is as

follows $For\; k$ iterations, first select an initial$state$ according to the current strategy$p\left(i_k\right)$ interacts with the environment to get an episode, and we have to go through each element in this episode. Then operate on each element, which is divided into two steps. The first step is to do value update, which is to use the Monte Carlo method to estimate$q_t$ ，从 $(s_t,a_t)$ to add up all the rewards obtained later. The next step is policy update, the qt q_tthat will be obtained$q_t$Substitute into the formula to update $i_t$, and finally take the final obtained $i_T$As a new $i_k$Perform iterative updates.