Brief description of the policy gradient algorithm

This article briefly introduces the policy gradient method in deep reinforcement learning (deep reinforcement learning) by sorting out the content of Mr. Li Hongyi's machine learning tutorial.

The B station link of Li Hongyi's course:
Li Hongyi, deep reinforcement learning, policy gradient

Related Notes:
Proximal Policy Optimization Algorithm Brief
DQN (deep Q-network) Algorithm Brief
Actor-Critic Algorithm Brief

Assume:
the trajectory of a game (trajectory): $\tau$
player (actor) strategy (policy):$i$

Then the expected value of incentive (reward) can be estimated by N sampling (sampling) (incentive $R$ is a random variable):
$${\bar{R}}_{}=\sum _{t}R\left(\tau \right)P(\tau |\theta )\approx \frac{1}{N}\sum _{n=1}^{N}R(t^n)$$

The optimal strategy is:
$$i^{\ast }=\arg \underset{i}{\max }{\bar{R}}_{}$$

Give the following equation for a gradient ascent
$$▽{\bar{R}}_{}=\sum _{t}R\left(\tau \right)▽P(\tau |\theta )=\sum _{t}R(\tau \left)P\right(\tau |\theta )\frac{▽P(\tau |\theta )}{P(\tau |\theta )}=\sum _{t}R\left(\tau \right)P(\tau |\theta )▽\ln P(\tau |\theta )\approx \frac{1}{N}\sum _{n=1}^{N}R(t^n)▽\ln P(t^n|\theta )$$

Among them, the operation principle of logarithm:
$$\frac{d\ln (f(x))}{dx}=\frac{1}{f(x)}\frac{df(x)}{dx}$$

Since the probability of the trajectory occurring under the condition of the policy:
$$P(\tau |\theta )=p(s_1)p(a_1|s_1,i)p(r_1,s_2|s_1,a_1)p(a_2|s_2,i)p(r_2,s_3|s_2,a_2)\cdots =p(s_1)\prod _{t=1}^{T}p(a_t|s_t,i)p(r_t,s_{t+1}|s_t,a_t)$$

Among them, $s$ is the game state (state) at each moment,$a$ is the player's action.
Only$p(a_t|s_t,\theta )$ part and the player's strategy$\theta$ is related, the other two terms$p(s_1)$ 和 $p(r_t,s_{t+1}|s_t,a_t)$ are independent of player strategy.

Let us define the equation:
$$\begin{gathered} \ln P(\tau |\theta )=\ln p(s_1)+\sum _{t=1}^T[\ln p(a_t|s_t,i)+\ln p(r_t,s_{t+1}|s_t,a_t)] \\ ▽\ln P(\tau |\theta )=\sum _{t=1}^T▽\ln p(a_t|s_t,i) \end{gathered}$$

We have the following equations:
$$▽{\bar{R}}_{}\approx \frac{1}{N}\sum _{n=1}^{N}R(t^n)▽\ln P(t^n|\theta )=\frac{1}{N}\sum _{n=1}^{N}R(t^n)\sum _{t=1}^{T_n}▽\ln p(a_t^n|s_t^n,i)=\frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}R\left(t^n\right)▽\ln p(a_t^n|s_t^n,i)$$

Note the following points:

First, the incentive multiplied by the above formula is the global benefit, not a single-step incentive, otherwise it will not be possible to learn the actions that motivate the subsequent moments. (The fourth point will be improved accordingly)

Second, the reason for taking the logarithm:

Since taking the logarithm and then finding the gradient is equivalent to calculating the gradient of the probability and dividing it by the probability itself:
$$▽\ln p(a_t^n|s_t^n,i)=\frac{▽p(a_t^n|s_t^n,i}{p(a_t^n|s_t^n,i)}$$

And dividing by the probability itself can prevent certain actions with low incentives from being sampled multiple times, resulting in the accumulation of excessive incentives:

Third, introduce the baseline (baseline):

When the incentive of the game is constant and non-negative, in order to prevent the probability value of the unsampled high incentive action from decreasing, the baseline is added:

One of the simplest ways to set the baseline is for $R\left(\tau \right)$ is averaged:
$$b\approx E\left[R\right(\tau \left)\right]$$

Fourth, assign appropriate credits to each action:

Actions at each moment, only considering the sum of all incentives after this time point until the end of the game:
$$▽{\bar{R}}_{}\approx \frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}(\sum _{t^{\prime }=t}^{T_n}{r}_{t^{\prime }}^n-b)▽\ln p(a_t^n|s_t^n,i)$$

Further, discount future incentives, that is, the longer the time, the smaller the influence: ▽
$$▽{\bar{R}}_{}\approx \frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}(\sum _{t^{\prime }=t}^{T_n}{c}^{t^{\prime }-t}{r}_{t^{\prime }}^n-b)▽\ln p(a_t^n|s_t^n,i)$$

Among them, the discount factor $The\; value\; range\; of\; \gamma$ is$[0,1]$ , usually take$0.9$ or$0.99$ 0.99$\_$$0.99$ , if take$0$ , it means only care about immediate incentives, if it is$1$ , it means that future incentives are equal to immediate incentives.