A brief description of actor-critic related algorithms

This article briefly introduces the relevant algorithms based on actor-critic in deep reinforcement learning (deep reinforcement learning) by sorting out the content of teacher Li Hongyi's machine learning tutorial.

The B station link of Li Hongyi's course:
Li Hongyi, deep reinforcement learning, actor-critic

Related Notes:
A brief description of the policy gradient algorithm A brief description of
the proximal policy optimization algorithm A brief description of
the DQN (deep Q-network) algorithm

asynchronous advantage actor-critic（A3C）

References:
Volodymtr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, “Asynchronous Methods for Deep Reinforcement Learning”, ICML

Among the related algorithms based on actor-critic, the most well-known method is asynchronous advantage actor-critic, or A3C for short.
If you remove asynchronous (asynchronous), it is an advantage actor-critic, that is, A2C.

首先回顾一下策略梯度法：
$$▽{\bar{R}}_{}\approx \frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}(\sum _{t^{\prime }=t}^{T_n}{c}^{t^{\prime }-t}{r}_{t^{\prime }}^n-b)▽\ln p_{}(a_t^n|s_t^n)$$

Record the cumulative incentive as $G$：
$$G_t^n=\sum _{t^{\prime }=t}^{T_n}{c}^{t^{\prime }-t}{r}_{t^{\prime }}^n$$

However, due to the randomness of the interaction process itself, the cumulative incentive may be very unstable, and the number of samples before each update of the parameters cannot be many, and problems may arise:

So consider using cumulative incentives to $The\; expected\; value\; of\; G$ is used instead of the sampled value, and the method to obtain the expected value is a value-based method, namely DQN.

DQN has two kinds of critics, namely the state value function (state value function) $V^{\pi }\left(s\right)$and the state-action value function (state-action value function)$Q^{\pi }(s,a)$：

Among them, both the Monte-Carlo based approach (MC) and the temporal-difference approach (TD) can be used for estimation. The temporal difference is relatively stable, and Monte Carlo is more accurate.

In fact, the cumulative incentive $The\; expected\; value\; of\; G$ is$Q$，即：
$$E[G_t^n]=Q^{{Pi}_{}}(s_t^n,a_t^n)$$

Regarding the baseline, a common practice is to use the state-value function $V^{{Pi}_{}}\left(s_t^n\right)$ to represent, that is:

The disadvantage of the above method is that it is necessary to estimate $Q$ and$V$ two networks, the risk of inaccurate estimates is doubled. Actually, we can estimate only one network,$V$ network, with$The\; value\; of\; V$ to representthe QQThe value of$Q.$Since:
$$Q^{\pi }(s_t^n,a_t^n)=E[r_t^n+V^{\pi }(s_{t+1}^n)]$$

Remove the expectation and use $V$ stands for$Q$：
$$Q^{\pi }(s_t^n,a_t^n)=r_t^n+V^{\pi }\left(s_{t+1}^n\right)$$

Then get the expression of advantage function (advantage):
$$r_t^n+V^{\pi }\left(s_{t+1}^n\right)-V^{\pi }\left(s_t^n\right)$$

update strategy $Give\; the\; \pi values$ ​​as follows:
$$▽{\bar{R}}_{}\approx \frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}(r_t^n+V^{\pi }(s_{t+1}^n)-V^{\pi }(s_t^n))▽\ln p_{}(a_t^n|s_t^n)$$

As for the reason for removing the expected value, it is the result of the original A3C paper trying various methods for experimental comparison.

The overall process is as follows:

In the strategy gradient method, after collecting the data, it is necessary to update the strategy;
but in the actor-critic method, instead of directly using those data to update the strategy, first use these data to estimate the value function $V$ (the method based on Monte Carlo or the method based on time series difference can be used), and then based on the value function, use the above formula to update the strategy$p$ .

There are two tricks here.

First, when implementing actor-critic, we need to estimate two networks, $The\; network\; of\; V$ functions and the network of policies$\pi$ . Since the input of these two networks is the state$s$ , so the first few layers can be shared, especially for games where the input is an image:

Second, we need a mechanism for exploration. A common approach is to policy $The\; output\; of\; \pi$ is a constraint so that the entropy of the output distribution should not be too small, that is, the probability of exploring different actions is expected to be average. In this way, you can try various actions and explore the environment better.

The above algorithm is the advantage actor-critic (A2C), and finally explain what is asynchronous (asynchronous).

As shown in the figure below, each actor is running in parallel, that is, "each does its own thing, regardless of each other". When a process finishes running and prepares to return parameters, the original parameters may have been overwritten by other actors, but the It doesn't matter, just update directly, which is "asynchronous". (Note: △ θ \triangle \theta
in the figure$△\theta$ should be$▽\theta$）

pathwise derivative policy gradient

参考文献：
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller, “Deterministic Policy Gradient Algorithms”, ICML 2014
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nocolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, “Continuous Control with Deep Reinforcement Learning”, ICLR 2016

Next, the pathwise derivative policy gradient algorithm is introduced. The idea of ​​this algorithm is that the critic not only evaluates the quality of the action, but also tells the actor what action is good, namely:
$$\arg \underset{a}{\max }{Q}^{\pi }(s,a)$$

But if it is a continuous action scene, it will be more difficult to solve the above optimization problem, so you can learn another network to solve this optimization problem (analogous to GAN):

The algorithm flow is, first use a strategy $\pi$ interacts with the environment, estimating$Q$ function, then fix$Q$ , and then learn the strategy$\pi$ , get a better actor, and so on, as shown in the figure below:

Similar to the real QQ in the target network$Q$ Network and Target$Q$ network (see DQN section), there are also two actors in this algorithm, that is, the real$\pi$ and target$\widehat{Pi}$ .
In addition, the techniques mentioned before, such as replay buffer and exploration, etc. can be used.

The specific process is as follows (compared with the original DQN):

The relationship between actor-critic and GAN

Actor-critic is very similar to GAN. For details, please refer to the paper:
David Pfau, Oriol Vinyals, "Connecting Generative Adversarial Networks and Actor-Critic Methods", arXiv preprint 2016

Both are known to be difficult to train, so the article collects various methods on how to train GAN. And because there are two groups of people doing GAN and actor-critic, the article lists which methods of the two algorithms have been tried, as shown in the figure below: