Introduction to Deep Reinforcement Learning (DRL) and Classification of Common Algorithms (DQN, DDPG, PPO, TRPO, SAC)

Briefly introduce the basic concepts of deep reinforcement learning, common algorithms, processes and their classification (continuously updated), so that everyone can better understand and apply reinforcement learning algorithms, and better solve cutting-edge problems in their respective fields. Welcome everyone to leave a message to discuss and make progress together.

(PS: If you only pay attention to the algorithm implementation, you can directly read the content of parts 3 and 4.)

1. Reinforcement Learning

Reinforcement Learning (RL): Reinforcement Learning
Reinforcement Learning is a type of machine learning. It is different from supervised learning and unsupervised learning. Through the continuous interaction between the agent and the environment (that is, taking actions), and then obtain rewards, so as to continuously optimize its own action strategy , in anticipation of maximizing its long-term payoff (sum of rewards). Reinforcement learning is particularly well suited to sequential decision problems (involving a series of ordered decision problems).

In practical applications, we often cannot accurately label each data or state for certain tasks, but we can use reinforcement learning to know or evaluate the current situation or whether the data is good or bad. For example, play Go, Starcraft II (Starcraft II), and other games.

1.1 Definition of Reinforcement Learning

Agent interacts with its surroundings known as the environment. Agent will get a reward from the environemnt once it takes an action in the current enrivonment. Meanwhile, the environment evolves to the next state. The goal of the agent is to maximize its total reward (the Return) in the long run.

The agent continuously interacts with the environment (i.e. takes an action in a given state), and thus obtains a reward, while the environment moves from one state to the next. By continuously optimizing its own action strategy, the agent expects to maximize its long-term reward or benefit (sum of rewards).

1.2 Related concepts of reinforcement learning

（1）状态 State (S): agent’s observation of its environment;
（2）动作 Action (A): the approaches that agent interacts with the environment;
（3）奖励 Reward ( $R_t$ ): the bonus that agent get once it takes an action in the environment at the given time step $t$ .
Return (Return) is the sum of rewards obtained by the Agent.
(4) Transition Probability Transistion Probability ( P ): the transition possibility that environment evolves from one state to another. The transition of the environment from one state to another
can be a deterministic transition process, for example,$S_{t+1}=f(S_t,A_t)$ , it can also be a random transfer process, such as$S_{t+1}\sim p\left(S_{t+1}|S_t,A_t\right)$ .
(5) Discount factor Discount factor ($\gamma$ ): to measure the importance of future reward to agent at the current state.
(6) Trajectory is a series of states, actions, and rewards, which can be expressed as:
$$t=\left(S_0,A_0,R_0,S_1,A_1,R_1,...\right)$$

Use the trajectory $\tau$ to record how the agent interacts with the environment. The initial state of the trajectory is randomly sampled from the distribution of starting states. A track is sometimes called an episode (Episode) or a round, which is a sequence from the initial state (Initial State, such as the opening of the game) to the final state (Terminal State, such as death or victory in the game).
(7) Exploration-Exploitation Tradeoff (Exploration-Exploitation Tradeoff)
Here, exploration means that the agent obtains more information by interacting with the environment, and utilization means using the currently known information to make the agent perform optimally , for example, the greedy strategy. At the same time, you can only choose one of the two. Therefore, how to balance exploration and utilization to maximize long-term returns is a very important issue in reinforcement learning.

Therefore, you can use (S, A, P, R, $\gamma$ ) to describe the reinforcement learning process.

1.3 Mathematical Modeling for Reinforcement Learning

(1) Markov Process (MP) is a discrete random process with Markov properties.

The Markov property refers to the next state $S_{t+1}$only depends on the current state $S_t$.

$$p\left(S_{t+1}|S_t\right)=p\left(S_{t+1}|S_0,S_1,S_2,\dots ,S_t\right)$$

A finite state set $S$ and state transition matrix$P$ means that the MP process is$<\mathcal{S},\mathbf{P}>$.

(2) Markov Reward Process (Markov Reward Process, MRP)

In order to be able to describe the environment's feedback rewards to the Agent, the Markov reward process changes the above MP from $<S,P>$ Expanded to$<S,\mathbf{P},R,c>$ . Here,$R$ represents the reward function, and$\gamma$ represents the reward discount factor.

$$R_t=R(S_t)$$

Return (Return) is the cumulative reward of the Agent on a trajectory. The discounted return is defined as follows:

$$G_{t=0:T}=R\left(\tau \right)=\sum _{t=0}^T{c}^t{R}_t$$

Value Function $V\left(s\right)$ is the agent in state$s$ expected return (Expected Return).
$$V^{\pi }\left(s\right)=E\left[R\right(\tau )|S_0=s]$$

(3) Markov Decision Process (Markov Decision Process, MDP)
MDP is widely used in economics, cybernetics, queuing theory, robotics, network analysis and many other fields.
Immediate rewards for Markov decision processes (Reward, $R$ ) is related to state and action. MDP can be used$<\mathcal{S},\mathcal{A},\mathbf{P},R,c>$ to characterize.
$A$ represents a limited set of actions. At this time, the immediate reward becomes
$$R_t=R(S_t,A_t)$$

Policy (Policy) is used to describe the way the Agent takes actions based on environmental observations. Policy is from a state $s\in S$ sum action$a\in A$ to action probability distribution$\pi (a|s)$ projection,$\pi (a|s)$ means in state$s$ , take action$a$的概率。
$$\pi (a|s)=p(A_t=a|S_t=s),\exists t$$

Expected Return (Expected Return) refers to the expected return value of all possible trajectories under a given strategy, which can be expressed as:
$$J\left(p\right)={\int }_{}p(\tau |\pi \left)R\right(\tau )={\mathbb{E}}_{\tau \sim \pi }[R(\tau )]$$

Here, $p(\tau |\pi )$ represents the given initial state distribution$r_0$and strategy $\pi$ , a TTin a Markov decision processTrajectory τ \tauof$T\; step$$The\; probability\; of\; occurrence\; of\; \tau$ is as follows:

$$p(\tau |\pi )=r_0(s_0)\prod _{t=0}^{T-1}p(S_{t+1}|S_t,A_t)p(A_t|S_t)]$$

Reinforcement learning optimization problems improve policies by optimization methods to maximize expected rewards. Optimal policy ${Pi}^{\ast }$ can be expressed as

$${Pi}^{\ast }=\arg \underset{Pi}{\max }J\left(p\right)$$

Given a strategy $\pi$ , value function$V\left(s\right)$ , the expected return at a given state, can be expressed as

$$V^{\pi }\left(s\right)={\mathbb{E}}_{\tau \sim \pi }\left[R\right(\tau )|S_0=s]=E_{A_t\sim \pi (\cdot |S_t)}\left[\sum _{t=0}^{\infty }{c}^{t}R(S_t,A_t)|S_0=s\right]$$

In MDP, given an action, there is an Action-Value Function (Action-Value Function), which is based on the state and the expected return of the action. It is defined as follows:

$$Q^{\pi }(s,a)={\mathbb{E}}_{\tau \sim \pi }\left[R\right(\tau )|S_0=s,A_0=a]=E_{A_t\sim \pi (\cdot |S_t)}\left[\sum _{t=0}^{\infty }{c}^{t}R(S_t,A_t)|S_0=s,A_0=a\right]$$

According to the above definition, we can get:
$$V^{\pi }\left(s\right)={\mathbb{E}}_{a\sim \pi }[Q^{\pi }(s,a)]$$

2. Deep reinforcement learning

Deep Learning + Reinforcement Learning = Deep Reinforcement Learning (DRL)
deep learning DL has strong abstraction and representation capabilities, especially suitable for modeling value functions in RL, for example: action value function $Q^{Pi}\left(s,a\right)$ .
The combination of the two greatly expands the application range of RL.

3. Common Deep Reinforcement Learning Algorithms

There are many algorithms for deep reinforcement learning, the common ones are: DQN, DDPG, PPO, TRPO, A3C, SAC and so on.
(The brief process of each algorithm will be supplemented later)

3.1 Deep Q-Networks （DQN）

The DQN network combines Q-Learning and deep learning, and introduces two novel techniques to solve the problem of using nonlinear function approximators such as neural networks to represent the action value function Q ( s , a ) Q(s,a $Q(s,a)$ The resulting instability problem:
Technique$\stackrel{◯}{1}$ : Experience Replay Buffer (Replay Buffer): Store the experience obtained by the Agent in the cache, and then uniformly adopt (or consider priority-based sampling) small batch samples from the cache for Q-Learning update; Technology
2$\stackrel{◯}{2}$ : Target Network: Introduce an independent network to replace the required Q network to generate the Q-Learning target and further improve the stability of the neural network.

Here, technique $\stackrel{◯}{1}$ It can improve the efficiency of sample usage, reduce the correlation between samples, and smooth the learning process; technology$\stackrel{◯}{2}$ It can be that the target value is not affected by the latest parameters, and there is much less divergence and shock.

Tested on various games in Atari, the DQN algorithm shows excellent performance.

The specific description of the DQN algorithm is as follows:

1. Hyperparameter : playback cache capacity $B$ , reward discount factor$\gamma$ , target network update delay step size$C$ ;
2.Input: playback buffer$B$ , initialize the action value function network$Q(s,a|{\theta }^Q)$, target network$Q^{\prime }(s,a|{\theta }^{Q^{\prime }})$$；
3. \qquad Initialize the target network parameters $Q^{\prime }$ :$i^{Q^{\prime }}\leftarrow i^Q$；
4. for $episode=1,\cdots ,M$ do
5. \qquad Initialize the environment;
6. \qquadThe initial state is $S_0$；
7. \qquad for $t=1,\cdots ,T$ do
8. \qquad\qquad With probability $ϵ$ choose a random action$A_t$; Otherwise, choose action $A_t=\arg \underset{a}{\max }Q\left(S_t，a|{\theta }^Q\right)$；
9. \qquad\qquad Execute action $A_t$get reward $R_t$, observe the next state $S_{t+1}$；
10. \qquad\qquad Store state transition samples $\left(S_t,A_t,R_t,S_{t+1}\right)$ to$\mathcal{B}$；
11. \qquad\qquad from cache B \mathcal{B}Random samplingNN$in\; B$$N$个状态转移$\left(S_i,A_i,R_i,S_{i+1}\right)$；
12. \qquad\qquad 设置 $Y_i=R_i+c\underset{a}{\max }{Q}^{\prime }\left(S_{i+1},a|{\theta }^{Q^{\prime }}\right)$；
13. \qquad\qquad Update the network Q by minimizing the loss function $Q(s,a|{\theta }^Q)$：
14. \qquad\qquad $L=\frac{1}{N}{\sum }_i{\left(Y_i-Q(S_i,A_i|{\theta }^Q)\right)}^2$；
15. \qquad\qquad per $Step\; C$ , for the target network$Q^{\prime }(s,a|{\theta }^{Q^{\prime }})$to synchronize:
16.\qquad\qquad $i^{Q^{\prime }}\leftarrow i^Q$；
17. \qquad end for
18. end for

Note: here random action selection probability$ϵ$ generally decreases gradually with the increase of iteration Episode and Time Step, the purpose is to reduce the influence of random strategy and gradually improve the influence of Q network on Agent action selection.

Solve,Line 14 is a simple equation:
$$i^Q\leftarrow i^Q+b\sum _{i\in \mathcal{N}}\frac{\partial Q(s,a|{\theta }^Q)}{\partial {\theta }^Q}[y_i-Q(s,a|{\theta }^Q)]$$
Among them, the setN \mathcal{N}NNfor minibatch in$N$$N$个$\left(S_t,A_t,R_t,S_{t+1}\right)$ experience sample set,$\beta$ represents the iteration step size in one gradient iteration.

References:
[1] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 101-1 529–533, Feb. 2015.

3.2 Deep Deterministic Policy Gradient（DDPG）

The DDPG algorithm can be regarded as the combination of the Deterministic Policy Gradient (DPG) algorithm and the deep neural network, which is an extension of the above-mentioned deep Q network (DQN) in the continuous action space.

DDPG establishes Q value function (Critic) and strategy function (Actor) at the same time. Here, Critic is the same as DQN, and is updated using the TD method; while Actor uses the estimate of Critic to update through the strategy gradient method.

The DDPG algorithm is specifically described as follows:

1. Hyperparameter : soft update factor $\rho$ , reward discount factor$\gamma$ ;
2.Input: playback buffer$B$ , initialize the critic network$Q(s,a|{\theta }^Q)$actor function$\pi (s|{\theta }^{\pi })$
3__Target network $Q^{\prime }$、${Pi}^{\prime }$；
4. \qquad Initialize the target network parameters $Q^{\prime }$ and${Pi}^{\prime }$：
5. \qquad $i^{Q^{\prime }}\leftarrow i^Q$ ,$i^{{Pi}^{\prime }}\leftarrow i^{\pi }$；
6. for $episode=1,\cdots ,M$ do
7. \qquad Initialize the random process $N$ , used to add exploration to actions;
8.\qquadThe initial state is $S_1$；
9. \qquad for $t=1,\cdots ,T$ do
10. \qquad\qquad Let $A_t=\pi (S_t|{\theta }^p)+{\mathcal{N}}_t$；
11. \qquad\qquad Execute action $A_t$get reward $R_t$, observe the next state $S_{t+1}$；
12. \qquad\qquad Store state transition samples $\left(S_t,A_t,R_t,S_{t+1}\right)$ to$\mathcal{B}$；
13. \qquad\qquad from cache B \mathcal{B}Random samplingNN$in\; B$$N$个状态转移$\left(S_i,A_i,R_i,S_{i+1}\right)$；
14. \qquad\qquad 设置 $Y_i=R_i+\gamma Q^{\prime }\left(S_{i+1},{Pi}^{\prime }(S_{i+1}|{\theta }^{{Pi}^{\prime }})|{\theta }^{Q^{\prime }}\right)$；
15. \qquad\qquad Update the Critic network by minimizing the loss function:
16. \qquad\qquad$L=\frac{1}{N}{\sum }_i{\left(Y_i-Q(S_i,A_i|{\theta }^Q)\right)}^2$；
13. \qquad\qquad Update the actor network by sampling the policy gradient:
14. \qquad\qquad ${\nabla }_{i^{}}J\approx \frac{1}{N}{\sum }_i{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=S_i,a=\pi (S_i)}{\nabla }_{{\theta }^{\pi }}\pi (s|{\theta }^{\pi }){|}_{S_i}$；
15. \qquad\qquad 更新目标网络：
16. \qquad\qquad ${\theta }^{Q^{\prime }}\leftarrow \rho {\theta }^Q+(1-\rho ){\theta }^{Q^{\prime }}$，
17. \qquad\qquad ${\theta }^{{\pi }^{\prime }}\leftarrow \rho {\theta }^{\pi }+(1-\rho ){\theta }^{{\pi }^{\prime }}$；
18. \qquad end for
19. end for

原论文中采用Ornstein-Uhlenbeck过程（O-U过程）作为添加噪声项$\mathcal{N}$，也可以采用时间不相关的零均值高斯噪声（相关实践表明，其效果也很好）。

参考文献：
[1] Lillicrap, Timothy P., et al. “Continuous control with deep reinforcement learning”，arXiv preprint, 2015, online: https://arxiv.org/pdf/1509.02971.pdf

3.3 Proximal Policy Optimization（PPO）

PPO算法是对信赖域策略优化算法（Trust Region Policy Optimization，TRPO）的一个改进，用一个更简单有效的方法来强制策略${\pi }_{\theta }$θ ′ \pi_{\theta}^{\prime }${Pi}_i^{\prime }$For example
, TRPO has a range of voltages:
$$\begin{gathered} \underset{{Pi}_i^{\prime }}{\max }{\mathcal{L}}_{{Pi}_{}}\left(p_i^{\prime }\right) \\ s.t.{\mathbb{E}}_{s\sim {\rho }_{{Pi}_{}}}\left[D_{KL}\left(p_{}||{\pi }_i^{\prime }\right)\right]\le \delta \end{gathered}$$
and the PPO algorithm directly optimizes the regular version of the above problem, namely:

$$\underset{{Pi}_i^{\prime }}{\max }{\mathcal{L}}_{{Pi}_{}}\left(p_i^{\prime }\right)-\lambda {\mathbb{E}}_{s\sim {\rho }_{{Pi}_{}}}\left[D_{KL}\left(p_{}||{\pi }_i^{\prime }\right)\right]$$

Here, $\lambda$ is the regularization coefficient, corresponding to each$\delta$ , there is a corresponding$\lambda$ , so that the above two optimization problems have the same solution. However,$The\; value\; of\; \lambda$ depends on${Pi}_{}$, therefore, in PPO, it is necessary to use a dynamically adjustable $lambda$ . Specifically, there are two methods:λ \lambda
by checking the KL divergence value$Whether\; \lambda$ is increased or decreased, this version of the PPO algorithm is called PPO-Penalty;
(2) Directly truncate the objective function used for the policy gradient to obtain a more conservative update, this method is called PPO-Clip.

The specific description of the PPO-Clip algorithm is as follows:

1. Hyperparameters : truncation factor $ϵ$ , the number of sub-iterations$M,B$ ;
2.Input: Initialization strategy parameter$\theta$ , initial value parameter$\varphi$；
3. for $k=1,\cdots$ do
4. \qquad Enforce policy π θ k \pi_{\theta_k} in the environment${Pi}_{i_k}$, and save the trajectory set ${\mathcal{D}}_k=\left\{v_i\right\}$；
5. \qquad Calculate the reward G ^ t \hat{G}_t will get${\hat{G}}_t$；
6. \qquad Based on the current value function $V_{{\varphi }_k}$, calculate the advantage function ${\hat{A}}_t$(using any method of estimating odds);
7. \qquadfor $m=1,\cdots ,M$ do
8. \qquad\qquad ${\ell }_t(i^{\prime })=\frac{{\pi }_{\theta }(A_t|S_t)}{{Pi}_{i_{old}}(A_t|S_t)}$；
9. \qquad\qquad 采用Adam随机梯度上升算法最大化PPO-Clip的目标函数来更新策略：
10. \qquad\qquad ${\theta }_{k+1}=\arg \underset{\theta }{\max }\frac{1}{|{\mathcal{D}}_k|T}\sum _{\tau \in {\mathcal{D}}_k}\sum _{t=0}^{T}f(i^{\prime })$；
11. \qquad end for
12. \qquad for $b=1,\cdots ,B$ do
13. \qquad \qquad Learn the value function by minimizing the mean squared error using gradient descent:
14. \qquad\qquad ${\varphi }_{k+1}=\arg \underset{i}{\max }\frac{1}{|{\mathcal{D}}_k|T}\sum _{\tau \in {\mathcal{D}}_k}\sum _{t=0}^T{\left(V_{}(S_t)-{\hat{G}}_t\right)}^2$；
15. \qquad end for
16. end for


其中，$f(i^{\prime })=\min \left({\ell }_t(i^{\prime })A^{{Pi}_{i_{old}}}(S_t,A_t),clip({\ell }_t(i^{\prime }),1-ϵ,1+ϵ)A^{{Pi}_{i_{old}}}(S_t,A_t)\right)$

For example, $clip(x,1-ϵ,1+ϵ)$ means that$x$ truncated at$[1-ϵ,1+ϵ]$中。

参考文献：
[1] Schulman, J. , et al. “Proximal Policy Optimization Algorithms”，arXiv preprint, 2017, online: https://arxiv.org/pdf/1707.06347.pdf
[2] Schulman J, Levine S, Abbeel P, et al. “Trust region policy optimization”, International conference on machine learning. PMLR, 2015: 1889-1897, online: http://proceedings.mlr.press/v37/schulman15.pdf

4. Classification of Deep Reinforcement Learning Algorithms

There are many classification standards for deep reinforcement learning algorithms. The common classification methods and specific categories are summarized as follows:

4.1 According to whether the strategies used in Agent training and testing are consistent

4.1.1 off-policy (off-track strategy, offline strategy)

The strategy used by the Agent during training (generating data) ${Pi}_1$The strategy used when testing with the agent (method evaluation and promotion) ${Pi}_2$Inconsistent.

For example, in the DQN algorithm, during training, $ϵ-greedy$ strategy; when testing performance or actual use, use$a^{\ast }=arg\underset{a}{\max }{Q}^{Pi}\left(s,a\right)$ Strategy.

Common algorithms are: DDPG, TD3, Q-learning, DQN, etc.

4.1.2 on-policy (same track strategy, online strategy)

The strategy used by Agent during training (data generation) and the strategy used during testing (method evaluation and promotion) are the same strategy $p$ .

Common algorithms include: Sarsa, Policy Gradient, TRPO, PPO, A3C, etc.

4.2 Different ways of strategy optimization

4.2.1 Value-based algorithms (value-based algorithms)

Value-based methods usually imply that the action-value function $Q^{\pi }(s,a)$ optimization, the optimal strategy selects the function$Q^{\pi }(s,a)$ The action corresponding to the maximum value, that is,${Pi}^{\ast }\approx \arg \underset{Pi}{\max }{Q}^{\pi }(s,a)$ , here,$\approx$ is caused by the function approximation error.

The value-based algorithm has the advantages of relatively high sampling efficiency, small value function estimation variance, and is not easy to fall into local optimum. The disadvantage is that it usually cannot deal with continuous action space problems, and the final strategy is usually a deterministic strategy.

Common algorithms include Q-learning, DQN, Double DQN, etc., which are suitable for Discrete action space. Among them, the DQN algorithm is based on the state-action function $Q(s,a)$ to select the optimal action.

4.2.2 Policy-based algorithms (policy-based algorithms)

The strategy-based method directly optimizes the strategy, and iteratively updates the strategy through customs clearance to maximize the cumulative reward (return). It has the advantages of simple policy parameterization and fast convergence, and is suitable for continuous or high-dimensional action spaces.

The policy gradient method (Policy Gradient Method, PGM) is a class of reinforcement learning methods that directly target the expected return through gradient descent (Gradient Descent, for the minimization problem) for policy optimization. It does not need to solve the optimization problem of value maximization in the action space, so it is more suitable for continuous and high-Dimension action space, and can also model random strategies naturally.

The PGM method directly optimizes the agent's strategy on the parameters of the neural network through the method of gradient ascent.

According to related theories, the expected return $J\left(f_{}\right)$ about the parameter$The\; gradient\; of\; \theta$ can be expressed as:

$${\nabla }_{}J(f_{})={\mathbb{E}}_{\tau \sim {\pi }_{}}\left[\sum _{t=0}^T{R}_t{\nabla }_{}\sum _{t^{\prime }=0}^T\log {Pi}_{}(A_{t^{\prime }}|S_{t^{\prime }})\right]={\mathbb{E}}_{\tau \sim {\pi }_{}}\left[\sum _{t^{\prime }=0}^T{\nabla }_{}\log {Pi}_{}(A_{t^{\prime }}|S_{t^{\prime }})\sum _{t=0}^T{R}_t\right]$$

当$T\to When\; \infty$ , the above formula can be expressed as:

$${\nabla }_{}J\left(f_{}\right)=E_{\tau \sim {\pi }_{}}\left[\sum _{t^{\prime }=0}^{\infty }{\nabla }_{}\log {Pi}_{}(A_{t^{\prime }}|S_{t^{\prime }})c^{t^{\prime }}\sum _{t=t^{\prime }}^{\infty }{c}^{t-t^{\prime }}{R}_t\right]$$
In practice,$c^{t^{’}}$ , so as to avoid the problem of overemphasizing the early states of the trajectory.

The above methods tend to have larger methods for gradient estimation (reward $R_t$The randomness of may grow exponentially with the trajectory length L). For this reason, the common method is to introduce a benchmark function $b(S_i)$ , just state$S_i$The function. The gradient above can be modified to:

$${\nabla }_{}J(f_{})={\mathbb{E}}_{\tau \sim {\pi }_{}}\left[\sum _{t^{\prime }=0}^{\infty }{\nabla }_{}\log {Pi}_{}(A_{t^{\prime }}|S_{t^{\prime }})\left(\sum _{t=t^{\prime }}^{\infty }{c}^{t-t^{\prime }}{R}_t-b(S_{t^{\prime }})\right)\right]$$

Common PGM algorithms include REINFORCE, PG, PPO, TRPO, etc.

4.2.3 Actor-Critic algorithms (actor-critic method)

The Actor-Critic method combines the above-mentioned value-based method and policy-based method, uses the value-based method to learn the Q-value function or the state value function V to improve the sampling efficiency (Critic), and uses the policy-based method to learn the policy function (Actor ), thus suitable for continuous or high-dimensional action spaces. Its shortcomings also inherit the shortcomings of the two. For example, Critic has an overestimation problem, while Actor has a problem of underexploration.

Common algorithms include DDPG, A3C, TD3, SAC, etc., suitable for continuous and high-Dimension action space,

4.3 Parameter update methods are different

Parameters updating methods

4.3.1 Monte Carlo method

Monte Carlo method: must wait for a trajectory $t_k$Cannot be updated until (true value) is generated.

Common algorithms include: Policy Gradient, TRPO, PPO, etc.

4.3.2 Temporal Difference method

Time-difference method: Bootstrapping (estimated value) can be updated in time at each step of action execution.

Common algorithms are: DDPG, Q-learning, DQN, etc.