4. Reinforcement learning--model free control

All the content of this course will finally focus on the content of this lecture. Through the study of this lecture, we will learn how to train an Agent so that it can complete tasks better in a completely unknown environment and get as many rewards as possible. .
The previous lecture mainly explained how to make predictions when the model is unknown. The so-called prediction is to evaluate a given strategy, that is, to determine the value function of a state (or state-behavior pair) under a given strategy . The content of this lecture is mainly about how to optimize the value function under unknown conditions of the model . This process is also called model-independent control.

Strategy iteration under model

The core of general policy iteration is policy optimization between two alternating processes. One process is strategy evaluation and the other is improvement strategy.
Note that using the dynamic programming algorithm to improve the strategy requires knowing all subsequent states of a certain state and the transition probabilities between states :
$${Pi}^{\prime }(s)=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{\mathcal{R}}_s^a+{\mathcal{P}}_{s{s}^{\prime }}^{a}V\left(s^{\prime }\right)$$

• For model-free policy iteration
  , is this method suitable for model-unknown Monte Carlo learning? The answer is no, and there are at least two problems . One is that when the model is unknown, it is impossible to know all subsequent states of the current state, and thus it is impossible to determine what behavior is more appropriate in the current state. The way to solve this problem is to use the value Q ( s , a ) Q(s,a) under the state-behavior pair$Q(s,a)$ instead of state value:
  The purpose of this is to improve the strategy without knowing the entire model. You only need to know what kind of behavior has the greatest value in a certain state. Specifically: we start from an initial Q and policy$Starting\; from\; \pi$ , first update Q (s, a) Q(s,a)of each state-behavior pair according to this strategy$Q(s,a)$ value, s is then determined based on the updated Q by the improved greedy algorithm.
  Even so, there is at least one problem, that is, when we use the greedy algorithm to improve the strategy every time, it will very likely lead to a suboptimal strategy due to insufficient sampling experience. We need to try from time to time. Some new behaviors, this is exploration.

Ɛ-Greedy exploration (MC-control)

Ɛ-The goal of greedy exploration is to ensure that all possible behaviors in a certain state have a certain non-zero probability of being selected for execution, which ensures continuous exploration, $1-$Choose the behavior that is currently considered best under the probability of$ϵ$$The\; probability\; of\; choosing\; ϵ$ among all possible actions (including the currently best action). The mathematical expression is as follows:
$$\pi (a|s)=\{\begin{array}{ll}ϵ/m+1-ϵ& \text{ if }{a}^{\ast }=\underset{a\in A}{\mathrm{argmax}}Q(s,a)\\ ϵ/& \text{ otherwise }\end{array}$$
m is the total number of actions.

$ϵGreedy$ strategy improvement theorem:
For any$ϵ-greedy$ strategy$In\; terms\; of\; \pi$ , use the corresponding$q_{}$The obtained $ϵ-greedy$ strategy${Pi}^{\prime }$ is atπ \piA strategy improvement on$\pi$$v_{{Pi}^{\prime }}(s)\ge v_{}\left(s\right)$

Proof:
Refer to https://zhuanlan.zhihu.com/p/54272316.
Of course, I feel that the above reference link is not complete and has flaws, so I will improve it: introduce ϵ − soft − greedy \epsilon-soft
here. $ϵ-soft-greedy$ concept, which means that for all states and actions,$\pi (a|s)\ge \frac{}{m}$, where $ϵ>0$ ϵ −$ϵ-greedy$属$ϵ-soft-greedy$ ,determine the max weight sum of the weighted average
value of the ppt:
$$\begin{array}{rl}{q}_{}\left(s,{Pi}^{\prime }(s)\right)& =\sum _{a\in \mathcal{A}}{Pi}^{\prime }(a|s)q_{}(s,a)\\ & =ϵ/m\sum _{a\in A}{q}_{}(s,a)+(1-)\_\underset{a\in A}{\max }{q}_{}(s,a)\\ & \ge ϵ/m\sum _{a\in A}{q}_{}(s,a)+(1-)\_\sum _{a\in A}\frac{\pi (a|s)-\frac{}{m}}{1-ϵ}{q}_{}(s,a)\\ & =\sum _{a\in A}\pi (a|s\left)q_{}\right(s,a)=v_{}(s)\end{array}$$
The most elusive origin of greater than or equal to is proved by the following method:
According to $ϵ-soft$ function,$\pi (a|s)\ge \frac{}{m}$,indefinitelyπ $\pi (a|s)=\frac{}{m}+\Delta maintaining$ a free function:
$$\pi (a|s)=\{\begin{array}{ll}1-ϵ-(m-1)D+\frac{}{m}& \text{ for greedy action }\\ \frac{}{m}+& \text{ for every non-gre }\end{array}$$
It is assumed here:

1. There are m-1 non-greedy actions, there are 1 greedy actions, and $The\; sum\; of\; \pi (a|s)$ = 1, so for greedy action$\pi (a|s)$ :
   $$1-ϵ+m/ϵ-(m-1)D$$
2. In addition, all non-greedy Q(s,a) values ​​are uniformly equal to the largest Q value $q_s$;The Q value of greedy action is $q_m$。
   $$\underset{a\in \mathcal{A}}{\max }{q}_{}(s,a)=q_m$$
   所什有下式：
   $$\begin{array}{rl}\sum _{a\in A}\frac{\pi (a|s)-\frac{}{m}}{1-ϵ}{q}_{}(s,a)& <=\frac{\left(1-ϵ-\Delta m\_+D+\frac{}{m}\right)-\frac{}{m}}{1-ϵ}{q}_m+\frac{\left(\frac{}{m}+D\right)-\frac{}{m}}{1-ϵ}{q}_s(m-1)\\ & =\frac{(1-ϵ-\Delta m\_+D)q_m+D(m-1)q_s}{1-ϵ}\\ & =\frac{(1-ϵ)q_m-D(m-1)q_m+D(m-1)q_s}{1-ϵ}\\ & =q_m+D(m-1)\left(q_s-q_m\right)\\ & <=q_m=\underset{a\in A}{\max }{q}_{}(s,a)\end{array}$$
   The above proves that $q_{}(s,{Pi}^{\prime }(s\left)\right)$ratiov$v^{\pi }\left(s\right)$is larger, but it does not prove that$v^{{Pi}^{\prime }}\left(s\right)$ratiov$v^{\pi }(s)$要大，所以还要再证一发这个：
   $$\begin{array}{rl}{Q}^{\pi }(s,{Pi}^{\prime }(s))& =E[r_{t+1}+V^{\pi }(s_{t+1})|s_t=s,a_t={Pi}^{\prime }(s_t)]\\ & <=E[r_{t+1}+Q^{\pi }(s_{t+1},{Pi}^{\prime }(s_{t+1}))|s_t=s,a_t={Pi}^{\prime }\left(s_t\right)]\\ & =E[r_{t+1}+r_{t+2}+V^{\pi }(s_{t+2})|....]\\ & <=E[r_{t+1}+r_{t+2}+Q^{\pi }(s_{t+2},{Pi}^{\prime }(s_{t+2}))|....]\\ & ......\\ & <=V^{{Pi}^{\prime }}(s)\end{array}$$
   It is fully proved through the above formula: $v^{{Pi}^{\prime }}(s)\ge v^{\pi }\left(s\right)$.
   After solving the above two problems, we finally see the whole picture of Monte Carlo control: using the Q function for strategy evaluation and using Ɛ-greedy exploration to improve the strategy. This method can eventually converge to the optimal policy.
   
   Each upward or downward arrow in the figure corresponds to multiple Episodes. That is to say, we generally perform sequential Q function updates or strategy improvements after experiencing multiple Episodes. In fact, we can also update the Q function or improve the strategy after each episode. But no matter which method is used, under Ɛ-greedy exploration calculation we can always only get the approximate Q function based on a certain strategy, and the algorithm does not have a termination condition because it is always exploring. Therefore, we must pay attention to the following two aspects: on the one hand, we do not want to lose any better information and status; on the other hand, as our strategy improves, we ultimately hope to end up with an optimal strategy, because in fact, the optimal strategy should not Includes some random behavior choices. Another theoretical concept is introduced for this purpose:GLIE.

GLIE(Greedy in the Limit with Infinite Exploration)

The idea of ​​this algorithm is to explore infinite possibilities in a limited time. The specific performance is:

1. All experienced state-action pairs will be explored infinitely;
2. In addition, with the infinite extension of exploration, the Ɛ value in the greedy algorithm tends to 0. For example if we take $ϵ=1/k$ (k is the number of episodes explored), then the Ɛgreedy Monte Carlo control has GLIE characteristics.

The Monte Carlo control process based on GLIE is as follows:

Theorem about GLIE: GLIE Monte Carlo control can converge to the optimal state behavior value function. That is $Q(s,a)$ will converge to$q_{\ast }(s,a)$(Find relevant papers to supplement the proof)

TD Control

on-policy TD Control

As mentioned in the previous lecture, TD has many advantages over MC: low variance, online real-time learning, incomplete Episode learning, etc.
So it is natural to think whether we can use TD learning instead of MC learning on control problems? The answer is yes, this is SARS that will be explained below.

SAUCE

The name of SARSA comes from the sequence description shown in the figure below: for a state S and a specific behavior A, a state-behavior pair (S, A) is generated, which interacts with the environment. After the environment receives the individual's behavior, it will tell The individual's immediate reward [formula] and the subsequent state S' it enters; then the individual follows the existing strategy to generate a behavior A', and obtains the value Q of the next state-behavior pair (S', A') based on the current state behavior value function (S',A'), use this Q(S',A') value to update the value of the previous state behavior to Q(S,A).
The flow chart of the algorithm is shown below:

A more intuitive explanation is this: an Agent is in a certain state S, in which it can try various behaviors. When following a certain strategy, it will choose based on the current strategy. For a behavior A, the individual actually performs this behavior and actually interacts with the environment. The environment will give an immediate reward R according to its behavior and enter the next state S'. In this subsequent state S', it will follow the current strategy again and generate a behavior. A', at this time, the individual does not perform the behavior , but obtains the value of the (S', A') state-behavior pair through its current state-behavior value function, and uses this value to take behavior A in combination with the individual's state S. The immediate reward obtained updates the (state) behavior value of the individual taking action A in state S.

Pseudocode of SARSA strategy:

Note:
1. Q(s,a) in the algorithm is stored in a large table, which is not suitable for solving large-scale problems;
2. For each Episode, use Behavior A is based on the current strategy, and this behavior is also the behavior that actually occurs in Episode. In the value cycle of updating the Q(S,A) state behavior pair, the individual does not actually perform the behavior A' under S', but Is to leave behavior A' to the next loop execution. That is, in fact, the last line of pseudo code assigning A to A' is redundant, because the action of the next state is selected in the next loop, and the next loop executes the strategy obtained by the new round of value function . This is the meaning of on-policy. I use the policy obtained by the current value function to select the action of this cycle, the updated value function and the state of the next cycle, and provide a new strategy for the selection of the next action.

Regarding the convergence theorem of SARSA : When the following two conditions are met, the Sarsa algorithm will converge to the optimal behavior value function. Condition 1: Strategy π t ( a ∣ s ) \pi_t(a|s)
at any time${Pi}_t(a|s)$ conforms to the GLIE characteristics;
condition 2: step coefficient$a_t$满足：
$$\begin{array}{l}{\sum }_{t=1}^{\infty }{a}_t=\infty \\ {\sum }_{t=1}^{\infty }{a}_t^2<\infty \end{array}$$
But according to what was taught in class and based on experience, convergence can be achieved even if the above two conditions are not met.

n walking SARSA

In TD( λ \lambdan-steps reward was mentioned in$\lambda$

): A similar idea can be used in SARSA to define the Q return of n-step:
$$q_t^{(n)}=R_{t+1}+\gamma R_{t+2}+\dots +c^{n-1}{R}_{t+n}+c^{n}Q\left(S_{t+n}\right)$$
here$q_t$Corresponding to a state-behavior pair $<s_t,a_t>$ , represents the value of taking a certain action in a certain state. If n=1, it means the state behavior pair$<s_t,a_t>$ The Q value can be expressed in two parts. One part is the immediate reward R t + 1 R_{t+1}obtained by leaving state st.$R_{t+1}$, the immediate reward is only related to the state and has nothing to do with the behavior taken in that state; the other part is the new state behavior pair $<s_{t+1},a_{t+1}>$ Q value: The environment gives the individual a new state$s_{t+1}$, observed at $s_{t+1}$The state is the behavior obtained based on the current strategy $a_{t+1}$Q at $Q(s_{t+1},a_{t+1})$ , the subsequent Q value considers the attenuation coefficient. When$n=When\; 2$ , use the immediate reward of 2 steps forward, and then replace it with the Q value of the new state; if$n=\infty$ , which means that the Q value is calculated with the immediate reward until the end of the Episode, the individual enters the termination state and obtains the immediate reward of the termination state.
In addition, this definition formula does not reflect the concept of state-behavior pairs, and it is easy to beconfused with the previous n-step G harvest. In fact, Q itself includes behavior, that is, the behavior generated based on a certain state under the current strategy. There is a certain relationship between Q harvest and G harvest, which can be understood in conjunction with the Bellman equation.

With the above definition, n-step Sarsa can be expressed as n-step Q harvest, as follows:
$$Q\left(S_t,A_t\right)\leftarrow Q\left(S_t,A_t\right)+a\left(q_t^{(n)}-Q\left(S_t,A_t\right)\right)$$

SARSA( $l$ )

Define the q return at this time:
$$q_t^{}=(1-l)\sum _{n=1}^{\infty }{l}^{n-1}{q}_t^{\left(n\right)}$$
更新公式为：
$$Q\left(S_t,A_t\right)\leftarrow Q\left(S_t,A_t\right)+a\left(q_t^{}-Q\left(S_t,A_t\right)\right)$$
Forward understanding: using it to update the Q value requires traversing the complete Episode. As shown in the figure below:

The method of reverse understanding: the same as the reverse understanding of TD(λ), the concept of Eligibility Trace is introduced. The difference is that this time the E value is not for a state, but for a state. Behavior pair:
$$\begin{array}{l}{E}_0(s,a)=0\\ E_t(s,a)=cl{E}_{t-1}(s,a)+1\left(S_t=s,A_t=a\right)\end{array}$$
It reflects the causal relationship between a result and a certain state-behavior pair, the state-behavior pair closest to the result, and those state-behavior pairs that occurred frequently before that have the greatest impact on the result.
The following formula introduces $E_t$Concept of SARS( $\lambda$)之后的Q值更新描述：
$$\begin{array}{l}{d}_t=R_{t+1}+\gamma Q\left(S_{t+1},A_{t+1}\right)-Q\left(S_t,A_t\right)\\ Q(s,a)\leftarrow Q(s,a)+a{d}_t{E}_t(s,a)\end{array}$$
Introduce $E_t$concept, while using SARS( $\lambda$ ) will enable more effective online learning because it is not necessary to learn the complete Episode and the data can be discarded after it is used up. $E_t$It is usually more used in online learning algorithms.
SARSA( λ \lambdaThe pseudocode of the$\lambda$

) algorithm is as follows: What should be mentioned here is that E(s,a) needs to be reset to 0 after each episode is browsed, which reflects the$E_t$Only works in one Episode. And here, all (s, a) that have been experienced before are updated at once, while SARSA only updates the value function of the current state-behavior pair.

off-policy

The characteristic of On-Policy Learning is that the currently followed strategy is the strategy for individual learning improvement. Off-Policy Learning refers to following a policy $\mu (a|s)$ while evaluating another policy$\pi (a|s)$ , that is, calculate and determine the state value function v π ( s ) v_\pi(s)under this other strategy$v_{}\left(s\right)$ sum$q_{}(s,a)$ . Why do you do that? Because in this way it is easier to learn from human experience or the experience of other individuals, you can also learn from some old strategies, and you can compare the advantages and disadvantages of the two strategies. Perhaps the main reason is to follow an exploratory strategy and optimize the existing strategy. It can also be divided into Monte Carlo-based and TD-based based on whether it has experienced a complete Episode.

The importance of off-policy is:
1. You can learn by observing other agents;
2. You can use previous experience or old strategies;
3. Learn the optimal strategy through exploratory strategies;
4. When following a strategy You can learn a variety of strategies while playing.

important sampling

Before using off-policy, you need to know this importance sampling:
$$\begin{array}{rl}{E}_{X\sim P}[f(X)]& =\sum P(X)f(X)\\ & =\sum Q(X)\frac{P(X)}{Q\left(X\right)}f(X)\\ & =E_{X\sim Q}\left[\frac{P\left(X\right)}{Q\left(X\right)}f(X)\right]\end{array}$$
That is, using a different distribution to estimate expectations.

For the MC method, we generally do not use the off-policy strategy because the variance is too large and we need to sample until the end of the round to compare the differences between the two strategies. Based on Li Hongyi’s explanation of importance sampling, it can be seen that two distributions with excessive variance are not suitable for importance sampling.

TD’s off-policy method

The task of offline policy TD learning is to use the TD method to follow a policy $\mu (a|s)$ while evaluating another policy$\pi (a|s)$。具体数学表示为：
$$\begin{array}{rl}V\left(S_t\right)& \leftarrow V\left(S_t\right)+a\left(\frac{Pi\left(A_t|S_t\right)}{m\left(A_t|S_t\right)}\left(R_{t+1}+\gamma V\left(S_{t+1}\right)\right)-V\left(S_t\right)\right)\end{array}$$
Note: The V function here is for the strategy $\pi$ -like!
This formula can be explained like this: the individual is in state $S_t$, based on strategy $\mu$ produces an action$A_t$, after executing this behavior, enter the new state $S_{t+1}$, so how to adjust the value of the original state according to the value of the new state under the current strategy? The method of offline strategy is that in state $S_t$When compared respectively according to another strategy $\pi$ and the currently followed strategy$\mu$ produces behavior$A_t$The probability size, if the strategy π \piThe probability value obtained by$\pi$ is the same as following the current strategy μ \muThe probability value obtained by$\mu$$S_{t+1}$Value to update $S_t$The value of is supported by two strategies at the same time, and this update operation is more convincing. It also shows that in the state $S_t$When , the two strategies have close probability selection behavior $A_t$. If this probability ratio is small, it means that if A t A_t is chosen according to the evaluated strategy$A_t$The chance is very small, at this time we are updating $S_t$When considering the value, we should not consider too much the state S t + 1 S_{t+1} obtained based on the current strategy.$S_{t+1}$the value of.Similarly, when the probability ratio is greater than 1, it means that more people still believe in the evaluated strategy. This amounts to drawing on the experience of the strategy being evaluated to update our own strategy.

Q-learning

The key point is that when updating the Q-value of a state-behavior pair, it is not the Q-value of the next state-behavior pair currently following the policy that is used, but the Q-value of the next state-behavior pair generated by the adopted strategy to be evaluated. The formula is as follows:
$$Q\left(S_t,A_t\right)\leftarrow Q\left(S_t,A_t\right)+a\left(R_{t+1}+\gamma Q\left(S_{t+1},A^{\prime }\right)-Q\left(S_t,A_t\right)\right)$$
where the action$A_{t+1}$By strategy $\mu$ to produce, target action$A^{\prime }$ By strategy$\pi$ is generated. There is no need to use importance sampling when using Q-learning, because the Q value is used at this time, not the V value.
The main form of Q learning is: the strategy followed by the individual is based on the current state behavior value function$Q(s,a)$ of a$ϵ-greedy$ strategy, and the target strategy is based on the current state behavior value function$Q(s,a)$ Does not contain$ϵ$ 's simple$greedy$ policy (target policy$\pi$）
$$Pi\left(S_{t+1}\right)=\underset{a^{\prime }}{\mathrm{argmax}}Q\left(S_{t+1},a^{\prime }\right)$$
In this way, the TD target value of Q learning can be greatly simplified:
$$\begin{array}{rl}& R_{t+1}+\gamma Q\left(S_{t+1},A^{\prime }\right)\\ =& R_{t+1}+\gamma Q\left(S_{t+1},\underset{a^{\prime }}{\mathrm{argmax}}Q\left(S_{t+1},a^{\prime }\right)\right)\\ =& R_{t+1}+\underset{a^{\prime }}{\max }\gamma Q\left(S_{t+1},a^{\prime }\right)\end{array}$$
The update formula of the Q value can be written in the following form:
In this way, in the state $S_t$依据$ϵ-greedy$ follows the policy and gets the behavior$A_t$The value of Q will be towards $S_{t+1}$The direction of the maximum Q value of the state is updated at a certain proportion. This algorithm can make $greedy$ strategy$\pi$ eventually converges to the optimal strategy. Since the individual actually follows$ϵ-greedy$ strategy, which ensures that sufficiently rich new states are experienced.

The update formula of the Q function is as follows:
$$Q(S,A)\leftarrow Q(S,A)+a\left(R+c\underset{a^{\prime }}{\max }Q\left(S^{\prime },a^{\prime }\right)-Q(S,A)\right)$$
The algorithm pseudocode of Q-learning is as follows:

Theorem: Q-learning will converge to the optimal state behavior value function:$Q(s,a)\to q_{\ast }(s,a)$。

Summarize the relationship between DP and TD

The following two figures summarize various DP algorithms and various TD algorithms, and also reveal the differences and connections between various algorithms. In general, TD is sampling + data bootstrap, and DP is full width + actual data . If we look at it from the perspective of the Bellman expectation equation: iterative policy evaluation (DP) and TD learning focus on the value of the state itself, and Q-policy iteration (DP) and SARSA focus on the value function of state behavior; if we look at the state From the perspective of the Bellman optimization equation of the behavioral value function, it is Q-value iteration (DP) and Q learning.