3. Reinforcement learning--model free decision-making

introduce

We understand how to solve a known MDP theoretically: evaluate a given strategy through dynamic programming, and obtain the optimal value function, and determine the optimal strategy based on the optimal value function; it can also be done directly without any The state value of the strategy is iterated to obtain the optimal value function and optimal strategy.
This lecture discusses solving a problem that can be considered an MDP but does not grasp the specific details of the MDP. That is, it describes how to obtain an estimated optimal value function and optimal strategy directly from the interaction between the Agent and the environment. This part is also divided into two parts. The first part focuses on strategy evaluation, that is, prediction. To put it bluntly, it is the estimation of the final reward that the Agent will get when the details of the MDP are not known for a given strategy.

Monte Carlo algorithm

The Monte Carlo algorithm refers to: without knowing the MDP state transition and immediate rewards, learning the state value directly from the complete Episode. Usually the value of a state is equal to all the values ​​calculated based on the state in multiple Episodes. Harvest average.
Note: Harvest is not specific to Episode, it exists within Episode and is specific to a certain state in Episode. The sum of the decaying immediate rewards obtained after going through the Episode starting from this state. From an Episode, we can get the harvest of all states in the Episode. When a state appears multiple times in an Episode, the harvest of that state has different calculation methods.
A complete Episode means that it must start from a certain state, the Agent interacts with the Environment until the termination state, and the environment gives immediate harvest of the termination state. A complete Episode does not require that the starting state must be a specific state, but requires that the individual eventually enters a certain terminal state recognized by the environment.
The Monte Carlo algorithm has the following characteristics: it is not based on the model itself, but learns directly from the experienced Episodes. It must be a complete Episode. The idea used is to replace the value with the average harvest value. Theoretically, the more episodes there are, the more accurate the results will be. It uses the idea of ​​the law of large numbers, which is to replace expectations with average values, which are unbiased estimates of expectations.

Monte Carlo strategy evaluation

Goal: Under a given strategy, learn the state value function under the strategy from a series of complete Episode experiences.
The mathematical description is as follows:
Based on a specific strategy $An\; Episode\; information\; of\; \pi$ can be expressed as the following sequence:
$$S_1,A_1,R_2,S_2,A_2,\dots ,S_t,A_t,R_{t+1},\dots ,S_k\sim \pi$$
t time state$S_t$ 的收获：
$$G_t=R_{t+1}+\gamma R_{t+2}+\dots +c^{T-1}{R}_T$$
T indicates state termination.
The value of a certain state s under this strategy:
$$v_{}(s)=E_{}\left[G_t|S_t=s\right]$$
Many times, instant rewards only appear at the end of the Episode, but it cannot be denied that there may be instant rewards in the intermediate state. R t R_tin the formula$R_t$It refers to the immediate reward obtained in any state, which should be paid special attention to.
During the state transfer process, it may happen that a state returns to that state one or more times after a certain transition. At this time, how to calculate the number of times this state occurs in an Episode and calculate the harvest of the Episode? There are two methods available:

First access to Monte Carlo strategy evaluation

When a given policy is used to evaluate a certain state s using a series of complete Episodes, for each Episode, only the first occurrence of the state is included in the calculation:

Monte Carlo strategy evaluation per visit

Given a policy and using a series of complete episodes to evaluate a certain state s, for each episode, each time the state s appears in the state transition chain, the specific calculation formula is the same as above, but the specific meaning is different.

Updated after each episode visit

Mentioned here is a method commonly used in actual operations to update the mean in real time, so that when calculating the average harvest, there is no need to store all previous harvests. Instead, every time a harvest is obtained, the average harvest is calculated.
The formula is as follows:
$$\begin{array}{rl}{m}_k& =\frac{1}{k}\sum _{j=1}{x}_j\\ & =\frac{1}{k}\left(x_k+\sum _{j=1}^{k-1}{x}_j\right)\\ & =\frac{1}{k}\left(x_k+(k-1)m_{k-1}\right)\\ & =m_{k-1}+\frac{1}{k}\left(x_k-m_{k-1}\right)\end{array}$$
where $m_k$is the average value of the kth time, $x_k$is the kth element.
This formula is relatively simple. Applying this method to Monte Carlo strategy evaluation results in the following Monte Carlo real-time updates.

Update algorithm after each visit to episode

For each of a series of Episodes: $S_1,A_1,R_2,S_2,A_2,\dots ,S_t,A_t,R_{t+1},\dots ,S_k$
For each state S t S_t in Episode$S_t$, there is a gain $G_t$, every time $S_t$, use the following formula to calculate the average value of the state $V(S_t)$ ：
$$\begin{array}{l}N\left(S_t\right)\leftarrow N\left(S_t\right)+1\\ V\left(S_t\right)\leftarrow V\left(S_t\right)+\frac{1}{N\left(S_t\right)}\left(G_t-V\left(S_t\right)\right)\end{array}$$
When dealing with non-static problems, it is very useful to use this method to track a real-time updated average, and to discard the episode information that has been calculated. At this time, the parameter α \alpha can be introduced$\alpha$来更新状态价值：
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a\left(G_t-V\left(S_t\right)\right)$$
The above is the main idea and description of the Monte Carlo learning method. Since the Monte Carlo learning method has many shortcomings (will be discussed in detail later), there are not many practical applications. Next, we introduce the commonly used TD learning methods.

Temporal difference learning (TD-learning)

introduce

Temporal difference learning is referred to as TD learning. Its characteristics are as follows: Like Monte Carlo learning, it also learns from Episodes and does not need to know the model itself; but it can learn incomplete Episodes and guess the Episodes through self-sampling (bootstrapping). As a result, this speculation will continue to be updated at the same time. That is, the estimated value function is used to update the value function.
We have already learned, in Monte-Carlo learning, to use the actual return $G_t$来更新价值（Value）：
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a\left(G_t-V\left(S_t\right)\right)$$
In TD learning, when the algorithm estimates the value of a certain state, it uses the immediate reward$R_{t+1}$And the next state $V_{t+1}$The estimated state value is multiplied by the attenuation coefficient $\gamma$组成，这符合Bellman方程的描述：
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_t\right)\right)$$ For example: I go from point A to point C through point B and end at point C. I originally couldn't update the value function of point A at point B. I can only update it when I go to point C. Now that I am at point B, it is estimated that For rewards from B to C, the value function of point A can be updated immediately at point B. $R_{t+1}+\gamma V\left(S_{t+1}\right)$:称作TD估计值，$d_t=R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_t\right)$ is called TD error.
BootStrappingrefers to the TD target value$R_{t+1}+\gamma V\left(S_{t+1}\right)$ instead of cumulative rewards.

BootStrapping

Bootstrapping literally translates as the bootstrap method, also known as the bootstrapping method. It is a method of estimating the population by resampling samples.
https://zhuanlan.zhihu.com/p/54201828 See this post on Zhihu for details.

MC and TD comparison example

Imagine you are driving home after get off work and need to estimate how long the entire trip will take. Suppose a person suddenly encounters a dangerous situation while driving home: a car coming from the opposite direction feels like it will collide with you. In serious cases, he may face death threats, but in the end both parties take measures and no collision actually occurs. If Monte Carlo learning is used, the negative rewards that may be triggered by this dangerous situation on the road will not be taken into account and will not affect the total prediction time; but in TD learning, if such a dangerous situation is encountered, the person will immediately Updating the value of this state, and later discovering that this is worse than the previous state, will immediately consider the decision to slow down and gain time, which means you don't have to update the value of the state until after he dies like Monte Carlo learning, in that case There is also no way to update the status value. The TD algorithm is equivalent to constantly updating the time it takes to finally return home based on the time that has been consumed and the time that is expected to be needed throughout the entire process of returning home (an Episode).

Based on the data shown in the table above, the figure below shows two different learning strategies, Monte Carlo learning and TD learning, to update the value function (the value of each state). What is used here is the estimated time it takes to get home from a certain state to indirectly reflect the value of a certain state: the longer the estimated time it takes to get home from a certain location, the lower the value of the location, and it is necessary to avoid entering this state when optimizing decisions. For the Monte Carlo learning process, when the driver encounters various situations on the road, he will not update his estimated time to return home. When he returns home and obtains the actual time it takes to return home, he will re-estimate the time it takes to return home. On the way home, check the time when each major node status arrives home, and use the new estimated time to help make decisions when you return home next time; for TD learning, when you first leave the office, you may estimate that the total time will take 30 minutes. , but when you pick up the car and find it raining, you will immediately think that the original estimate was too optimistic, because past experience tells you that rain will extend the total time you need to return home. At this time, you will update the current state value estimate. , increased from the original 30 minutes to 40 minutes. Similarly, when you drive off the highway, the estimated remaining time to return home will be based on the current status (location, road conditions, etc.) until you return home to get the actual total time to return home. During this process, you will update the value of the state in real time based on changes in the state.

Comparison between MC and TD algorithms

The figure below compares the efficiency of MC and TD algorithms. The abscissa is the number of episodes experienced, and the ordinate is the mean square error of the value of each state under the calculated state function and the actual state function. The black color is the learning curve of the MC algorithm under different step-sizes, and the gray curve uses the TD algorithm. It can be seen that TD is more efficient than MC.

Comparison of convergence between MC and TD

The MC algorithm will minimize the MSE and bring the distance between the value function and the cumulative reward closer.
$$\sum _{k=1}^K\sum _{t=1}^{T_k}{\left(G_t^k-V\left(s_t^k\right)\right)}^{In\; formula\; 2}$$
, k represents the Episode serial number, K is the total Episode number, t is the state serial number in an Episode (the 1st, 2nd, 3rd... state, etc.),$T_k$It represents the total number of states of the k-th Episode, $G_t^k$Represents the state st s_t at time t in the kth Episode$s_t$The final gain obtained, $V(s_t^k)$ represents the state$s_t$the value of.
The TD algorithm converges to the state value of the largest possible Markov model constructed based on existing experience. That is to say, the TD algorithm will first estimate the transition probability between states based on existing experience, and at the same time estimate the immediate reward of a certain state:
$$\begin{array}{rl}{\hat{\mathcal{P}}}_{s,s^{\prime }}^a& =\frac{1}{N(s,a)}\sum _{k=1}^K\sum _{t=1}^{r_k}1\left(s_t^k,a_t^k,s_{t+1}^k=s,a,s^{\prime }\right)\\ {\hat{\mathcal{R}}}_s^a& =\frac{1}{N(s,a)}\sum _{k=1}^K\sum _{t=1}^{T_k}1\left(s_t^k,a_t^k=s,a\right)r_t^k\end{array}$$
It can be seen from the comparison that the TD algorithm uses the Markov properties of the MDP problem and is more effective in the Markov environment; but the MC algorithm does not use the Markov properties and is usually more effective in the non-Markov environment.

TD($l$ )

The TD algorithms introduced earlier are actually TD(0) algorithms. The number 0 in the parentheses means looking one step ahead in the current state. What will happen if we look two steps ahead to update the state value? This introduces the concept of n-step.

Define n-step harvest:
$$G_t^{(n)}=R_{t+1}+\gamma R_{t+2}+\dots +c^{n-1}{R}_{t+n}+c^{n}V\left(S_{t+n}\right)$$
Then, the update formula of the n-step TD learning state value function is:
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a\left(G_t^{\left(n\right)}-V\left(S_t\right)\right)$$
Since there is n-step prediction, when does n have the best effect?
Here we introduce a new parameter: λ. By introducing this new parameter, predictions of all steps can be comprehensively considered without increasing computational complexity. This is lambda prediction and lambda harvest.

• $\lambda harvest\lambda$ \
  $\lambda$ harvest weights the cumulative reward obtained at each step, and it imposes a certain weight$(1-l)l^n$ . Through such weight design, the following formula is obtained:
  $$G_t^{}=(1-l)\sum _{n=1}^{\infty }{l}^{n-1}{G}_t^{\left(n\right)}$$

TD($\lambda$)更新公式：
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a\left(G_t^{}-V\left(S_t\right)\right)$$

• TD($\lambda$ ) For the distribution of weights,
  
  this picture is relatively easy to understand. For example, for the 3-step harvest of n=3, give it inλ \lambdaThe weight in lambda harvesting is such as the area of ​​the shaded part on the left, and for the T-step harvesting in the terminal state, the area of ​​all shaded parts after$T.$The sum of the areas of all segments is 1. The design of this geometric series also takes into account the computational convenience of algorithm implementation.

TD( λ \lambda Two ways of understanding$\lambda \; )$

forward understanding

After introducing λ, you will find that to update the state value of a state, you must go through the entire Episode to obtain the instant rewards of each state and the instant rewards of the final state. This is the same requirement as the MC algorithm, so the TD(λ) algorithm has the same disadvantages as the MC method. The value range of λ is [0,1]. When λ=1, the corresponding MC algorithm is used. This practical calculation brings inconvenience.

backward understanding

TD(λ) on the other hand provides a single-step update mechanism, which is illustrated by the following example.
The mouse received an electric shock after receiving three consecutive bells and one light signal. So when analyzing the cause of the electric shock, is the ringing factor more important or the light factor being more important?

Frequency heuristic : Attribute the cause to the state that occurs most frequently.
Recency heuristic : Attributing the cause to recent states.
Introduce a value to each state: Eligibility Traces (ES), also translated as "Eligibility Traces", which can be combined with the above two heuristics. Definition:
$$\begin{array}{l}{E}_0(s)=0\\ E_t\left(s\right)=cl{E}_{t-1}\left(s\right)+1\left(S_t=s\right)\end{array}$$
Here $E_t\left(s\right)$ is for a certain state, appears once +1, and will decay over time. EEhere$E$ is not a matrix.

The abscissa of the figure is time, the position with a vertical line under the abscissa represents the current state s, and the ordinate is the utility tracking value E. It can be seen that when a certain state appears continuously, the E value will increase by a unit value based on a certain attenuation. At this time, the proportion of the state's contribution to the final harvest will be increased, so the value of the state can be updated more Consider the impact on the final harvest. At the same time, if the state is farther from the final state, its contribution to the final harvest is smaller, and there is no need to consider the final harvest too much when updating the state.
Backward understanding believes that the state value of the latter state is related to all previous states.

The updated formula based on Eligibility Traces is as follows:
$$\begin{array}{rl}{d}_t& =R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_t\right)\\ V(s)& \leftarrow V(s)+a{d}_t{E}_t(s)\end{array}$$
当$l=When\; 0$ , only the current state is updated, that is:
$$\begin{array}{l}{E}_t\left(s\right)=1\left(S_t=s\right)\\ V\left(s\right)\leftarrow V\left(s\right)+a{d}_t{E}_t\left(s\right)\end{array}$$
This is equivalent to the update of TD(0):
$$V\left(S_t\right)\leftarrow V\left(S_t\right)+a{d}_t$$
当$l=$When$1$
, it is equivalent to the update method of MC. The proof is as follows: Assuming that state s appears for the first time at time-step k, the update method of eligibility traces of TD(1) is as follows:
$$\begin{array}{rl}{E}_t\left(s\right)& =c{E}_{t-1}(s)+1\left(S_t=s\right)\\ & =\{\begin{array}{ll}0& \text{ if }t<k\\ c^{t-k}& \text{ if }t\ge k\end{array}\end{array}$$

Therefore, if we want to prove that TD(1) is equivalent to MC, we only need to prove that the following formula is true:
$$\sum _{t=1}^{T-1}a{d}_t{E}_t\left(s\right)=a\sum _{t=k}^{T-1}{c}^{t-k}{\delta }_t=a\left(G_k-V\left(S_k\right)\right)$$
The proof process is as follows:
It is known that the expression of TD-error is as follows:
$$d_t\doteq R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_t\right)$$
则有：
$$\begin{array}{rl}{G}_t-V\left(S_t\right)& =R_{t+1}+c{G}_{t+1}-V\left(S_t\right)+\gamma V\left(S_{t+1}\right)-\gamma V\left(S_{t+1}\right)\\ & =d_t+c\left(G_{t+1}-V\left(S_{t+1}\right)\right)\\ & =d_t+c{d}_{t+1}+c^2\left(G_{t+2}-V\left(S_{t+2}\right)\right)\\ & =d_t+c{d}_{t+1}+c^2{d}_{t+2}+\cdots +c^{T-t-1}{\delta }_{T-1}+c^{T-t}\left(G_T-V\left(S_T\right)\right)\\ & =d_t+c{d}_{t+1}+c^2{d}_{t+2}+\cdots +c^{T-t-1}{\delta }_{T-1}+c^{T-t}(0-0)\\ & =\sum _{k=t}^{T-1}{c}^{k-t}{\delta }_k\end{array}$$

At this point we have discussed $In\; the\; two\; cases\; where\; \lambda$ is equal to 0 and 1, it is necessary to prove the equivalence, that is, for any$\lambda$ is established, so it is necessary to prove the following theorem:

The update formula of the eligibility trace at this time is as follows:
$$\begin{array}{rl}{E}_t(s)& =cl{E}_{t-1}\left(s\right)+1\left(S_t=s\right)\\ & =\{\begin{array}{ll}0& \text{ if }t<k\\ (cl{)}^{t-k}& \text{ if }t\ge k\end{array}\end{array}$$
即证：
$$\sum _{t=1}^{T}a{d}_t{E}_t\left(s\right)=a\sum _{t=k}^T(cl{)}^{t-k}{\delta }_t=a\left(G_k^{}-V\left(S_k\right)\right)$$
其实本质上和=1的情况基本一样，证明如下：
$$\begin{array}{rl}{G}_t^{}-V\left(S_t\right)=-V\left(S_t\right)& +(1-l)l^0\left(R_{t+1}+\gamma V\left(S_{t+1}\right)\right)\\ & +(1-l)l^1\left(R_{t+1}+\gamma R_{t+2}+c^{2V}\_\left(S_{t+2}\right)\right)\\ & +(1-l)l^2\left(R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+c^{3V}\_\left(S_{t+3}\right)\right)\\ & +\cdots \\ =-V\left(S_t\right)& +(cl{)}^0\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-\gamma \lambda V\left(S_{t+1}\right)\right)\\ & +(cl{)}^1\left(R_{t+2}+\gamma V\left(S_{t+2}\right)-\gamma \lambda V\left(S_{t+2}\right)\right)\\ & +(cl{)}^2\left(R_{t+3}+\gamma V\left(S_{t+3}\right)-\gamma \lambda V\left(S_{t+3}\right)\right)\\ & +\cdots \\ & +(cl{)}^0\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_t\right)\right)\\ & +(cl{)}^2\left(R_{t+2}+\gamma V\left(S_{t+2}\right)-V\left(S_{t+1}\right)\right)\\ & +\cdots \\ =& d_t+cl{d}_{t+1}+(cl{)}^2{d}_{t+2}+\dots +\gamma V\left(S_{t+3}\right)-V\left(S_{t+2}\right))\end{array}$$
The following table shows the relationship between different algorithms in different situations when λ takes various values.

Still don’t understand the difference between offline updates and online updates