Reinforcement Learning: The Bellman Optimal Formula

Strategy Improvement Case

  The purpose of reinforcement learning is to find the optimal policy. It involves two core concepts, optimal state value and optimal strategy, and a tool: Bellman optimal formula.
  First, we give a familiar example of how the Bellman equation improves a policy.

According to the given strategy, we can easily get the Bellman equation as follows:
$$v_{}(s_1)=-1+v{\_}_{}\left(s_2\right)$$$$v_{}\left(s_2\right)=+1+v{\_}_{}\left(s_4\right)$$$$v_{}\left(s_3\right)=+1+v{\_}_{}\left(s_4\right)$$$$v_{}\left(s_4\right)=+1+v{\_}_{}\left(s_4\right)$$
当$c=0.9$时，求得
$$v_{}(s_4)=v_{}\left(s_3\right)=v_{}\left(s_2\right)=10v_{}\left(s_1\right)=8$$
  Once you know the state value, you can get the action value, and use the state$s_1$As an example, the result is as follows:
$$q_{}(s_1,a_1)=-1+v{\_}_{}\left(s_1\right)=6.2$$$$q_{}(s_1,a_2)=-1+v{\_}_{}\left(s_2\right)=8$$$$q_{}(s_1,a_3)=0+v{\_}_{}\left(s_3\right)=9$$$$q_{}(s_1,a_4)=-1+v{\_}_{}\left(s_1\right)=6.2$$$$q_{}(s_1,a_5)=0+v{\_}_{}\left(s_1\right)=7.2$$
  Through the above calculations and intuitive feelings, we know that this strategy is not good, so how should we improve the strategy when the strategy is not good? This depends on the action value. We can express the current strategy in mathematical form as follows:
$$\pi (a_1|s_1)=\{\begin{array}{c}1a=a_2\\ 0a\ne a_2\end{array}$$

  By calculating the previous state $s_1$Taking the action value obtained as an example, we can know that $q_{}(s_1,a_3)$ is the largest, so we can consider choosing$a_3$As a new strategy, the new strategy is expressed as follows:
$${Pi}_{new}(a|s_1)=\{\begin{array}{c}1a=a^{\ast }\\ 0a\ne a^{\ast }\end{array}$$
$a=a^{The\; \ast }$ period probability is 1, indicating that the new strategy will definitely choose$a^{\ast }$ , and in this example$a^{\ast }=ma{x}_a{q}_{}(s_1,a)=a_3$

Bellman Optimality Equation: Definition

  Through the calculation of the previous example, we can now give the definition of the optimal strategy, which is defined as follows:
$$if{v}_{{Pi}^{\ast }}(s)\ge v_{}\left(s\right)foralls\in SThen\; it\; means{\pi }^{\ast }is\; the\; optimal\; strategy$$

最优策略这一定义引发了许多问题：
	最优策略存在吗？
	最优策略唯一吗？
	最优策略是随机的还是确定性的？
	如何获得最优策略？

To answer these questions, we need to study the Bellman optimality formula.

  The second formula in the above figure is the Bellman optimal formula. Compared with the Bellman formula, we can find that the Bellman optimal formula is the Bellman formula under the optimal strategy condition. The deformation form and vector form of Bellman are given below, as shown in the figure below:

  We can find that there are two unknowns $vand\pi$ , how to solve an equation with two unknowns? We show in the following formula that it can be solved.


  We know from the above two examples that if$q(s,a)$ know, then the maximum value is equal to$max(q(s,a))$ ，当 $a=a^{\ast }=ma{x}_{a}q(s,a)$ , its mathematical expression and conditions are as follows:

Bellman Optimality Equation: Solving

  When solving the Bellman optimality equation, we introduce $f\left(v\right)$ , its form is as follows:

  Before solving the above equation, we introduce several concepts:fixed point, contraction map

$$Known\; functionf\left(x\right),Ifx\; exists,such\; thatf\left(x\right)=xthen\; point\; (x,f(x))isa\; fixed\; point\; of\; functionf(x)$$$$Let(X,d_X)and(Y,d_Y)is\; a\; metric\; space,f：X\to Yis\; the\; mapping.If\; there\; is\; a\; constantk\in [0,1)such\; that||d_Y-d_X||\le \gamma ||X-Y||Thenf\; iscalled\; the\; compression\; map,and\; kis\; called\; the\; compression\; coefficient$$
  . With the above two concepts, now we introduce the Banach fixed point theorem (Contraction Mapping Theorem), which is an important theorem in mathematical analysis. It asserts that a compressive map in a complete metric space must have a unique fixed point.

According to the Banach fixed point theorem, the following properties can be obtained:
as long as $x=An\; equation\; of\; the\; form\; f\left(x\right)$ if$f$ is a compression map, then
  1.$There\; must\; be\; a\; fixed\; point{x}^{\ast },such\; thatf(x^{\ast })=x^{\ast }$
  $2.Fixed\; point{x}^{\ast }is\; the\; only\; one\; that\; exists$
  $3.The\; fixed\; point\; can\; be\; obtained\; by\; iteration:x_{k+1}=f(x_k),x_k\approx x^{\ast },\; whenktendsto\infty ;$

  Now, we can use the above Banach fixed point theorem to solve the Bellman optimality equation, before solving, we have to prove that $f\left(v\right)$ is a compression map, but here we only give it to the application, friends who are interested in the proof process can go and see for themselves. From the Banach fixed point theorem, we can know that there must be a unique$v^{\ast }$ , and can be obtained by iteration.

Bellman optimal formula solution example

  We use a simple example to deepen our understanding of the process of solving the Bellman equation. As shown in the figure below, the robot has three behaviors: left $a_l$, to the right $a_r$, in situ $a_0$, the reward is set to $r_{}=+1$ ,$r_{}=-1$

  According to the rules formulated above, we can get the robot's$q(s,a)$

The question now is how to find the optimal state value$v^{\ast }(s_i)$ and optimal policy${Pi}^{\ast }$
Using the method introduced above, take$c=0.9$ , when$k=When\; 0$ , we first give an initial value at random

.

It can be seen that we have found the optimal strategy.

Properties of Optimal Strategies

  What factors do we think about that affect the optimal strategy? According to the following formula, the factors affecting the optimal strategy are $r,\gamma ,\; and\; the\; model\; environment$