1. Reinforcement learning---Markov decision process

Markov process

The known past process is:
ht = { s 1 , s 2 , s 3 , . . . . . st } h_t = \{s_1,s_2,s_3,...s_t\}ht​={ s1​,s2​,s3​,.....st​}
The state with Markov properties has the following properties:
$$\begin{array}{rl}p(s_{t+1}|s_t)& =p(s_{t+1}|h_t)\\ p(s_{t+1}|s_t,a_t)& =p(s_{t+1}|h_t,a_t)\end{array}$$
The state transition matrix is:

Markov Reward Process (MRP)

MRP is Markov chain + reward.
MRP is defined as:

1. S is a set of finite states;
2. P is a dynamic transition probability model $P(S_{t+1}=s^{\prime }|s_t=s)$
3. R is a reward function $R(s_t=s)=E[r_t|s_t=s]$
4. Discount factor $c\in [0,1]$

If the state is finite, R is a vector

MRP example:

about $r_t$and RRUnderstanding$R$ : First see$R$ is for$t$ seeks expectation, so it is a function about state and has nothing to do with time. Sorr$r$ is a random process,$r_t$is a random variable, what we usually call reward refers to $R$。

value function

The definition of return (return): the discounted cumulative reward from time t to the end of one epsiode, expressed as $G_t$来表示：
$$G_t=R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+.....+c^{T-t-1}{R}_T$$
Note: $R_t$is a random variable, $R$ is not!
The definition of the value function for the MRP process is:
$$\begin{array}{rl}{V}_t(s)& =E[G_t|s_t=s]\\ & =E[R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+.....+c^{T-t-1}{R}_T|s_t=s]\end{array}$$
It can be seen from the expression that $G_t$is a random variable, $V_t\left(s\right)$ is a relation about$t$ and$The\; binary\; function\; of\; s$ represents the current moment$tCurrent$ statusssG t G_tunder$s$$G_t$The expectation is a scalar, so it will change with time and state. The size of the value function also reflects the size of the expected reward that can be obtained in the current state at the current point in time. (This expectation is for $G_t$The distribution of this random variable is integrated. )

MRP example: As

can be seen from the above example, since $G_t$is about $The\; random\; variable\; of\; t$ , so at different points in time, the return from the same state (return)$G_t$are different, and will vary with the length of the time step and $The\; size\; of\; \gamma$ changes greatly.

Bellman equation of MRP:

The following recursive formula can be obtained through the definition of the value function:
$$V(s)=R(s)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s)V(s^{\prime })$$
Proof: First introduce a lemma: the expected sum equals the expected sum.
$$\begin{array}{rl}And[\_+Y|S]& =\iint (x+y)f(x,y|s)dxdy\\ & =\iint xf(x,y|s)dxdy+\iint yf(x,y|s)dxdy\\ & =\int x{f}_X(x|s)dx+\int y{f}_Y(y|s)dy\\ & =E[X|S]+E[Y|S]\end{array}$$
所以
$$\begin{gathered} V\left(s\right)=E[R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+.....+c^{T-t-1}{R}_T|s_t=s] \\ =E[R_{t+1}|s_t=s]+\gamma \mathbb{E}[G_{t+1}|s_t=s] \end{gathered}$$
It can be known from the above that$G_t$is given by $R_t$It is formed by summing up. From the definition formula, we know $G_{t+1}$There is no Markovianity here. ( $G_t=R_{t+1}+\gamma R_{t+2}....$)
故
$$E[G_{t+1}|s_t=s]=E[G_{t+1}]=\sum P(s^{\prime }|s)E[G_{t+1}|s_{t+1}=s^{\prime }]$$

We can also write the recursion in matrix form:

the V vector can be obtained by solving the above equation, but because the complexity is too high, this method is generally not used.

Iterative algorithm to find the value function of MRP

A. Monte Carlo algorithm

(The meaning of t here is the value function at the current moment (after iterating N times).)
The MC method uses sampling and averaging to replace the expectation with the expected unbiased estimate average.

B. Iterative solution

According to the Bellman equation of MRP, iterate until the value function vector becomes stable.

Markov Decision Process (MDP)

1. $S$ is a set of finite states.
2. $A$ is a set of finite actions.
3. $P^a$ is a transfer model =$P(s_{t+1}=s^{\prime }|s_t=s,a_t=a)$

MDP function(S,A,P,R, $\gamma$ ) composition.
In the MDP process, R is not only related to the state but also to the actions taken.

Policy in MDP

Policy is the distribution of actions in a given state.
Policy: $\pi (a|s)=P(a_t=a|s_t=s)$

According to the Policy, MDP(S,A,P,R, $c$ ) 和policy$\pi$ ) and the MRP process (S,$P^{\pi }$ ,$R^{\pi }$ ,$\gamma$)等价：
$$\begin{gathered} P^{\pi }(s^{\prime }|s)=\sum _{a\in A}\pi (a|s)P(s^{\prime }|s,a\left) \\ {R}^{\pi }\right(s)=\sum _{a\in A}\pi (a|s)R(s,a) \end{gathered}$$
Schematic diagram comparing the MP/MRP process and the MDP process:

MDP will also go through one more step of sampling actions on the action distribution to determine the transition probability of the next state.

MDP value function

Value function (state-value) in MDP $v^{\pi }\left(s\right)$represents the state s, and the policy is$The\; expected\; return\; of\; \pi$ . The action-value function is$q^{\pi }(s,a)$。
$$\begin{gathered} v^{\pi }\left(s\right)=E[G_t|s_t=s] \\ {q}^{\pi }(s,a)=E[G_t|s_t=s,A_t=a] \end{gathered}$$
$v^{\pi }\left(s\right)$和$q^{\pi }(s,a)$ the relationship between:
$$\begin{gathered} v^{\pi }\left(s\right)=\sum _{a\in A}\pi (a|s\left)q^{\pi }\right(s,a\left) \\ {q}^{\pi }\right(s,a)=R_a^s+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime }) \end{gathered}$$

Bellman expectation equation

$$\begin{gathered} v^{\pi }\left(s\right)=E_{}[R_{t+1}+v{\_}^{\pi }(s_{t+1})|s_t=s\left] \\ {q}^{\pi }\right(s,a)=E_{}[R_{t+1}+\gamma q^{\pi }(s_{t+1},A_{t+1})|s_t=s,A_t=a] \end{gathered}$$
According to the previous Bellman equation of MRP, the Bellman equation of MDP can be easily obtained:
$$\begin{gathered} v^{\pi }\left(s\right)=\sum _{a\in A}\pi (a|s\left)\right(R(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)v^{\pi }\right(s^{\prime })) \\ {q}^{\pi }(s,a)=R(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)\sum _{a^{\prime }\in A}\pi \right(a^{\prime }|s^{\prime })q^{\pi }(s^{\prime },a^{\prime }) \end{gathered}$$
Then intuitively understand the meaning expressed by the following equations:


The codes for the two exercises in the figure below are attached:

#练习1
S = list(range(7))
V1 = np.array([0,0,0,0,0,0,0])
V = np.array([999,999,999,999,999,999,999])
R = np.array([5,0,0,0,0,0,10])
epsilon = 10
gamma = 0.5
pro=np.array([[0,0,0,0,0,0,0],[1,0,0,0,0,0,0],[0,1,0,0,0,0,0],[0,0,1,0,0,0,0],
              [0,0,0,1,0,0,0],[0,0,0,0,1,0,0],[0,0,0,0,0,1,0]])
#terminal_state = 0

while np.abs(V1.sum()-V.sum())>0.0001:
    V = V1
    for s in S:
        all_sum = 0
        for s_ in S:
            all_sum = all_sum + pro[s][s_]*V[s_]
        V1[s] = R[s] + gamma*all_sum
print(V)

#练习2
S = list(range(7))
V1 = np.array([0,0,0,0,0,0,0])
V = np.array([999,999,999,999,999,999,999])
R = np.array([5,0,0,0,0,0,10])
epsilon = 10
gamma = 0.5
pro=np.array([[0.5,0.5,0,0,0,0,0],[0.5,0,0.5,0,0,0,0],[0,0.5,0,0.5,0,0,0],[0,0.5,0,0.5,0,0,0],
              [0,0,0,0.5,0,0.5,0],[0,0,0,0,0.5,0,0.5],[0,0,0,0,0,0.5,0.5]])
while np.abs(V1.sum()-V.sum())>0.0001:
    V = V1
    for s in S:
        all_sum = 0
        for s_ in S:
            all_sum = all_sum + pro[s][s_]*V[s_]
        V1[s] = R[s] + gamma*all_sum
print(V)

optimal value function

The optimal value function and action value function refer to traversing all policy selections to make the value function $v_{}\left(s\right)$ flow$q_{}(s,a)$ Maximum strategy and use the maximum value as the optimal value function.
$$\begin{gathered} v_{\ast }(s)=\underset{Pi}{\max }{v}_{}\left(s\right) \\ {q}_{\ast }(s,a)=\underset{Pi}{\max }{q}_{}(s,a) \end{gathered}$$
The optimal value function shows the possible optimal performance of MDP.

optimal policy

Definition: If for any state $v_{}\left(s\right)>=v_{{Pi}^{\prime }}\left(s\right)$ Then there is,$Pi>={Pi}^{\prime }$
Theorem: For any MDP, the following properties exist:

• There must be an optimal strategy ${Pi}_{\ast }$
• The optimal strategy must be able to produce the optimal value function $v_{{Pi}_{\ast }}\left(s\right)=v_{\ast }\left(s\right)$
• The optimal strategy must be able to produce the optimal action value function $q_{{Pi}_{\ast }}(s,a)=q_{\ast }(s,a)$

The optimal policy can be obtained by maximizing the action value function:
$${Pi}_{\ast }(a|s)=\{\begin{array}{ll}1& \text{ if }a=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{q}_{\ast }(s,a)\\ 0& \text{ otherwise }\end{array}$$
Note: There must be an optimal and decisive policy for any MDP.

Prediction and control in MDP problems

1、prediction：

• 输入：MDP$<S,A,P,R,c>$ and ploicy$\pi variance$ <$<S,P^{Pi},R^{Pi},c>$
• Output: Value function $v^{Pi}$

2、control：

• 输入：MDP$<S,A,P,R,c>$
• Output: optimal value function $v^{\ast }$ and the optimal policy${Pi}^{\ast }$

Both of the above problems can be solved using dynamic programming. Because the original problem can be recursively decomposed into multiple sub-problems, if the global optimum is achieved, it will also be optimal on any sub-problem.