[Reinforcement Learning] 03 - Markov Decision Process

Recommend another article here [Autonomous Driving Decision Planning] Introduction to POMDP

1. Markov Decision Process (MDP)

• It provides a mathematical framework for modeling the decision-making process in which the results are partly random and partly under the control of the decision-maker .
• MDP formally describes a reinforcement learning environment
  • The environment is fully observable
  • The current state can completely characterize the process (Markov property)
• Almost all RL problems can be converted to MDP to solve
  • Optimal control mainly deals with continuous MDP
  • Some unsightly problems can be converted into MDP
  • A multi-armed bandit is a single-state MDP

1.1. Markov properties

“The future is independent of the past given the present”

The probability distribution of future states is only related to the current state and has nothing to do with past states .

Definition :

• StatusS $S_t$is Markov's if and only if $$\mathbb{P}\left[S_{t+1}\mid S_t\right]=P\left[S_{t+1}|S_1,...,S_t\right]$$

Properties :

• Status captures all relevant information in history
• Once you know this state, you can abandon history
• That is, the current state is a sufficient statistics of the future

1.2. State transition matrix

$P_{s{s}^{\prime }}$For the slave state $s$ transitions to state$s^{\prime }$ probability, also known as one-step state transition probability. $P$ is a one-step state transition matrix.

$$\begin{array}{c}P[S_{t+1}|S_t]=P[S_{t+1}|S_1,\dots ,S_t]\\ {\mathbf{P}}_{s{s}^{\prime }}=P[S_{t+1}=s^{\prime }|S_t=s]\\ P=\left[\begin{array}{cccc}{P}_{11}& P_{12}& \dots & P_{1n}\\ P_{21}& P_{22}& \dots & P_{2n}\\ \dots & & & \\ \dots & & & \\ P_n& P_{n2}& \dots & P_{nn}\end{array}\right]\end{array}$$

矩阵有以下性质：

1. 非负性性质,$P_{ij}\ge 0$
2. 行元素和为1,$\sum P_{ij}=1,i=1,2,...,n$

以上图为例，$S_1$转移到自身和$S2$的概率分别为0.1，0.9；$S_2$转移到自身和$The\; probabilities\; of\; S1$ are 0.2, 0.8 respectively. The state transition matrix can be expressed as:$$\mathbf{P}=\left[\begin{array}{cc}0.1& 0.9\\ 0.8& 0.2\end{array}\right]$$

1.3. Markov process

Markov process refers to a random process with Markov properties, also known as Markov chain . The Markov chain can be composed of the binary group $(S,P)$ description. The state transition probability does not change with time.

1.3.1. A simple example

$Class1$ is the initial state,$Sleeep$ is the terminal state, and the number in it represents the transition probability between each state. The sum of the probabilities of transitioning from each state to other states is 1.

We can write the state transition matrix of this Markov process:

Given a Markov process, we can start from a certain state and generate a state sequence (episode) based on its state transition matrix . This step is also called sampling . For example, we may get the following sampling results:

• C1 C2 C3 Pass Sleep
• C1 FB FB C1 C2 Sleep
• C1 C2 C3 Pub C2 C3 Pass Sleep
• C1 FB FB C1 C2 C3 Pub C1 FB FB
• FB C1 C2 C3 Pub C2 Sleep

2. Markov reward process

Add the reward function rr based on the Markov process$r$ and the discount factor$\gamma$ , you can get the Markov reward process (Markov reward process). A Markov reward process consists of$⟨\mathcal{S},\mathcal{P},\mathcal{R},\gamma 〉$ consists of, the meaning of each component element is as follows.

• $S$ : state set,S = { s 1 , s 2 , . . . , sn } S=\{s_1, s_2, ..., s_n\}S={ s1​,s2​,...,sn​}
• $P(s^{\prime }|s)$: state transition probability, expressed in the current state$Under\; s$ , take action$After\; a$ , transfer to state$s^{The\; probability\; of\; \prime }$ ,$s^{\prime }\in S$，$s,s^{\prime }\in S$。${\mathcal{P}}_{s{s}^{\prime }}=P\left[S_{t+1}=s^{\prime }\mid S_t=s\right]$
• $R(s,s^{\prime })$: Immediate reward function, expressed in the current state$Under\; s$ , take action$After\; a$ , transfer to state$s^{\prime }$ The immediate reward obtained,$s,s^{\prime }\in S$。${\mathcal{R}}_s=\mathbb{E}\left[R_{t+1}\mid S_t=s\right]$
• $c\in (0,1)$ : Discount factor, indicating the weight ratio of current rewards and future rewards.
  • if $c\to 1$ , means more belief in the current state, that is, "short-sighted myopic ";
  • if $c\to 0$ , means more belief in the future state, that is, " far-sighted ";

2.1. Return

Definition :
In a Markov reward process, from ttState at time$t$$S_t$From the beginning to the end state, the sum of the decay of all rewards is called return (Return) $G_t$, the formula is as follows:

$$G_t=R_{t+1}+\gamma R_{t+2}+...=\sum _{k=0}^{\infty }{c}^k{R}_{t+k+1}$$

Still using the above example:
for example, enter the state $Class2$ can get reward$-2$ , indicating that we do not want to enter, enter$Pass$ can get the highest reward$10$ , but enter$AfterSleep\; the\; reward\; is\; zero\; andthe$ sequence terminates at this point. A calculation example:$Class1\to Class2\to Class3\to Pass\to Sleep$：$-2+0.5\times (-2)+(0.5{)}^2\times (-2)+(0.5{)}^3\times 10=-2.25$

expressed in code:

import numpy as np

# Define the transition Matrix
P = [
    [0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 0.0],
    [0.0, 0.0, 0.8, 0.0, 0.0, 0.0, 0.2],
    [0.0, 0.0, 0.0, 0.6, 0.4, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
    [0.2, 0.4, 0.4, 0.0, 0.0, 0.0, 0.0],
    [0.1, 0.0, 0.0, 0.0, 0.0, 0.9, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]
]
P = np.array(P)
RewardVector = [-2, -2, -2, 10, 1, -1, 0]

# 给定一条序列,计算从某个索引（起始状态）开始到序列最后（终止状态）得到的回报
def ComputeSequenceReward(Start_idx, Sequence, RewardVector, gamma=0.5):
    TotalReward = 0.0
    for i in reversed(range(Start_idx, len(Sequence))):
        TotalReward = gamma * TotalReward + RewardVector[Sequence[i] - 1]
    return TotalReward

def test01():
    chain = [1, 2, 3, 4, 7]
    start_index = 0
    print("根据本序列计算得到回报为：%s。"% ComputeSequenceReward(start_index, chain, RewardVector, gamma=0.5))
    
if __name__ == "__main__":
    test01()

根据本序列计算得到回报为：-2.25。

$c=$Examples at$0$$,$$0.9$$,$$and\; 1\; (the\; numbers\; in\; the\; picture\; are\; the\; values\; of\; each\; node):$

2.2. Value function

In the Markov reward process, the expected return of a state (that is, the expectation of future cumulative rewards starting from this state) is called the value of this state . The values ​​of all states form a value function. The input of the value function is a certain state, and the output is the value of this state. We write the value function as $v(s)=\mathbb{E}\left[G_t\mid S_t=s\right]$，展开为$$\begin{array}{rl}V(s)& =\mathbb{E}\left[G_t\mid S_t=s\right]\\ & =E\left[R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+...\mid S_t=s\right]\\ & =\mathbb{E}\left[R_{t+1}+c\left(R_{t+2}+\gamma R_{t+3}+...\right)|S_t=s\right]\\ & =E\left[R_{t+1}+c{G}_{t+1}|S_t=s\right]\\ & =E\left[R_{t+1}+\gamma V(S_{t+1})|S_t=s\right]\end{array}$$

In the last equal sign of the above formula, on the one hand, the expectation of immediate reward is exactly the output of the reward function, that is, $E[R_{t+1}|S_t=s]={\mathcal{R}}_s]$ ; On the other
hand, the remaining part of the equation$\mathbb{E}[\gamma V(S_{t+1})|S_t=s]$ can be based on the slave state$The\; transition\; probability\; starting\; from\; s$ is obtained, that is,$$V(s)={\mathcal{R}}_s+c\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}V(s^{\prime })$$

The above formula is the very famous Bellman equation in the Markov reward process , which is true for every state. We can write this in matrix form:
$$\mathcal{V}=R+\gamma \mathcal{P}\mathcal{V}$$$$\left[\begin{array}{c}V(s_1)\\ ⋮\\ V(s_n)\end{array}\right]=\left[\begin{array}{c}{\mathcal{R}}_1\\ ⋮\\ R_n\end{array}\right]+c\left[\begin{array}{ccc}{\mathcal{P}}_{11}& \dots & P_{1n}\\ ⋮& & \\ P_{11}& \dots & P_{nn}\end{array}\right]\left[\begin{array}{c}V\left(s_1\right)\\ ⋮\\ V\left(s_n\right)\end{array}\right]$$
We can directly solve it based on matrix operations and get the following analytical solution: $$\begin{array}{rl}\mathcal{V}& =\mathcal{R}+\gamma \mathcal{P}\mathcal{V}\\ (I-\gamma P)V& =R\\ \mathcal{V}& =(I-\gamma \mathcal{P}{)}^{-1}\mathcal{R}\end{array}$$

The computational complexity of the above analytical solution is $O\left(n^3\right)$, where$n$ is the number of states, so this method is only suitable for small Markov reward processes. When solving the value function in a larger-scale Markov reward process, you can usedynamic programmingalgorithms,methods(see 3.6) andtemporal differences(temporal difference)

Next, write code to implement the analytical solution method for solving the value function, and calculate the value of all states in the Markov reward process accordingly.

# Exploit Bellman equation to compute value of all states
def ComputeValue(RewardVector, Statesize, TransitionMatrix=P, gamma=0.5):
    RewardVector = np.array(RewardVector).reshape(-1, 1)
    Value = np.dot(np.linalg.inv(np.eye(Statesize, Statesize) - gamma * TransitionMatrix),
                   RewardVector)
    return Value

print("MRP中每个状态价值分别为\n", ComputeValue(RewardVector, 7))

MRP中每个状态价值分别为
 [[-2.90815722]
 [-1.55006913]
 [ 1.12482718]
 [10.        ]
 [ 0.62413589]
 [-2.08255975]
 [ 0.        ]]

3. Markov decision process

3.1. MDP quintuple

MDP (Markov Decision Process) is a mathematical model used to describe the interaction between an agent and the environment. MDP can be expressed as a five-tuple $(S,A,P,R,c)$ :

• $S$ : state set,S = { s 1 , s 2 , . . . , sn } S=\{s_1, s_2, ..., s_n\}S={ s1​,s2​,...,sn​} , including lane, environment, world model and other information.
• $A$ : action set,A = { a 1 , a 2 , . . . , am } A=\{a_1, a_2, ..., a_m\}A={ a1​,a2​,...,am​} , the vehicle’s decision-making space, including lane changing, following, overtaking, etc.
• $P(s^{\prime }|s,a)$ : State transition probability function, expressed in the current state$Under\; s$ , take action$After\; a$ , transfer to state$s^{The\; probability\; of\; \prime }$ ,$s^{\prime }\in S$，$s,s^{\prime }\in S$，$a\in A$。${\mathcal{P}}_{{\mathbf{ss}}^{\prime }}^a=P\left[S_{t+1}=s^{\prime }\mid S_t=s,A_t=a\right]$
• $R(s,a,s^{\prime })$: Immediate reward function, expressed in the current state$Under\; s$ , take action$After\; a$ , transfer to state$s^{\prime }$ The immediate reward obtained,$s,s^{\prime }\in S$，$a\in A$。${\mathcal{R}}_{\mathfrak{s}}^a=\mathbb{E}\left[R_{t+1}\mid S_t=s,A_t=a\right]$
• $c\in (0,1)$ : Discount factor, indicating the weight ratio of current rewards and future rewards.

Note : In the above definition of MDP, we no longer use the state transition matrix method similar to the MRP definition, but directly express it as a state transition function .

• This is done because the state transition is also related to the action at this time and becomes a three-dimensional array instead of a matrix (two-dimensional array);
• The second reason is that the state transition function has more general meaning. For example, if the state set is not finite, it cannot be represented by an array, but it can still be represented by the state transition function.

Unlike a Markov reward process, in a Markov decision process there is usually an agent that performs the action. The Markov decision-making process is a time-related ongoing process, and there is a continuous interaction process between the agent and the environment. Generally speaking, the interaction between them is a cyclic process as shown below: The agent determines the current state $S_t$Select action $A_t$;For state $S_t$and action $A_t$, the environment obtains S t + 1 S_{t+1} based on the reward function and state transition function$S_{t+1}$and $R_{t+1}$and feedback to the agent. The goal of the agent is to maximize the cumulative reward obtained. The function by which an agent selects an action from a set of actions based on the current state is called a policy.

3.2. Strategy

The agent's policy is usually represented by the letters $\pi$ represents. In input statessTake action aa$under\; s$ situation$The\; strategy\; for\; the\; probability\; of\; a$ is shown in the following formula. $$\pi (a|s)=\mathbb{P}\left[A_t=a\mid S_t=s\right]$$

• The strategy only needs to be related to the current state and does not need to consider the historical state.
• The policy is fixed (independent of time) $A_t\sim \pi (\cdot |S_t),\forall t>0$

3.3. Value function

3.3.1. State value function

We use $V^{\pi }\left(s\right)$π \pibased on MDPThe state- value functionof$\pi$ is defined as starting from state$s$ starts and follows the strategy$The\; mathematical\; expression\; of\; the\; expected\; return\; that\; \pi$ can obtain is:
$$V^{\pi }\left(s\right)={\mathbb{E}}_{}[G_t|S_t=s]$$

3.3.2. Action value function

Different from MRP, in MDP, due to the existence of actions, we define an additional action- value function . We use $Q^{\pi }(s,a)$ means following the policy$When\; \pi$ , for the current state$s$ executes action$The$ expected return for a:
$$Q^{\pi }(s,a)={\mathbb{E}}_{}[G_t|S_t=s,A_t=a]$$

3.4. Bellman’s expectation equation

The word "expectation" is added to the Bellman equation to distinguish it from the following Bellman optimal equation .

The state value function can be decomposed into the reward at the current moment plus the discounted state value of the subsequent state
$$V_{}(s)={\mathbb{E}}_{}\left[R_{t+1}+\gamma V_{}(S_{t+1})\mid S_t=s\right]$$

The action value function can also be decomposed:
$$Q_{}(s,a)=E_{}\left[R_{t+1}+\gamma Q_{}(S_{t+1},A_{t+1})|S_t=s,A_t=a\right]$$

The relationship between state value function and action value function: using strategy $\pi$中，状态$s$的价值等于在该状态下基于策略$\pi$采取所有动作$a_i$的概率与相应的价值相乘再求和的结果：
$$V_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a|s)Q_{\pi }(s,a)$$

使用策略$\pi$时，状态$s$下采取动作$a$的价值等于即时奖励加上经过衰减后的所有可能的下一个状态的状态转移概率与相应的价值的乘积：

$$Q_{}(s,a)={\mathcal{R}}_s^a+c\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{V}_{}(s^{\prime })$$

Superposing the two, we can get

$$V_{}(s)=\sum _{a\in \mathcal{A}}\pi (a|s)\left({\mathcal{R}}_s^a+c\sum _{s^{\prime }\in S}{P}_{s{s}^{\prime }}^a{V}_{}(s^{\prime })\right)$$

以及

$$Q_{}(s,a)={\mathcal{R}}_s^a+c\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\sum _{a^{\prime }\in \mathcal{A}}\pi (a^{\prime }|s^{\prime })Q_{}(s^{\prime },a^{\prime })$$the Bellman expectation equation
of the two value functions obtained through simple derivation.

Also use an example to explain: The figure is based on the strategy $\pi$ takes all actions$a_i$The probability is: $\pi (a|s)=0.5$ . CalculationClass$The\; state\; value\; of\; Class3$ needs to calculate the sum of its state values ​​in the next step, so it can be obtained as shown in the formula in the figure:$7.4=0.5\ast (1+0.2\ast -1.3+0.4\ast 2.7+0.4\ast 7.4)+0.5\ast 10$

MRP can also be used for solution. We can marginalize the action selection of the strategy to obtain MRP without actions. Specifically, for a certain state, we weight it according to the probability of all actions of the strategy, and the obtained reward sum can be considered as the reward of an MRP in this state, that is: R π = R s ′ $${\mathcal{R}}^{Pi}=R_s^{{}^{\prime }}=\sum _{a\in \mathcal{A}}\pi (a|s)R_s^a$$
In the same way, we calculate the probability of taking action and make $s$ is transferred to$s^{\prime }$ , and then add these products together, the sum is the state$s$ is transferred from to$s^{\prime }$ Equations:$${\mathcal{P}}^{Pi}=P_{s{s}^{\prime }}^{{}^{\prime }}=\sum _{a\in \mathcal{A}}\pi (a|s)P_{s{s}^{\prime }}^a$$

In this way, you can continue to use the previous MRP steps: $$\begin{gathered} V_{}={\mathcal{R}}^{Pi}+\gamma {\mathcal{P}}^{\pi }{V}_{} \\ {V}_{}=(1-\gamma P^p{)}^{-1}{\mathcal{R}}^{Pi} \end{gathered}$$

def test02():
    # Define the transition Matrix
    # C1 C2  Pass FB Sleep
    P_TransformMDP2MRP = [
        [0.0, 0.5, 0.0, 0.5, 0.0],
        [0.0, 0.0, 0.5, 0.0, 0.5],
        [0.1, 0.2, 0.2, 0.0, 0.5],
        [0.5, 0.0, 0.0, 0.5, 0.0],
        [0.0, 0.0, 0.0, 0.0, 1.0]
    ]
    P_TransformMDP2MRP = np.array(P_TransformMDP2MRP)
    R_TransformMDP2MRP = [-1.5, -1, 5.5, -0.5, 0.0]
    print("MDP中每个状态价值分别为\n", ComputeValue(R_TransformMDP2MRP, 5, P_TransformMDP2MRP, gamma=1))

MDP中每个状态价值分别为
 [[-1.30769231]
 [ 2.69230769]
 [ 7.38461538]
 [-2.30769231]
 [ 0.        ]]

3.5. Optimal strategy

The goal of reinforcement learning is usually to find a strategy that allows the agent to obtain the maximum expected return from the initial state. We first define the partial ordering relationship between strategies: if and only if for any state $s$都有$V^{\pi }\left(s\right)\ge V^{{Pi}^{\prime }}\left(s\right)$，记$Pi>{Pi}^{{}^{\prime }}$ . Therefore, in the MDP with a finite set of states and actions, there is at least one policy that is better than all other policies or at least one policy that is no worse than all other policies. This policy is the optimalpolicy. There may be many optimal strategies, which we all express as${Pi}^{\ast }(s)$。

The optimal strategies all have the same state value function, which we call the optimal state value function , expressed as:
$$V^{\ast }(s)=\underset{Pi}{\max }{V}^{\pi }\left(s\right),\forall s\in \mathcal{S}$$

In the same way, we define the optimal action value function :
$$Q^{\ast }(s,a)=\underset{Pi}{\max }{Q}^{\pi }(s,a),\forall s\in \mathcal{S},a\in \mathcal{A}$$

Examples of optimal state values

​​Examples of optimal action values

By maximizing $Q^{\ast }(s,a)$ ,Let us have the following function:
$${Pi}_{\ast }(a|s)=\{\begin{array}{cc}1& \mathrm{if}a=argmax{q}_{\ast }(s,a)\\ 0& otherwise\end{array}$$

• For all MDPs, there is a certain optimal strategy
• Just find $Q^{\ast }(s,a)$ , we can get the optimal strategy

The red arc arrow in the figure below represents the choice of the optimal strategy

3.5.1. Bellman’s optimal equation

The equation is also derived recursively:

The optimal state value is the state value when choosing the action that maximizes the optimal action value at this time:

$$V_{\ast }(s)=\underset{a}{\max }{Q}_{\ast }(s,a)$$

The relationship between the optimal state value function and the optimal action value function:

$$Q_{\ast }(s,a)={\mathcal{R}}_s^a+c\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{V}_{\ast }(s^{\prime })$$
and then superimposed, we can get

V ∗ ( s ) = max ⁡ a ∈ A { R sa + γ ∑ s ′ ∈ SP ss ′ a V ∗ ( s ′ ) } V_*(s)=\max_{a\in \mathcal{A}}\{\mathcal{R}_s^a+\gamma\sum_{s^{\prime}\in\mathcal{S}}\mathcal{P}_{ss^{\prime}}^ aV_*(s^{\prime})\}V∗​(s)=a∈Amax​{ Rsa​+cs′∈S∑​Pss′a​V∗​(s′)}

$$Q_{\ast }(s,a)={\mathcal{R}}_s^a+c\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\underset{a^{\prime }\in \mathcal{A}}{\max }{Q}_{\ast }(s^{\prime },a^{\prime })$$
In this way, we obtainthe Bellman optimality equation.

3.5.2. Optimal strategy solution

• Bellman's optimal equation is nonlinear
• Generally speaking, there is no closed-form solution
• Some iteration methods:
  • Value Iteration
  • Policy Iteration
  • Q-learning
  • dressing

3.5.3. Extended MDP

• Infinite and continuous MDPs
• Partially observable MDPs(POMDP)
• Undiscounted, average reward MDPs（Ergodic Markov Process）

3.6 Monte-Carlo method

When solving the value function in a larger-scale Markov reward process, dynamic programming algorithms, Monte-Carlo methods, and temporal differences can be used. This section introduces the Monte-Carlo method .

Monte-Carlo methods , also known as statistical simulation methods, are a numerical calculation method based on probability statistics. When using the Monte Carlo method, we usually use repeated random sampling, and then use probability and statistics methods to summarize the numerical estimate of the target we want to obtain from the sampling results. A simple example is using the Monte Carlo method to calculate the area of ​​a circle. For example, if a number of points are randomly generated inside the square shown in the figure below, and the number of points falling on the midpoint of the circle is counted, the ratio of the area of ​​the circle to the area of ​​the square is equal to the ratio of the number of midpoints of the circle to the number of midpoints of the square. . The more points we randomly generate, the closer the calculated area of ​​the circle will be to the real area of ​​the circle.
We now describe how to use Monte Carlo methods to estimate the state value function of a policy in a Markov decision process. Recall that the value of a state is its expected return, so a very intuitive idea is to use a strategy to sample many sequences on the MDP, calculate the return starting from this state, and then find the expectation. The formula is as follows: V $$V^{\pi }\left(s\right)={\mathbb{E}}_{}[G_t|S_t=s]\approx \frac{1}{N}\sum _{i=1}^N{G}_t^{(i)}$$

In a sequence, this state may not appear before, it may only appear once, or it may appear many times. The Monte Carlo value estimation method we introduce calculates the payoff for that state every time it occurs . Another option is to only calculate the reward once for a sequence, that is, calculate the cumulative reward when the state appears for the first time in the sequence, and ignore it when the state appears again later.

Suppose we now use the policy $\pi$ from state$s$ starts the sampling sequence, based on which the state value is calculated. We maintain a counter and total return for each state. The specific process of calculating the state value is as follows.

1. Use strategy $\pi$采样若干条序列：$$s_0^{(i)}\stackrel{a_0^{\left(i\right)}}{\to }{r}_0^{\left(i\right)},s_1^{\left(i\right)}\stackrel{a_1^{\left(i\right)}}{\to }{r}_1^{\left(i\right)},s_2^{\left(i\right)}\stackrel{a_2^{\left(i\right)}}{\to }\cdots \stackrel{a_{T-1}^{\left(i\right)}}{\to }{r}_{T-1}^{\left(i\right)},s_T^{\left(i\right)}$$
2. For each time step tt in each sequence$t$ ’s status$s$ performs the following operations:

• update status ssCounter of$s$$N(s)=N\left(s\right)+1$
• update status ssTotal return of$s$$M(s)=M\left(s\right)+G_t$;
• Or use an incremental update strategy: $V(s)\leftarrow V\left(s\right)+\frac{1}{N(s)}(G-V(S))$

3. The value of each state is estimated as the average return $V\left(s\right)=M\left(s\right)/N\left(s\right)$ . According to the law of large numbers, when$N\left(s\right)\to \infty$ , with$V\left(s\right)\to V_{}(s)$ 。

Some results of code sampling

[('C2', 'Study', -2, 'Pass'), ('Pass', 'Pub', 1, 'Pass'), ('Pass', 'Pub', 1, 'Pass'), ('Pass', 'Pub', 1, 'Pass'), ('Pass', 'Study', 10, 'Sleep')]
[('C1', 'Study', -2, 'C2'), ('C2', 'Sleep', 0, 'Sleep')]
[('FB', 'Quit', 0, 'C1'), ('C1', 'Facebook', -1, 'FB'), ('FB', 'Quit', 0, 'C1'), ('C1', 'Facebook', -1, 'FB'), ('FB', 'Quit', 0, 'C1'), ('C1', 'Study', -2, 'C2'), ('C2', 'Study', -2, 'Pass'), ('Pass', 'Pub', 1, 'Pass'), ('Pass', 'Pub', 1, 'Pass')]
[('C1', 'Study', -2, 'C2'), ('C2', 'Sleep', 0, 'Sleep')]
[('C2', 'Study', -2, 'Pass'), ('Pass', 'Pub', 1, 'C2'), ('C2', 'Sleep', 0, 'Sleep')]

It can be seen that the final result is still relatively close.

使用蒙特卡洛方法计算MDP的状态价值为
 {
    
    'C1': -1.6584167352261565, 'C2': 0.5744913689985154, 'Pass': 6.330419227770518, 'FB': -1.1820907116805823, 'Sleep': 0}
MDP中每个状态价值分别为
 [[-1.67666232]
 [ 0.51890482]
 [ 6.0756193 ]
 [-1.22555411]
 [ 0.        ]]

3.7. Occupancy metrics

The value functions of different strategies are different. This is because for the same MDP, the probability distributions of states visited by different strategies are different. We need to understand the fact that different strategies will cause the agent to access states with different probability distributions, which will affect the value function of the strategy.

First we define the initial state distribution of MDP as $n_0\left(s\right)$ , in some materials, the initial state distribution will be defined into the component elements of the MDP. We use$P_t^{}\left(s\right)$ representsthe adoption of strategy$\pi$ makes the agent at time$t$ state isssThe probability of$s$ , so we have$P_0^{}(s)=n_0\left(s\right)$ , then you can definevisitation distribution:

$$n^{\pi }\left(s\right)=(1-c)\sum _{t=0}^{\infty }{c}^t{P}_t^{}\left(s\right)$$
where,$1-\gamma$ is the normalization factor used to make the probabilities sum to 1. State access probability represents the distribution of states that a policy will access when interacting with MDP. It should be noted that theoretically it is necessary to interact after infinite steps when calculating the distribution, but in fact the interaction between the agent and the MDP is limitedin a sequence. However, we can still use the above formula to express the idea of ​​state access probability. State access probability has the following properties:$$n^{\pi }(s^{\prime })=(1-c)n_0(s^{\prime })+c\int P(s^{\prime }|s,a\left)\pi \right(a|s)n^{\pi }\left(s\right)dsda$$
In addition, we can also definethe occupancy measure of the policy:$$r^{\pi }(s,a)=(1-c)\sum _{t=0}^{\infty }{c}^t{P}_t^{}\left(s\right)\pi (a|s)$$
which represents the action state pair$(s,a)$ Probability of being visited. There is the following relationship between the two:$$r^{\pi }(s,a)=n^{\pi }\left(s\right)\pi (a|s)$$
further leads to the following two theorems. Theorem 1: Agents adopt strategies${Pi}_1$Sum ${Pi}_2$Occupancy metric ρ π 1 obtained by interacting with the same MDP \rho^{\pi_1}$r^{{Pi}_1}$Sum $r^{{Pi}_2}$π π 1 = ρ π 2 ⟺ π 1 = π 2 \rho^{\pi_1}=\rho^{\pi_2}\iff\pi_1=\ pi_2$$r^{{Pi}_1}=r^{{Pi}_2}⟺{Pi}_1={Pi}_2$$
Theorem 2 : Given a legal occupancy metric, the only strategy that can generate the occupancy metric is $${Pi}_{}=\frac{p(s,a)}{{\sum }_{a^{\prime }}p(s,a^{\prime })}$$
Note: The "legal" occupancy metric mentioned above refers to the probability that there is a policy that enables the state-action pairs generated by the interaction between the agent and the MDP to be accessed.

# Occupancy
def test04():
    # 策略2
    Policy_2 = {
    
    
        "C1-Study": 0.6,
        "C1-Facebook": 0.4,
        "FB-Facebook": 0.3,
        "FB-Quit": 0.7,
        "C2-Study": 0.5,
        "C2-Sleep": 0.5,
        "Pass-Study": 0.1,
        "Pass-Pub": 0.9,
    }
    MAXTimeStep = 8
    MDP, Policy_1 = Set_MDPParameterAndPolicy()
    Sequences1 = MonteCarloSampling(MDP, Policy_1, MAXTimeStep, SamplingNum=1000)
    Sequences2 = MonteCarloSampling(MDP, Policy_2, MAXTimeStep, SamplingNum=1000)
    # Sequences1 = sample(MDP, Policy_1, MAXTimeStep, 1000)
    # Sequences2 = sample(MDP, Policy_2, MAXTimeStep, 1000)
    rho1 = ComputeOccupancy("Pass", "Pub", Sequences1, MAXTimeStep, MDP)
    rho2 = ComputeOccupancy("Pass", "Pub", Sequences2, MAXTimeStep, MDP)
    print(rho1, rho2)

The resulting occupancy measure is different

0.058 0.1145

code

import numpy as np

# 给定一条序列,计算从某个索引（起始状态）开始到序列最后（终止状态）得到的回报
def ComputeSequenceReward(Start_idx, Sequence, RewardVector, gamma=0.5):
    TotalReward = 0.0
    for i in reversed(range(Start_idx, len(Sequence))):
        TotalReward = gamma * TotalReward + RewardVector[Sequence[i] - 1]
    return TotalReward

# Exploit Bellman equation to compute value of all states
def ComputeValue(RewardVector, Statesize, TransitionMatrix, gamma=0.5):
    RewardVector = np.array(RewardVector).reshape(-1, 1)
    try:
        Value = np.dot(np.linalg.inv(np.eye(Statesize, Statesize) - gamma * TransitionMatrix),
                   RewardVector)
    except:
        print("-------------状态转移矩阵为奇异矩阵，存在求解误差-------------")
        TransitionMatrix[Statesize - 1][Statesize - 1] += 1e-7
        I = np.eye(Statesize, Statesize)
        Value = np.dot(np.linalg.inv(I - gamma * TransitionMatrix),
                   RewardVector)
    return Value

def Set_MDPParameterAndPolicy():
    # 状态集合
    S = ["C1", "C2", "Pass", "FB", "Sleep"]
    # 动作集合
    A = ["Facebook", "Study", "Sleep", "Pub", "Quit"]
    # 状态转移函数
    P = {
    
    
        "C1-Study-C2": 1.0,
        "C1-Facebook-FB": 1.0,
        "FB-Facebook-FB": 1.0,
        "FB-Quit-C1": 1.0,
        "C2-Study-Pass": 1.0,
        "C2-Sleep-Sleep": 1.0,
        "Pass-Study-Sleep": 1.0,
        "Pass-Pub-C1": 0.2,
        "Pass-Pub-C2": 0.4,
        "Pass-Pub-Pass": 0.4,
    }
    # 奖励函数
    R = {
    
    
        "C1-Study": -2,
        "C1-Facebook": -1,
        "FB-Facebook": -1,
        "FB-Quit": 0,
        "C2-Study": -2,
        "C2-Sleep": 0,
        "Pass-Study": 10,
        "Pass-Pub": 1,
    }
    # 折扣因子
    gamma = 0.5
    MDP = (S, A, P, R, gamma)

    # 策略1,随机策略
    Pi_1 = {
    
    
        "C1-Study": 0.5,
        "C1-Facebook": 0.5,
        "FB-Facebook": 0.5,
        "FB-Quit": 0.5,
        "C2-Study": 0.5,
        "C2-Sleep": 0.5,
        "Pass-Study": 0.5,
        "Pass-Pub": 0.5,
    }
    # 策略2
    Pi_2 = {
    
    
        "C1-Study": 0.7,
        "C1-Facebook": 0.3,
        "FB-Facebook": 0.3,
        "FB-Quit": 0.7,
        "C2-Study": 0.5,
        "C2-Sleep": 0.5,
        "Pass-Study": 0.2,
        "Pass-Pub": 0.8,
    }
    return MDP, Pi_1

# 把输入的两个字符串通过“-”连接,便于使用上述定义的P、R变量
def join(str1, str2):
    return str1 + '-' + str2

def MonteCarloSampling(MDP, Policy, MAXTimeStep, SamplingNum):
    ''' 采样函数,策略Pi,限制最长时间步MaxTimeStep,总共采样序列数SamplingNum '''
    S, A, P, R, gamma = MDP
    StateNum = len(S)
    Sequences = []
    for _ in range(SamplingNum):
        Sequence = []
        TimeStep = 0
        # 随机选择一个除Sleep以外的状态s作为起点
        s = S[np.random.randint(StateNum - 1)]
        # 当前状态为终止状态或者时间步太长时,一次采样结束
        while s != "Sleep" and TimeStep <= MAXTimeStep:
            TimeStep += 1
            rand, temp = np.random.rand(), 0
            # 在状态s下根据策略选择动作
            for a_ in A:
                temp += Policy.get(join(s, a_), 0.0)
                if temp >= rand:
                    a = a_
                    r = R.get(join(s, a_), 0.0)
                    break
            rand, temp = np.random.rand(), 0
            # 根据状态转移概率得到下一个状态s_next
            for s_ in S:
                temp += P.get(join(join(s, a), s_), 0.0)
                if temp >= rand:
                    s_next = s_
                    break
            # 把（s,a,r,s_next）元组放入序列中
            Sequence.append((s, a, r, s_next))
            # s_next变成当前状态,开始接下来的循环
            s = s_next
        Sequences.append(Sequence)
    return Sequences

# 对所有采样序列计算所有状态的价值
def MonteCarloComputeValue(Sequences, MDP):
    gamma = MDP[4]
    V = {
    
    "C1": 0, "C2": 0, "Pass": 0, "FB": 0, "Sleep": 0}
    N = {
    
    "C1": 0, "C2": 0, "Pass": 0, "FB": 0, "Sleep": 0}
    for Sequence in Sequences:
        G = 0
        # 一个序列从后往前计算
        for i in reversed(range(len(Sequence))):
            s, r = Sequence[i][0], Sequence[i][2]
            G = r + gamma * G
            N[s] = N[s] + 1
            V[s] = V[s] + (G - V[s]) / N[s]
    return V

def ComputeOccupancy(s, a, Sequences, MAXTimeStep, MDP):
    ''' 计算状态动作对（s,a）出现的频率,以此来估算策略的占用度量 '''
    gamma = MDP[4]
    rho = 0
    total_times = np.zeros(MAXTimeStep)  # 记录每个时间步t各被经历过几次
    occur_times = np.zeros(MAXTimeStep)  # 记录(s_t,a_t)=(s,a)的次数
    for Sequence in Sequences:
        for i in range(len(Sequence)):
            try:
                s_, a_ = Sequence[i][0], Sequence[i][1]
                total_times[i] += 1
                if s_ == s and a_ == a:
                    occur_times[i] += 1
            except IndexError:
                continue
    for i in reversed(range(MAXTimeStep)):
        if total_times[i]:
            # 用频率来估算策略的占用度量
            rho = gamma ** i * occur_times[i] / total_times[i]
    return (1 - gamma) * rho

def SampleTEXT():
    MDP, Policy = Set_MDPParameterAndPolicy()
    Sequences = MonteCarloSampling(MDP, Policy, MAXTimeStep=8, SamplingNum=5)
    for Sequence in Sequences:
        print(Sequence)

def MonteCarloTEXT():
    MDP, Policy = Set_MDPParameterAndPolicy()
    Sequences = MonteCarloSampling(MDP, Policy, MAXTimeStep=8, SamplingNum=5000)
    V = MonteCarloComputeValue(Sequences, MDP)
    print("使用蒙特卡洛方法计算MDP的状态价值为\n", V)

def test01():
    # Define the transition Matrix
    # C1 C2 C3 Pass Pub FB Sleep
    P = [
        [0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 0.0],
        [0.0, 0.0, 0.8, 0.0, 0.0, 0.0, 0.2],
        [0.0, 0.0, 0.0, 0.6, 0.4, 0.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
        [0.2, 0.4, 0.4, 0.0, 0.0, 0.0, 0.0],
        [0.1, 0.0, 0.0, 0.0, 0.0, 0.9, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]
    ]
    P = np.array(P)
    RewardVector = [-2, -2, -2, 10, 1, -1, 0]
    chain = [1, 6, 6, 1, 2, 7]
    start_index = 0
    print("根据本序列计算得到回报为：%s。"% ComputeSequenceReward(start_index, chain, RewardVector, gamma=0.5))
    print("MRP中每个状态价值分别为\n", ComputeValue(RewardVector, 7, P))

# MDP2MRP
def test02():
    # Define the transition Matrix
    # C1 C2  Pass FB Sleep
    P_TransformMDP2MRP = [
        [0.0, 0.5, 0.0, 0.5, 0.0],
        [0.0, 0.0, 0.5, 0.0, 0.5],
        [0.1, 0.2, 0.2, 0.0, 0.5],
        [0.5, 0.0, 0.0, 0.5, 0.0],
        [0.0, 0.0, 0.0, 0.0, 1.0]
    ]
    P_TransformMDP2MRP = np.array(P_TransformMDP2MRP)
    R_TransformMDP2MRP = [-1.5, -1, 5.5, -0.5, 0.0]
    print("MDP中每个状态价值分别为\n", ComputeValue(R_TransformMDP2MRP, 5, P_TransformMDP2MRP, gamma=0.5))

# MonteCarlo
def test03():
    # SampleTEXT(
    MonteCarloTEXT()
    test02()

# Occupancy
def test04():
    # 策略2
    Policy_2 = {
    
    
        "C1-Study": 0.6,
        "C1-Facebook": 0.4,
        "FB-Facebook": 0.3,
        "FB-Quit": 0.7,
        "C2-Study": 0.5,
        "C2-Sleep": 0.5,
        "Pass-Study": 0.1,
        "Pass-Pub": 0.9,
    }
    MAXTimeStep = 8
    MDP, Policy_1 = Set_MDPParameterAndPolicy()
    Sequences1 = MonteCarloSampling(MDP, Policy_1, MAXTimeStep, SamplingNum=1000)
    Sequences2 = MonteCarloSampling(MDP, Policy_2, MAXTimeStep, SamplingNum=1000)
    rho1 = ComputeOccupancy("Pass", "Pub", Sequences1, MAXTimeStep, MDP)
    rho2 = ComputeOccupancy("Pass", "Pub", Sequences2, MAXTimeStep, MDP)
    print(rho1, rho2)

if __name__ == "__main__":
    test04()

reference

[1] Boyu AI
[2] https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver
[3] Hands-on learning reinforcement learning
[4] Reinforcement Learning