In-depth understanding of reinforcement learning - Markov decision process: occupancy measurement - [Basic knowledge]

Category Catalog:General Catalog of "In-depth Understanding of Reinforcement Learning"

---

mentioned in the article "In-depth understanding of reinforcement learning - Markov decision process: Bellman expectation equation-[Basic knowledge]" , the value functions of different strategies are different. This is because for the same Markov decision process, the probability distributions of states visited by different strategies are different. Imagine that there is now a policy in the Markov decision process in the figure below, and the execution of its actions will make the agent reach the terminal state as soon as possible$s_5$, then when the agent is in state $s_3$ will not take "Go$s_4$" action, and will only take the action "Go$s_5$" action, so the agent will not get the action in $s_4$In the state, take "Go$s_5$"The big reward that can be obtained is 10. It is conceivable that according to the Bellman equation, the probability of this strategy in the state will be relatively small. The reason is that it cannot reach the state. Therefore, we need to understand that different strategies will make intelligence Consider the fact that states from different probability distributions are accessed, which affects the value function of the strategy.

First we define the initial state distribution of the Markov decision process as${in}_0\left(s\right)$, in some data, the initial state distribution will be defined in Markov elements of the husband's decision-making process. We use $P_t^{\pi }\left(s\right)$Display strategy$\pi$Using intelligent body time$t$ 电影的的$s$ probability, so we have$P_0^{\pi }(s)={in}_0\left(s\right)$, then you can define a policy’s State Visitation Distribution ):
$${in}^{\pi }(s)=(1-\gamma )\sum _{t=1}^{\infty }{c}^t{P}_t^{\pi }\left(s\right)$$

In that,$1-\gamma$ is the normalization factor used to make the probability sum to 1. State access probability represents the distribution of states visited by a policy interacting with a Markov decision process. It should be noted that theoretically it is necessary to interact after infinite steps when calculating this distribution, but in fact the interaction between the agent and the Markov decision process is limited in a sequence. However, we can still use the above formula to express the idea of ​​state access probability. The state access probability has the following properties:
$${in}^{\pi }(s^{\prime })=(1-\gamma )v_0(s^{\prime })+c\int P(s^{\prime }|s,a)\pi (a|s)v^{\pi }(s)\text{d}s\text{d}a$$

In addition, we can also define the occupancy measure of the strategy (Occupancy Measure):
$$r^{\pi }(s,a)=(1-\gamma )\sum _{t=1}^{\infty }{c}^t{P}_t^{\pi }(s)\pi (a|s)$$

It represents the action state pair$(s,a)$approximate probability of being visited. The system of existence between two parties:
$$r^{\pi }(s,a)={in}^{\pi }(s)\pi (a|s)$$

Further we can derive the following two theorems:

• Theorem 1: Agents use strategies ${Pi}_1$sum${Pi}_2$The occupancy measure obtained by interacting with the same Markov decision process satisfies:$$r^{{Pi}_1}=r^{{Pi}_2}\iff {Pi}_1={Pi}_2$$
• Theorem 2: Given a legal occupancy metric$\rho$, the only strategy that can generate this occupancy metric is:$${Pi}_{\rho }=\frac{\rho (s,a)}{{\sum }_{a^{\prime }}\rho (s,a^{\prime })}$$

The "legitimate" occupancy metric mentioned above refers to the probability that a state-action pair generated by the interaction of the agent with the Markov decision process is accessed by a policy.

References:
[1] Zhang Weinan, Shen Jian, Yu Yong. Hands-on reinforcement learning [M]. People's Posts and Telecommunications Press, 2022.
[2] Richard S. Sutton, Andrew G. Barto. Reinforcement Learning (2nd Edition) [M]. Electronic Industry Press, 2019
[3] Maxim Lapan. Deep Reinforcement Learning Practice (2nd edition of the original book) [M]. Beijing Huazhang Graphic Information Co., Ltd., 2021
[4] Wang Qi, Yang Yiyuan, Jiang Ji. Easy RL: Reinforcement Learning Tutorial[M] . People's Posts and Telecommunications Press, 2022