机器学习-白板推导系列笔记（三十四）-MDP

此文章主要是结合哔站shuhuai008大佬的白板推导视频：马尔科夫决策过程_107min

全部笔记的汇总贴：机器学习-白板推导系列笔记

一、背景介绍

Random Variable：$XYX\perp Y$
Stochastic Process： { S t } t = 1 ∞ \{S_t\}_{t=1}^\infty { St​}t=1∞​
Markov Chain/Process，具有Markov Property的随机过程：$P(S_{t+1}|S_t,S_{t-1},\cdots ,S_1)=P(S_{t+1}|S_t)$
State Space Model：（HMM，Kalman Filter，Particle Filter）Markov Chain+Observation
Markov Reward Process：Markov Chain+Reward
Markov Decision Process：Markov Chain+Reward+Action

$$\begin{gathered} S:stateset\to S_t \\ A:actionset,\forall s\in S,A(s)\to A_t \\ R:rewardset\to R_t,R_{t+1} \end{gathered}$$

二、动态特性

Markov Chain：$S$
MRP：$S,R$
MDP：$S,A(s),R,P（动态特性）$

P : p ( s ′ , r ∣ s , a ) ≜ P r { S t + 1 = s ′ , R t + 1 = r ∣ S t = s , A t = a } P:p(s',r|s,a)\triangleq Pr\{S_{t+1}=s',R_{t+1}=r|S_t=s,A_t=a\} P:p(s′,r∣s,a)≜Pr{ St+1​=s′,Rt+1​=r∣St​=s,At​=a}
状态转移函数：

$$P(s^{\prime }|s,a)\sum _{r\in R}P(s^{\prime },r|s,a)$$

三、价值函数

Policy：$\pi$表示

1. 确定性策略：$$a\triangleq \pi (s)$$
2. 随机性策略： π ( a ∣ s ) ≜ P r { A t = a ∣ S t = s } \pi(a|s)\triangleq Pr\{A_t=a|S_t=s\} π(a∣s)≜Pr{ At​=a∣St​=s}

回报：
$$\begin{gathered} G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots +{\gamma }^{r-1}{R}_T=\sum _{i=0}^{\infty }{\gamma }^i{R}_{t+i+1}(T\to \infty ) \\ \gamma \in [0,1] \end{gathered}$$

价值函数：$$\begin{gathered} V_{\pi }(s)\triangleq E_{\pi }[G_t|S_s=s] \\ {q}_{\pi }(s,a)\triangleq E_{\pi }[G_t|S_s=s,A_t=a] \end{gathered}$$

四、贝尔曼期望方程

回溯图：

Value Function：$$\begin{gathered} V_{\pi }(s)\triangleq E_{\pi }[G_t|S_s=s]（加权平均值） \\ {q}_{\pi }(s,a)\triangleq E_{\pi }[G_t|S_s=s,A_t=a] \end{gathered}$$
$$V_{\pi }(s)=\sum _{a\in A(s)}\pi (a|s)\cdot q_{\pi }(s,a)$$

$$V_{\pi }(s)=\pi (a_1|s)\cdot q_{\pi }(s.a_1)$$

$$q_{\pi }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)\cdot [r+\gamma V_{\pi }(s^{\prime })]$$

贝尔曼期望方程：
$$V_{\pi }(s)=\sum _{a\in A}\pi (a|s)\cdot \sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma V_{\pi }(s^{\prime })]$$

$$V_{\pi }(s)=E_{\pi }[R_{t+1}+\gamma V_{\pi }(S_{t+1})|S_t=s]$$
$$q_{\pi }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)\cdot [r+\gamma \sum _{a^{\prime }\in A}\pi (a^{\prime }|s^{\prime })\cdot q_{\pi }(s^{\prime },a^{\prime })]$$

五、贝尔曼最优方程

$$\begin{gathered} V_{\ast }(s)\triangleq \underset{\pi }{\max }{V}_{\pi }(s) \\ {q}_{\ast }(s,a)\triangleq \underset{\pi }{\max }{q}_{\pi }(s,a) \end{gathered}$$

记${\pi }_{\ast }=\underset{\pi }{\mathrm{argmax}}{V}_{\pi }(s)=\underset{\pi }{\mathrm{argmax}}{q}_{\pi }(s,a)$，

$$\begin{gathered} V_{\ast }(s)=\underset{\pi }{\max }{V}_{\pi }(s)=V_{{\pi }_{\ast }}(s) \\ {q}_{\ast }(s,a)=\underset{\pi }{\max }{q}_{\pi }(s,a)=q_{{\pi }_{\ast }}(s,a) \end{gathered}$$

$$V_{{\pi }_{\ast }}(s)\le \underset{a}{\max }{q}_{{\pi }_{\ast }}(s,a)$$

假如$$V_{{\pi }_{\ast }}(s)<\underset{a}{\max }{q}_{{\pi }_{\ast }}(s,a)$$则可以构造一个${\pi }_{new}$，$$\begin{gathered} {\pi }_{new}(s)=\underset{a}{\mathrm{argmax}}{q}_{{\pi }_{\ast }}(s,a) \\ {\pi }_{new}(\overline{s})={\pi }_{\ast }(\overline{s})或{\pi }_{new}(a|\overline{s})={\pi }_{\ast }(a|\overline{s}) \end{gathered}$$
则，$$V_{{\pi }_{new}}(s)=\underset{a}{\max }{q}_{{\pi }_{\ast }}(s,a)>V_{{\pi }_{\ast }}(s)$$
则说明${\pi }_{new}$好于${\pi }_{\ast }$，这与${\pi }_{\ast }$最优矛盾，因此假设不成立，所以有：$$V_{{\pi }_{\ast }}(s)=\underset{a}{\max }{q}_{{\pi }_{\ast }}(s,a)$$

$$V_{{\pi }_{\ast }}(s)=\underset{a}{\max }{q}_{{\pi }_{\ast }}(s,a)$$

$$V_{\ast }(s)=\underset{a}{\max }{q}_{\ast }(s,a)$$
$$q_{\ast }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)\cdot [r+\gamma V_{\ast }(s^{\prime })]$$
贝尔曼最优方程：
$$V_{\ast }(s)=\underset{a}{\max }\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma V_{\ast }(s^{\prime })]$$

$$q_{\ast }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)\cdot [r+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]$$

   \;
   \;
   \;

下一章传送门：白板推导系列笔记（三十五）-动态规划（强化学习）