ADPRL - Approximate Dynamic Programming and Reinforcement Learning - Note 8 - Approximate Policy Iteration

Note 8 Approximate Policy Iteration

In Note 7, we introduced the concept of parametric function approximation and its application in approximation iterative algorithms. Although the convergence properties of AVI have proven promising, it is not the same as the original $The\; inherent\; limitations\; of\; the$$VI$ algorithm still exist. In this section, we develop a framework for an approximate policy iteration algorithm.

8.1 A Generic Framework

Similar to the approximate VI algorithm, we can build a system to approximate the policy evaluation and policy improvement steps, as follows

1. For a given policy ${Pi}_k$, our goal is to find the true total cost $J^{{Pi}_k}$Approximate value of $J_k$,
   $${\Vert J_k-J^{{Pi}_k}\Vert }_{\infty }\le d\text{\hspace{1em}( 8. 1 ) Please note that}$$$$\text{\hspace{1em}the}$$
   true total cost$J^{{Pi}_k}$In general it cannot be given. The idea of ​​Bellman residual minimization can be used here.

2. By taking the same equation as ( $For\; the\; same\; strategy\; as\; the\; approximate$ greedy step in 7.31), we can also relax it into an approximate$strategy$$improvement$$.$That is to say, given the$k$ value function estimates$J_k$, we find a policy ${Pi}_{k+1}$,theoretically
   $${\Vert {\mathrm{T}}_{{Pi}_{k+1}}{J}_k-{\mathrm{T}}_{\mathfrak{g}}{J}_k\Vert }_{\infty }\le ϵ,\text{\hspace{1em}( 8.2 )}$$
   where$$\text{\hspace{1em}ϵ}$$ > 0 \epsilon>$$\text{\hspace{1em}0}$$$ϵ>0$ is the accuracy of inexact policy improvement.

Such a general approximate PI algorithm is given in Algorithm 10.

In order to determine the error bound of the approximate PI algorithm, we need the following two lemmas (Lemma).

Lemma 8.1 Error bound under monotonicity

Give an infinite range MDP { X , U , p , g , γ } \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , and a fixed policy$\pi$。让$J\in {\mathbb{R}}^{In\; K}$ , the following conditions are met
$${\mathrm{T}}_{}J\le J+c1\text{\hspace{1em}(8.3)}$$

And there is $c>0$ , then the strategy$The\; total\; cost\; function\; of\; \pi$ has the following constraints
$$J^{Pi}\le J+\frac{c}{1-c}1\text{\hspace{1em}(8.4)}$$

---

Proof.

Bellman operator ${\mathrm{T}}_{}$The constant displacement property of means that for all $k\in Note$ π
$$T_{Pi}^{k}J\le T_{Pi}^{k-1}J+c^{k-1}c1\text{\hspace{1em}(8.5)}$$

Then we have any $k$ ratio
$$\begin{array}{rl}{\mathrm{T}}_{Pi}^{k}J-J& =T_{Pi}^{k}J-T_{Pi}^{k-1}J+T_{Pi}^{k-1}J-\dots +T_{}J-J\\ & =\sum _{t=1}^k\left({\mathrm{T}}_{Pi}^{k}J-T_{Pi}^{k-1}J\right)\\ & \le \sum _{t=1}^k{c}^{t-1}c1\end{array}\text{\hspace{1em}(8.6)}$$

The result is through $t\to \infty$ .

---

Lemma 8.2 Error bound of single approximate PI sweep

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , a fixed policy$\pi$ , one in${\mathbb{R}}^{}$EstimateJJ${}^{in\; K}$$J$ , and two fixed policies$\pi$和${Pi}^{\prime }$ , if the following two conditions are at some$d\ge 0$和$ϵ\ge$Established at$0\; o\text{'}clock$

$${\Vert J-J^{\pi }\Vert }_{\infty }\le d,\text{ and }{\Vert {\mathrm{T}}_{{Pi}^{\prime }}J-{\mathrm{T}}_{\mathfrak{g}}J\Vert }_{\infty }\le ϵ\text{\hspace{1em}(8.7)}$$

Then we have

$${\Vert J^{{Pi}^{\prime }}-J^{\ast }\Vert }_{\infty }\le c{\Vert J^{Pi}-J^{\ast }\Vert }_{\infty }+\frac{ϵ+2c}{1-c}\text{\hspace{1em}(8.8)}$$

---

Proof.

根据${\mathrm{T}}_{\mathfrak{g}}$和${\mathrm{T}}_{{\pi }^{\prime }}$的收缩特性，公式（8.7）中的第一个不等式意味着
$${\Vert {\mathrm{T}}_{{\pi }^{\prime }}J-{\mathrm{T}}_{{\pi }^{\prime }}{J}^{\pi }\Vert }_{\infty }\le \gamma \delta ,\text{ and }{\Vert {\mathrm{T}}_{\mathfrak{g}}J-{\mathrm{T}}_{\mathfrak{g}}{J}^{\pi }\Vert }_{\infty }\le \gamma \delta \text{\hspace{1em}(8.9)}$$

Let
$${\mathrm{T}}_{{Pi}^{\prime }}{J}^{Pi}\le T_{{Pi}^{\prime }}J+cd1,\text{ and }{T}_{\mathfrak{g}}J-T_g{J}^{Pi}\le cd1\text{\hspace{1em}(8.10)}$$

Similarly, the second inequality in equation (8.7) yields
$$T_{{Pi}^{\prime }}J\le T_{g}J+ϵ1\text{\hspace{1em}(8.11)}$$

then we get

$$\begin{array}{rl}{\mathrm{T}}_{{Pi}^{\prime }}{J}^{Pi}& \le T_{{Pi}^{\prime }}J+cd1\\ & \le T_{\mathfrak{g}}J+(ϵ+cd)1\\ & \le T_g{J}^{Pi}+(ϵ+2cd)1\\ & \le J^{Pi}+(ϵ+2cd)\end{array}\text{\hspace{1em}(8.12)}$$

Among them, the second inequality is due to formula (8.11), the third inequality is derived from the second inequality in formula (8.10), and the last inequality is due to ${\mathrm{T}}_{\mathfrak{g}}$The strategy improvement properties of $T_g{J}^{Pi}\le T_{}{J}^{Pi}=J^{Pi}$

Property Lemma 8.1,IfJ
$$J^{{Pi}^{\prime }}\le J^{Pi}+\frac{ϵ+2c}{1-c}1\text{\hspace{1em}(8.13)}$$

And further convert the Bellman operator $T_{{Pi}^{\prime }}$Apply to both sides of the inequality to get

$${\mathrm{T}}_{{Pi}^{\prime }}{J}^{{Pi}^{\prime }}=J^{{Pi}^{\prime }}\le T_{{Pi}^{\prime }}{J}^{Pi}+\frac{ϵ+2c}{1-c}c1.\text{\hspace{1em}(8.14)}$$

Subtract $J^{\ast }$，我们得到
$$\begin{array}{rl}{J}^{{\pi }^{\prime }}-J^{\ast }& \le {\mathrm{T}}_{{\pi }^{\prime }}{J}^{\pi }-J^{\ast }+\frac{ϵ+2\gamma \delta }{1-\gamma }\gamma 1\\ & \le {\mathrm{T}}_{\mathfrak{g}}{J}^{\pi }+(ϵ+2cd)1-J^{\ast }+\frac{ϵ+2c}{1-c}c1\\ & ={\mathrm{T}}_{\mathfrak{g}}{J}^{Pi}-T_g{J}^{\ast }+\frac{ϵ+2c}{1-c}1\end{array}\text{\hspace{1em}(8.15)}$$

Among them, the second inequality follows from the third inequality in formula (8.12), and the equality is due to the optimal Bellman operator $T_g$the only fixed point. Finally, we apply the infinite norm to equation (8.15)

$$\begin{array}{rl}{\Vert J^{{Pi}^{\prime }}-J^{\ast }\Vert }_{\infty }& \le {\Vert {\mathrm{T}}_{\mathfrak{g}}{J}^{Pi}-{\mathrm{T}}_g{J}^{\ast }\Vert }_{\infty }+\frac{ϵ+2c}{1-c}\\ & \le c{\Vert J^{Pi}-J^{\ast }\Vert }_{\infty }+\frac{ϵ+2c}{1-c}\end{array}\text{\hspace{1em}(8.16)}$$

This completes the proof.

---

Finally, we summarize the error bounds of the approximate PI algorithm as follows.

Proposition 8.1 Error bound of the approximate PI algorithm (Error bound of the approximate PI algorithm)

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , π k \pi_{k}produced by the approximate PI method${Pi}_k$Infinitesimal
$$\underset{k\to \infty }{\lim }{\Vert J^{{Pi}_k}-J^{\ast }\Vert }_{\infty }\le \frac{ϵ+2c}{(1-c{)}^2}.\text{\hspace{1em}(8.17)}$$

---

Proof.

Given an arbitrary ${Pi}_0$，Lemma $8.2$意味着
$${\Vert J^{{\pi }_1}-J^{\ast }\Vert }_{\infty }\le \gamma {\Vert J^{{\pi }_0}-J^{\ast }\Vert }_{\infty }+\frac{ϵ+2\gamma \delta }{1-\gamma }\text{\hspace{1em}(8.18)}$$

通过直接的归纳论证，对于任意的$k$ ,if there exists
$${\Vert J^{{Pi}_k}-J^{\ast }\Vert }_{\infty }\le c^k{\Vert J^{{Pi}_0}-J^{\ast }\Vert }_{\infty }+\left(\sum _{i=0}^{k-1}{c}^i\right)\frac{ϵ+2c}{1-c}\text{\hspace{1em}(8.19)}$$

The result is by letting $k\to \infty$ derived.

---

It should be noted that the error bounds of the policies produced by the approximate PI algorithm are not guaranteed to converge within the policy space. That is, the approximate PI algorithm can swing among a set of strategies, see Figure 14.

Figure 14: Illustration of potential convergence modes of the approximate PI algorithm. When the error constraints are relaxed, the policy produced by the approximate PI algorithm may swing among several candidates, such as $\left\{p_1,{Pi}_2,{Pi}_3,{Pi}_4\right\}.When$ the error constraint is strict enough, the resulting strategy may converge to a fixed value, such as${Pi}_1$。

However, in some cases the algorithm can converge to a single policy. In the remainder of the Note, we determine the error bounds of the approximate PI algorithm when the policy converges.

Proposition 8.2 Error bounds of approximate PI under convergence in policy space (Error bounds of approximate PI under convergence in policy space)

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,c } , letπ${Pi}^{\prime }$ as a strategy to approximate the convergence of the PI algorithm. Then we have

$${\Vert J^{{Pi}^{\prime }}-J^{\ast }\Vert }_{\infty }\le \frac{ϵ+2c}{1-c}\text{\hspace{1em}(8.20)}$$

---

Proof.

Let $J^{\prime }\in {\mathbb{R}}^K$ is given by${Pi}^{}$The policy produced by the approximate policy evaluation of${}^{\prime }$$J^{\prime }$ and${Pi}^{\text{'}}$ Satisfies the conditions of the approximate PI algorithm

$${\Vert J^{\prime }-J^{{Pi}^{\prime }}\Vert }_{\infty }\le d,\text{ and }{\Vert {\mathrm{T}}_{{Pi}^{\prime }}{J}^{\prime }-{\mathrm{T}}_{\mathfrak{g}}{J}^{\prime }\Vert }_{\infty }\le ϵ\text{. }\text{\hspace{1em}(8.21)}$$

Well, we have

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\mathfrak{g}}{J}^{{Pi}^{\prime }}-J^{{Pi}^{\prime }}\Vert }_{\infty }\le & {\Vert T_g{J}^{{Pi}^{\prime }}-T_g{J}^{\prime }\Vert }_{\infty }+{\Vert {\mathrm{T}}_g{J}^{\prime }-T_{{Pi}^{\prime }}{J}^{\prime }\Vert }_{\infty }+\\ & +{\Vert T_{{Pi}^{\prime }}{J}^{\prime }-J^{{Pi}^{\prime }}\Vert }_{\infty }\\ \le & c{\Vert J^{{Pi}^{\prime }}-J^{\prime }\Vert }_{\infty }+{\Vert T_g{J}^{\prime }-T_{{Pi}^{\prime }}{J}^{\prime }\Vert }_{\infty }+\\ & +c{\Vert J^{\prime }-J^{{Pi}^{\prime }}\Vert }_{\infty }\\ \le & ϵ+2c\end{array}\text{\hspace{1em}(8.22)}$$

Among them, the first inequality comes from the triangular property of infinite norm, and the second inequality is due to $T_g$和$T_{{Pi}^{\prime }}$contraction properties of , and the last inequality merely recalls the result in equation (8.21). Then, the inequality in equation (8.20) is a direct application of Lemma 3.4.

---

Obviously, the error bound of the approximate PI algorithm under stable convergence is much stricter than the case of divergence, especially when the discount coefficient $When\; \gamma$ is close to 1.

8.2 Approximate Policy Evaluation

The analysis of the convergence properties of the general API shows the importance of the performance of approximate strategy evaluation. Strategies similar to those used to develop AVI for minimizing Bellman residuals can also be applied to strategy evaluation.

Definition 8.1 Approximate total cost function

Given an infinite range MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , a fixed policy$\pi$ and a total cost function space$J$ , total cost function$J\in$Approximate total cost functionJ π J^{\pi}$of\; J$$J^{\pi }$ is given by minimizing the Bellman residual, i.e.

$$J_B^{}\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert {\mathrm{T}}_{}J-J\Vert }_{\infty }.\text{\hspace{1em}(8.23)}$$

Estimate JB π J_{B}^{\pi} by minimizing the Bellman residual error$J_B^{}$The error bounds are as follows.

Lemma 8.3 Approximate cost function bounds

Given an infinite range MDP {X, U, p, q, γ} MDP\{\mathcal{X}, \mathcal{U}, p, q, \gamma\}MDP{ X,U,p,q,γ }，让$J^{\pi }$ is a fixed strategyπ \piThe total cost function of$\pi \; .$Then, for any total cost function$J\in {\mathbb{R}}^{In\; K}$ , the following inequality holds

$${\Vert J-J^{\pi }\Vert }_{\infty }\le \frac{1}{1-\gamma }{\Vert J-{\mathrm{T}}_{\pi }J\Vert }_{\infty }.\text{\hspace{1em}(8.24)}$$

---

Proof.
直接的有
$$\begin{array}{rl}{\Vert J-J^{\pi }\Vert }_{\infty }& ={\Vert J-{\mathrm{T}}_{\pi }J+{\mathrm{T}}_{\pi }J-J^{\pi }\Vert }_{\infty }\\ & ={\Vert J-{\mathrm{T}}_{\pi }J\Vert }_{\infty }+{\Vert {\mathrm{T}}_{\pi }J-J^{\pi }\Vert }_{\infty }\\ & \le {\Vert J-{\mathrm{T}}_{\pi }J\Vert }_{\infty }+\gamma {\Vert J-J^{\pi }\Vert }_{\infty }\end{array}\text{\hspace{1em}(8.25)}$$

---

Proposition 8.3 Constraints between estimates and true total cost functions

Given an infinite range MDP { X , U , p , g , γ } MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , a fixed policy$\pi$ and a total cost function space$J$。 letJB$J_B^{}\in J$ is the global minimum of the MSBE problem. Then the estimated value and the true total cost function$J^{}$The error between${}^{\pi \; has\; the\; following\; constraints}$

$${\Vert J_B^{}-J^{\pi }\Vert }_{\infty }\le \frac{1+}{1-c}\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\pi }\Vert }_{\infty }.\text{\hspace{1em}(8.26)}$$

---

Proof.

By applying the triangle inequality with infinite norm, we get
$$\begin{array}{rl}{\Vert {\mathrm{T}}_{}J-J\Vert }_{\infty }& \le {\Vert T_{}J-J^{\pi }\Vert }_{\infty }+{\Vert J^{Pi}-J\Vert }_{\infty }\\ & \le (1+c){\Vert J-J^{\pi }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(8.27)}$$

直截了当地有
$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\pi }{J}_B^{\pi }-J_B^{\pi }\Vert }_{\infty }& =\underset{J\in \mathcal{J}}{\min }{\Vert {\mathrm{T}}_{\pi }J-J\Vert }_{\infty }\\ & \le (1+\gamma )\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\pi }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(8.28)}$$

结合不等式和Lemma $8.3$中的结果，证明了这一点。

---

Obviously, the MSBE cost given in equation (8.23) is still numerically difficult to optimize. Therefore, similar to AVI, we can define the following mean squared Bellman Error (MSBE) minimization problem
$$J_2^{}\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert {\mathrm{T}}_{}J-J\Vert }_2.\text{\hspace{1em}(8.29)}$$

If we adopt the matrix form of the Bellman operator and choose the function approximation space to be linear, that is, ${\mathrm{T}}_{}J=G_{}+\gamma P_{}{Phi}^{\top }h$, then there is an approximate form expression of the above problem

$$J_2^{}={\left(W_{Pi}^{\top }{W}_{}\right)}^{-1}{W}_{Pi}^{\top }{G}_{}\text{\hspace{1em}(8.30)}$$

Then $W_{}=\left(I_K-\gamma P_{}\right){Phi}^{\top }$ . While this solution is simple and guaranteed, unfortunately there are no meaningful error bounds that can be used to describe the quality of this approximation.

8.3 Approximate Policy Evaluation with Ergodicity

Although the MSBE minimization problem is well defined and has simple numerical solutions, it inherits the properties of DP, that is, the requirement for model information. In various practical applications of SDM, there is a great need for efficient solutions to problems for which there is no explicit model. Specifically, we study a special class of MDPs, which enables the development of model-free DP algorithms.

8.3.1 Ergodic MDP

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } and a fixed policy$\pi$ , it is well known that system transitions can be modeled as Markov chains. In order to retrieve complete model information by sampling, it must be assumed that each state is reachable from any other state and therefore has a unique stationary distribution over the states. Therefore, we use the underlying MDP model and policyπ \piThe Markov chain of state transitions specified by$\pi \; imposes\; the\; following\; assumptions$

Assumption 8.1 Transition matrix $P_{}$ergodicity of

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } and a fixed policy$\pi$ , by the transition matrix$P_{}$The defined Markov chain is ergodic.

Let us use $X_i$Represents the $The\; probability\; of\; i$ corresponding states. The ergodic hypothesis implies that all$i=1,\dots ,K$的${\xi }_i$都是正定的，也就是说，马尔科夫链有一个唯一的稳定状态分布。让我们定义$\xi :={\left[{\xi }_1,\dots ,{\xi }_K\right]}^{\top }\in {\mathbb{R}}^K$, 与 $x\in {\mathbb{R}}^K$。 $\xi$与过渡矩阵$P_{\pi }$之间的关系的特点是

$$P_{\pi }^{\top }\xi =\xi \text{\hspace{1em}(8.31)}$$

显然，向量$\xi$是$P_{\pi }^{\top }$的右特征向量，与特征值为1有关。此外，由于$\xi$的所有条目都是正的，我们可以将 ξ \xi The weighted norm of$\xi \; is\; defined\; as$

$$\Vert x{\Vert }_{}=\sqrt{\sum _{k=1}^K{X}_i{x}_i^2}\text{\hspace{1em}(8.32)}$$

Lemma 8.4 $\xi$ weighted norm

Given an infinite range MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , and a fixed policy$\pi$ , for any$K\times K$ transition probability matrix$P_{}$, has an invariant distribution $X=\left(x_1,\dots ,X_n\right)$ , has a positive component, we have

$${\Vert P_{}J\Vert }_{}\le \Vert J{\Vert }_{}\text{\hspace{1em}(8.33)}$$

---

Proof
令 P π = { p i j } P_{\pi}=\left\{p_{i j}\right\} Pp={ pij​} , the infinitesimal
$$\begin{array}{rlr}{\Vert P_{}J\Vert }_X^2& ={\sum }_{i=1}^n{X}_i{\left({\sum }_{j=1}^n{p}_{ij}{J}_j\right)}^2& \text{ (definition) }\\ & \le {\sum }_{i=1}^n{X}_i{\sum }_{j=1}^n{p}_{ij}{J}_j^2& \text{ (convexity) }\\ & ={\sum }_{j=1}^n{\sum }_{i=1}^n{X}_i{p}_{ij}{J}_j^2& \\ & ={\sum }_{j=1}^n{X}_j{J}_j^2& \\ \le & \Vert J{\Vert }_X^2& \text{ (definition) }\end{array}\text{\hspace{1em}(8.34)}$$

---

Proposition 8.4 ξ \xi Contractibility of Bellman operator under$\xi \; weighted\; norm$

Given an infinite range of MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , and a fixed policy$\pi$，那么贝尔曼算子${\mathrm{T}}_{\pi }$是模数$\gamma$相对于$\xi$加权范数的收缩，即：
$${\Vert {\mathrm{T}}_{\pi }J-{\mathrm{T}}_{\pi }{J}^{\prime }\Vert }_{\xi }\le \gamma {\Vert J-J^{\prime }\Vert }_{\xi }.\text{\hspace{1em}(8.35)}$$

---

Proof.
为了简单起见，我们使用贝尔曼算子 ${\mathrm{T}}_{\pi }J:=G_{\pi }+\gamma P_{\pi }J$的紧凑表示，然后，我们得到
$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\pi }J-{\mathrm{T}}_{\pi }{J}^{\prime }\Vert }_{\xi }& ={\Vert \gamma P_{\pi }\left(J-J^{\prime }\right)\Vert }_{\xi }\\ & \le \gamma {\Vert J-J^{\prime }\Vert }_{\xi }\end{array}\text{\hspace{1em}(8.36)}$$
这直接来自于Lemma 8.4。

---

通过采用这一特性，我们可以在 ξ \xi The mean square Bellman error (MSBE)$is\; defined\; below\; in\; the\; \xi$ weighted norm.
$$J_b^{}\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert {\mathrm{T}}_{}J-J\Vert }_{}\text{\hspace{1em}(8.37)}$$

Similar to the analysis in Section 8.2, we can derive that the MSBE is minimized in ξ \xiThe error bounds under the$\xi -\; weighted\; specification\; are\; as\; follows.$

Lemma 8.5 ξ \xiBoundary under$\xi \; weighted\; norm$

Given an infinite range MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ }，让$J^{\pi }$ is a fixed policyπ \piThe total cost function of$\pi \; .$Then, for any total cost function$J\in {\mathbb{R}}^{Let\; K}$ , one of the singular inequalities∥
$${\Vert J-J^{\pi }\Vert }_{}\le \frac{1}{1-c}{\Vert J-{\mathrm{T}}_{}J\Vert }_{}\text{\hspace{1em}(8.38)}$$

---

Proof.
Directly

$$\begin{array}{rl}{\Vert J-J^{\pi }\Vert }_{}& ={\Vert J-{\mathrm{T}}_{}J+T_{}J-J^{\pi }\Vert }_{}\\ & ={\Vert J-T_{}J\Vert }_{}+{\Vert {\mathrm{T}}_{}J-J^{\pi }\Vert }_{}\\ & \le {\Vert J-T_{}J\Vert }_{}+c{\Vert J-J^{\pi }\Vert }_{}\end{array}\text{\hspace{1em}(8.39)}$$

---

Proposition 8.5 ξ \xi Constraints between estimates under the$\xi \; -weighted\; norm\; and\; the\; true\; total\; cost\; function$

给出一个无限范围的 M D P { X , U , p , g , γ } M D P\{\mathcal{X}, \mathcal{U}, p, g, \gamma\} MDP{ X,U,p,g,γ}，一个固定的策略$\pi$和一个总成本函数空间$\mathcal{J}$。让$J_B^{\pi }\in \mathcal{J}$为MSBE问题的全局最小值。那么估计值与真实总成本函数$J^{\pi }$之间的误差有如下约束

$${\Vert J_{\beta }^{\pi }-J^{\pi }\Vert }_{\xi }\le \frac{1+}{1-\gamma }\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\pi }\Vert }_{}.\text{\hspace{1em}(8.40)}$$

---

Proof.
By applying the triangle inequality with infinite norm, we can get

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{}J-J\Vert }_{}& \le {\Vert T_{}J-J^{\pi }\Vert }_{}+{\Vert J^{Pi}-J\Vert }_{}\\ & \le (1+c){\Vert J-J^{\pi }\Vert }_{}.\end{array}\text{\hspace{1em}(8.41)}$$

Simply put, we have

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{}{J}_b^{}-J_b^{}\Vert }_{}& =\underset{J\in \mathcal{J}}{\min }{\Vert {\mathrm{T}}_{}J-J\Vert }_{}\\ & \le (1+c)\underset{J\in J}{\min }{\Vert J-J^{\pi }\Vert }_{}\end{array}\text{\hspace{1em}(8.42)}$$

Compare this inequality to $The\; results\; of\; 8.5arecombined$ to complete the proof.

8.3.2 Mean Squared Projected Bellman Error

Finally, if we limit ourselves to a linear function approximation scheme, we need an orthogonal projection to ${\mathcal{J}}_l$, relative to ξ \xiWeighted specification of$\xi \; .$Specifically, we need to solve the following minimization problem
$${Pi}_{}(J):={Phi}^{\top }\underset{h\in {\mathbb{R}}^m}{\mathrm{argmin}}{\Vert J-{Phi}^{\top }h\Vert }_X^2\text{\hspace{1em}(8.43)}$$

Since the least squares function is convex, the solution is characterized by solving the following equation $h$ to realize
$$FX{F}^{\top }h=FXJ\text{\hspace{1em}(8.44)}$$

Since $rk\left(\Phi \right)=m$ , the orthogonal projection is clearly defined as
$${Pi}_{}(J):={Phi}^{\top }{\left(FX{F}^{\top }\right)}^{-1}FXJ\text{\hspace{1em}(8.45)}$$

Lemma 8.6 Non-expansive projection operator ${Pi}_{}$

Given an infinite range MDP {MDP{ X,U,p,g,γ } , and a fixed policy$\pi$ . Then, the projection${Pi}_{}$At ξ − \xi-It is a non-expansive operator under the$\xi$$-\; norm,\; that\; is.$
$${\Vert {\Pi }_{}J-{Pi}_{}{J}^{\prime }\Vert }_{}\le {\Vert J-J^{\prime }\Vert }_{}.\text{\hspace{1em}(8.46)}$$

---

Proof.Easy
to find

$$\begin{array}{rl}{\Vert {\Pi }_{}J-{Pi}_{}{J}^{\prime }\Vert }_X^2& ={\Vert {\Pi }_{}\left(J-J^{\prime }\right)\Vert }_X^2\\ & \le {\Vert {\Pi }_{}\left(J-J^{\prime }\right)\Vert }_X^2+{\Vert \left(I-{Pi}_{}\right)\left(J-J^{\prime }\right)\Vert }_X^2\\ & ={\Vert J-J^{\prime }\Vert }_X^2\end{array}\text{\hspace{1em}(8.47)}$$

The last equation is derived from the Pythagorean theorem. End of proof.

---

Proposition 8.6. Projection operator ${Pi}_{}$shrinkage

Given an infinite range MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , and a fixed policy$\pi$ , then the projected Bellman operator${Pi}_{}{\mathrm{T}}_{}$is relative to $\Vert \cdot {\Vert }_{}$The modulus of is $Shrinkage\; of\; \gamma$ .

---

Proof.

Directly from Lemma $8.6Otherwise$ , there is a constraint∥
$$\begin{array}{rl}{\Vert {\Pi }_{}{\mathrm{T}}_{}J-{Pi}_{}{T}_{}{J}^{\prime }\Vert }_{}& \le {\Vert {\mathrm{T}}_{}J-T_{}{J}^{\prime }\Vert }_{}\\ & \le c{\Vert J-J^{\prime }\Vert }_{}.\end{array}\text{\hspace{1em}(8.48)}$$

---

This proposition shows that in J \mathcal{J}There is a unique fixed pointJ ~ π \widetilde{J}_{\pi}$in\; J$${\tilde{J}}_{}$，从而
$${\tilde{J}}_{\pi }={\Pi }_{\Phi }{\mathrm{T}}_{\pi }{\tilde{J}}_{\pi }.$$

由于$h\mapsto \Phi h$是单射的，因此存在一个唯一的$h_{\pi }\in {\mathbb{R}}^m$，这样$\Phi h_{\pi }={\Pi }_{\Phi }{\mathrm{T}}_{\pi }\left(\Phi h_{\pi }\right)$。这简单地导致了另一个流行的目标函数，即均方投影贝尔曼误差（Mean Squared Projected Bellman Error， MSPBE）。

$$\underset{h\in {\mathbb{R}}^m}{\min }{\Vert \Phi h-{Pi}_{}{\mathrm{T}}_{}(\Phi h)\Vert }_{}\text{\hspace{1em}(8.49)}$$

In the following, we describe the error bounds for minimizing the MSPBE fuction.

Proposition 8.7.

Given an infinite range MDP {X, U, p, g, γ} MDP\{\mathcal{X}, \mathcal{U}, p, g, \gamma\}MDP{ X,U,p,g,γ } , and a fixed policy$\pi$，讹$h_{}$Definition $\Phi h_{}={Pi}_{}{T}_{}(\Phi h_{}$∥J π − Φ ⊤ h π ∥ ξ ≤ 1 1 − γ 2 ∥ J π − Π Φ J π ∥ ξ (8.50) \left\|J^{\pi}-\Phi^{$\backslash$
$${\Vert J^{Pi}-{Phi}^{\top }{h}_{}\Vert }_{}\le \frac{1}{\sqrt{1-c^2}}{\Vert J^{Pi}-{Pi}_{}{J}^{\pi }\Vert }_{}\text{\hspace{1em}(8.50)}$$

---

Proof.
Simply put, we have

$$\begin{array}{rl}{\Vert J^{Pi}-{Phi}^{\top }{h}_{}\Vert }_X^2& ={\Vert J^{Pi}-{Pi}_{}{J}^{\pi }\Vert }_X^2+{\Vert {Pi}_{}{J}^{Pi}-{Phi}^{\top }{h}_{}\Vert }_X^2\\ & ={\Vert J^{Pi}-{Pi}_{}{J}^{\pi }\Vert }_X^2+{\Vert {Pi}_{}{\mathrm{T}}_{}{J}^{Pi}-{Pi}_{}{T}_{}\left(F^{\top }{h}_{}\right)\Vert }_X^2\\ & \le {\Vert J^{Pi}-{Pi}_{}{J}^{\pi }\Vert }_X^2+c^2{\Vert J^{Pi}-{Phi}^{\top }{h}_{}\Vert }_X^2\end{array}\text{\hspace{1em}(8.51)}$$

The first equation is generated by the Pythagorean theorem, the second equation is generated by the construction, and this inequality is due to ${Pi}_{}{T}_{}$caused by shrinkage characteristics.

---

When the true total cost function $J^{When\; \pi }$ is not in the linear function approximation space, that is,${\Vert J^{Pi}-{Pi}_{}{J}^{\pi }\Vert }_{}\ne 0$ ,then${\Vert J^{Pi}-{Phi}^{\top }{h}_{}\Vert }_{}$The error will be severely constrained, if $\gamma$ is close to 1. Therefore, ensure that the total cost function lies in the linear total cost function approximation space$J$ is crucial, that is,$J^{Pi}\in J_l$。

Since both the MSBE function and the MSPBE function are convex, both functions can guarantee a global minimum. Therefore, it is valuable to study the performance of solutions to these two problems. To do this, we define the difference in error bounds as

$$l\left(c\right):=\frac{1+}{1-c}-\frac{1}{\sqrt{1-c^2}}\text{\hspace{1em}(8.52)}$$

Obviously, $l(0)=0$ . Now we takellThe derivative of$l\; is$

$$l^{\prime }\left(c\right)=\frac{2}{(1-c{)}^2}+\frac{}{{\left(\sqrt{1-c^2}\right)}^3}\text{\hspace{1em}(8.53)}$$

Its value is for $c\in [0,1]$ is always positive. This fact means that the difference function$The\; function\; value\; of\; l$ increases monotonically from 0 to 1. The evaluation in Figure 15 clearly describes when$When\; \gamma$ is close to 1, the difference between the error bounds of MSBE minimization and MSPBE minimization becomes infinite. In other words, minimizing the MSPBE function is more advantageous than the MSBE function.

Figure 15: Error bound quotient for MSBE minimization and MSPBE minimization.

8.4 API supplement

8.4.1 Approximate PI (API)

• We will show three different APE methods: $el{l}_2$MSBE, MSBE with ergodic properties, MSPBE with ergodic properties.
• In the case of E-Bus, there is no approximation method in the policy improvement step.
• Policy networks in deep reinforcement learning: Approximate policy improvements.

8.4.2 APE via Bellman Residual Minimisation

• In Policy Iteration, Policy Evaluation (PE) via $T_{\pi }$ leads to a fixed point $J^{\pi }$ . (Quiz 2)
• In Approximate PE, there is a Bellman error since we restrict $J$ in a subspace $\left({\Phi }^{\top }h\right)$ if we apply Linear Function Approximation (LFA).

8.4.3 ${\ell }_2$ Based Bellman Residual Minimisation

• What is the difference between $\Vert \cdot {\Vert }_2^2$ and $\Vert \cdot {\Vert }_2$ ?
• $\Vert x{\Vert }_2^2=x^{\top }x,\Vert x{\Vert }_2=\sqrt{x^{\top }x}\left(x\in {\mathbb{R}}^n\right)$ .
• Both forms are strict convex, they have the same global minima. We did not make a strict distinction between these two terms since we only focus on the analytical solution.
• Quite different in numerical calculations, e.g., gradient.
• In this exercise, we keep using $\Vert \cdot {\Vert }_2^2$ , which is also more consistent with the name ‘Squared’ BE.

8.4.4 Recap: Closed form policy evaluation

Preliminaries: matrix derivation

• Matrix calculus
• Layout conventions: given $y\in R^m,x\in R^n$ .
  Numerator-layout:

$$\begin{array}{l}\text{ Numerator-layout: }\\ \frac{\partial y}{\partial x}:=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \dots & \frac{\partial y_1}{\partial x_n}\\ & \ddots & \\ \frac{\partial y_m}{\partial x_1}& \dots & \frac{\partial y_m}{\partial x_n}\end{array}\right]\in {\mathbb{R}}^{m\times n},\frac{\partial y}{\partial x}:=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \dots & \frac{\partial y_m}{\partial x_1}\\ & \ddots & \\ \frac{\partial y_1}{\partial x_n}& \dots & \frac{\partial y_m}{\partial x_n}\end{array}\right]\in {\mathbb{R}}^{n\times m},\end{array}$$

• This exercise follows denominator layout convention.
• This exercise has two kinds of matrix derivation:
• The derivative of a scalar y by a vector x : gradient (vector)
• The derivative of a vector y by a vector x : Jaccobian (matrix)

8.4.5 ${\ell }_2$ Based Bellman Residual Minimisation

• ${\ell }_2$ least square function:

$$\begin{array}{rl}{J}_2^{}\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}& {\Vert T_{}J-J\Vert }_2^2,\text{ where }J={Phi}^{\top }h.\\ {\Vert {\mathrm{T}}_{}J-J\Vert }_2^2& ={\Vert J-{\mathrm{T}}_{}J\Vert }_2^2={\Vert J-G_{}-\gamma P_{}J\Vert }_2^2\\ & ={\Vert \left(I_K-\gamma P_{}\right){Phi}^{\top }h-G_{}\Vert }_2^2\end{array}$$

• Let $W_{}=\left(I_K-\gamma P_{}\right){Phi}^{\top }\in {\mathbb{R}}^{K\times m}$ , we have

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{}J-J\Vert }_2^2& ={\Vert W_{}h-G_{}\Vert }_2^2\\ & ={\left(W_{}h-G_{}\right)}^T\left(W_{}h-G_{}\right)\end{array}$$

• Since the least square function is convex, we can get the minima when the derivation equals to zero.

• Let $\mathbf{u}=W_{\pi }h-G_{\pi }\in {\mathbb{R}}^{K\times 1}$ , we can get
  $$\frac{\partial {\mathbf{u}}^{\top }\mathbf{u}}{\partial \mathbf{u}}=2\mathbf{u},\frac{\partial {\mathbf{u}}^{\top }\mathbf{u}}{\partial h}=2\frac{\partial \mathbf{u}}{\partial h}\mathbf{u},where\frac{\partial u}{\partial h}=W_{Pi}^{\top }\text{(denominator layout)}$$

$$\begin{array}{rl}\frac{\partial {\left(W_{}h-G_{}\right)}^{\top }\left(W_{}h-G_{}\right)}{\partial h}& =2W_{Pi}^{\top }\left(W_{}h-G_{}\right)=0\in {\mathbb{R}}^{m\times 1}\\ \Rightarrow W_{Pi}^{\top }{W}_{}h-W_{Pi}^{\top }{G}_{}& =0\\ W_{Pi}^{\top }{W}_{}h& =W_{Pi}^{\top }{G}_{}\end{array}$$

• $W_{Pi}^{\top }$ is not a square matrix (non-invertable), so we move $\left(W_{Pi}^{\top }{W}_{}\right)\in {\mathbb{R}}^{m\times m}$ to the RHS:

$$\begin{array}{c}h={\left(W_{Pi}^{\top }{W}_{}\right)}^{-1}{W}_{Pi}^{\top }{G}_{}\\ J_2^{}={Phi}^{\top }h={Phi}^{\top }{\left(W_{Pi}^{\top }{W}_{}\right)}^{-1}{W}_{Pi}^{\top }{G}_{}\end{array}$$

8.4.6 Approximate PI (API) with LFA + MSBE

• What is $x?\to$ Ergodic MDP.
• $X\in R^{K\times K}$ : a diagonal matrix with diagonal element $X_i$ . (The 14th Greek letter $X,x$ )
  
  
• Similar as before, let $W_{}:=\left(I_K-\gamma P_{}\right){Phi}^{\top }\in R^{K\times m}$ :

$${\Vert T_{}\left(F^{\top }h\right)-{Phi}^{\top }h\Vert }_X^2={\Vert {Phi}^{\top }h-G_{}-\gamma P_{}{Phi}^{\top }h\Vert }_X^2={\Vert W_{}h-G_{}\Vert }_X^2$$

• $\xi$ -norm is defined as:

$${\Vert W_{}h-G_{}\Vert }_X^2={\left(W_{}h-G_{}\right)}^{\top }X\left(W_{}h-G_{}\right)$$

• \Xi \in \mathbb{R}^{K \times K} : a diagonal matrix with diagonal element $X_i$ .

• Again, the least square function is convex, the derivation should equal zero. Let $\mathbf{u}=W_{}h-G_{}\in R^{K\times 1}$ , we can get

$$\begin{array}{c}\frac{\partial u^{\top }\Xi \mathbf{u}}{\partial \mathbf{u}}=2\Xi \mathbf{u},\frac{\partial u^{\top }\mathbf{u}}{\partial h}=2\frac{\partial \mathbf{u}}{\partial h}\mathbf{u},\text{ where }\frac{\partial u}{\partial h}=W_{Pi}^{\top }\\ \frac{\partial {\left(W_{}h-G_{}\right)}^{\top }\Xi \left(W_{}h-G_{}\right)}{\partial h}=2W_{Pi}^{\top }X\left(W_{}h-G_{}\right)=0\in R^{m\times 1}\\ W_{Pi}^{\top }\Xi W_{}h=W_{Pi}^{\top }\Xi G_{}\\ h={\left(W_{Pi}^{\top }\Xi W_{}\right)}^{-1}{W}_{Pi}^{\top }\Xi G_{}\\ \Rightarrow J_X^{}={Phi}^{\top }{\left(W_{Pi}^{\top }\Xi W_{}\right)}^{-1}{W}_{Pi}^{\top }\Xi G_{}\end{array}$$

• When $\Xi$ is an identity matrix, we get the same result as ${\ell }_2$MSBE.

8.4.7 $\text{ Approximate PI (API) with LFA }+\xi \text{-weighted MSBE }$

8.4.8 Mean Squared Projected Bellman Error (MSPBE)

• Since ${Pi}_{}J=J$ ,

$$\begin{array}{rl}{\Vert {\Pi }_{}{T}_{}J-J\Vert }_X^2& ={\Vert J-{Pi}_{}\left(G_{}+\gamma P_{}J\right)\Vert }_X^2=\Vert {\Pi }_{}J-c{P}_{}{P}_{}J-{Pi}_{}{G}_{}){\Vert }_X^2,\\ & =\Vert {\Pi }_{}\left(\left(I_K-\gamma P_{}\right){Phi}^{\top }h-G_{}\right)){\Vert }_X^2\end{array}$$

• Let $W_{}=\left(I_K-\gamma P_{}\right){Phi}^{\top }\in R^{K\times m}$ , we have $\Vert {\Pi }_{}\left(W_{}h-G_{}\right)){\Vert }_X^2$ .

• The orthogonal projector ${Pi}_{}:={Phi}^{\top }{\left(FX{F}^{\top }\right)}^{-1}FX\in R^{K\times K}$ .

• Similarly as before, let $\mathbf{u}={Pi}_{}{W}_{}h-{Pi}_{}{G}_{}\in R^{K\times 1}$ , we can get

$$\begin{array}{c}\frac{\partial u^{\top }\Xi \mathbf{u}}{\partial \mathbf{u}}=2\Xi \mathbf{u},\frac{\partial {\mathbf{u}}^{\top }\mathbf{u}}{\partial h}=2\frac{\partial \mathbf{u}}{\partial h}\mathbf{u},\text{ where }\frac{\partial u}{\partial h}={\left(P_{}{W}_{}\right)}^{\top }\\ \frac{\partial {\left(P_{}{W}_{}h-P_{}{G}_{}\right)}^{\top }\Xi \left(P_{}{W}_{}h-P_{}{G}_{}\right)}{\partial h}=2W_{Pi}^{\top }{Pi}_{Phi}^{\top }X{P}_{}\left(W_{}h-G_{}\right)=0\in R^{m\times 1},\end{array}$$

• ${\left(FX{F}^{\top }\right)}^{-1}$ is diagonal, then${Pi}_{Phi}^{\top }=X{F}^{\top }{\left(FX{F}^{\top }\right)}^{-1}\Phi$ , hence we have:

$$\begin{array}{rl}{W}_{Pi}^{\top }\stackrel{{Pi}_{Phi}^{\top }}{\overbrace{X{F}^{\top }{\left(FX{F}^{\top }\right)}^{-1}Phi}}& \stackrel{{Pi}_{}}{\overbrace{{Phi}^{\top }{\left(FX{F}^{\top }\right)}^{-1}FX}}\left(W_{}h-G_{}\right)=0,\\ \underset{\text{full rank, invertable }}{\underbrace{W_{}X{F}^{\top }}}& {\left(FX{F}^{\top }\right)}^{-1}\\ & \Rightarrow X\left(W_{}h-G_{}\right)=0,\\ FX{W}_{}h=FX{G}_{},& \Rightarrow h={\left(FX{W}_{}\right)}^{-1}FX{G}_{},\\ & \Rightarrow J_X^{}={Phi}^{\top }{\left(FX{W}_{}\right)}^{-1}FX{G}_{}.\end{array}$$

• We have proved that ${Pi}_{}{T}_{}$ is a contraction mapping which leads to a fixed point, then the MSPBE should equal to zero:

$${Pi}_{}{T}_{}J-J=0\in R^{K\times 1}\Rightarrow {Pi}_{}\left(W_{}h-G_{}\right))={Phi}^{\top }{\left(FX{F}^{\top }\right)}^{-1}FX\left(W_{}h-G_{}\right))=0$$

• Left multiply with $FX\in R^{m\times K}$ at both sides:

$$\begin{array}{rl}FX{F}^{\top }{\left(FX{F}^{\top }\right)}^{-1}FX\left(W_{}h-G_{}\right))& =\Phi X0,\\ FX\left(W_{}h-G_{}\right))& =0,\\ FX{W}_{}h& =FX{G}_{},\\ \Rightarrow h={\left(FX{W}_{}\right)}^{-1}FX{G}_{},& \\ \Rightarrow J_X^{}={Phi}^{\top }{\left(FX{W}_{}\right)}^{-1}FX{G}_{}.& \end{array}$$

8.4.9 Approximate PI (API) with LFA + $\xi$-weighted MSPBE

8.4.10 Approximate PI Summary

• Three different APE methods in close-form: ${\ell }_2$ MSBE, MSBE with ergodicity, MSPBE with ergodicity;
• The estimation error bound \delta for the above three different APE methods are discussed in the lecture.