ADPRL - Approximate Dynamic Programming and Reinforcement Learning - Note 7 - Approximate Dynamic Programming

7. 近似动态规划 （Approximate Dynamic Programming）

在前面的章节中，我们研究了经典DP算法的理论基础和它们的高级变化。尽管这些算法具有良好的理论特性，但在许多实际应用中，这些算法仍然是低效的，甚至是不切实际的。这种现象主要是由于维数的诅咒，它在存储或计算方面都会造成潜在的高负担。SDM的一个具有挑战性的应用是边缘计算（edge computing），其中计算和数据存储被推到数据源上。显然，对于任何经典的DP算法来说，边缘的计算能力和存储容量都是非常有限的。更具体地说，本节重点讨论维度诅咒的存储角度。

图7：边缘计算应用。

7.1 近似架构 (Approximation architectures)

总成本函数近似的主要思想是构建一个相对低维的参数化空间来近似总成本函数。具体来说，一个参数化的总成本函数近似器将状态$x\in \mathcal{X}$映射为总成本函数估计值，即。

$$J:\mathcal{X}\times {\mathbb{R}}^m\to \mathbb{R},(x,\theta )\mapsto J(x,i)\text{\hspace{1em}(7.1)}$$

Among them $\theta$ is the parameter (vector) of the approximate structure. Through such a structure, the purpose is to change the dimension of the parameter space from$The\; vector\; space\; of\; K$ is reduced to$Vector\; space\; of\; m$ ,$m\ll K$ , which alleviates the curse of dimensionality. It is worth noting that the actual total cost function does not necessarily lie in such an approximation space.

7.1.1 Linear Function Approximation (LFA)

A simple framework for approximating the total cost function is linear function approximation (LFA) . Its purpose is to construct a set of real-valued features that capture the properties of all states, i.e.

$$\varphi :\mathcal{X}\to {\mathbb{R}}^m,x\mapsto \varphi \left(x\right)\text{\hspace{1em}(7.2)}$$

然后将总成本函数近似为这些特征的线性组合，即：
$$J_l:\mathcal{X}\times {\mathbb{R}}^m\to \mathbb{R},(x,\theta )\mapsto {\theta }^{\top }\varphi (x)\text{\hspace{1em}(7.3)}$$

通过将所有特征向量收集到一个矩阵中，即
$$\Phi (x):=\left[\varphi \left(x_1\right),\dots ,\varphi \left(x_K\right)\right]\in {\mathbb{R}}^{m\times K},\text{\hspace{1em}(7.4)}$$

We can determine the space of the approximate total cost function as Φ ( x ) \Phi(x)The span of the rows of$\Phi$$($$x$$)$
$${\mathcal{J}}_l:=\left\{\Phi (x{)}^{\top }h\mid x\in \mathcal{X},h\in {\mathbb{R}}^m\right\}\subset R^K\text{\hspace{1em}(7.5)}$$

Of course, it is assumed that all feature vectors are distinct, so a linear approximation of the total cost function is still distinguishable for inductive strategies. To ensure that every approximation $J\in J_l$All are determined by $h\in R^m$相对于指定的特征矩阵$\varphi (x)$所代表，线性映射$h\mapsto \Phi (x{)}^{\top }h$需要是单射（一对一映射）。因此，我们假设特征矩阵是满秩的。

Assumption 7.1 特征矩阵的秩

特征矩阵$\Phi (x)\in {\mathbb{R}}^{m\times K}$ 的秩是$m$。

有趣的是，经典的DP算法可以被容易地建模为一个线性总成本函数近似。让我们把表格查询特征（tabular lookup features）定义为
$${\varphi }^{\text{table }}(x):={\left(1_{x_1}(x),\dots ,1_{x_K}(x)\right)}^{\top },\text{\hspace{1em}(7.6)}$$

Among them, the indicator function 1 xi : X → { 0 , 1 } 1_{x_{i}}: \mathcal{X} \rightarrow\{0,1\}1xi​​:X→{ 0,1} 被定义为
$$1_{x_i}(x):=\{\begin{array}{ll}1,& \text{ if }x=x_i\\ 0,& \text{ otherwise }\end{array}\text{\hspace{1em}(7.7)}$$

尽管MDP的模型假定状态空间的有限性是$\mathrm{D}\mathrm{P}$和RL求解的基础，但许多工程问题并没有这种连续状态空间的便利，如机器人技术。为了使DP或RL方法在解决具有连续状态空间的MDP问题时都是可行的，建立一种构建线性总成本函数近似空间的机制具有很大的实际用途。最流行的技术之一是瓦片编码（tile coding），如图8所示。

图8：瓦片编码。对于具有连续状态空间的MDP问题，每个圆盘代表一个用于生成特征的局部指标函数。

具体来说，瓦片编码的主要思想是假设相邻的状态具有相同的重要性，因此总成本函数。让我们定义连续状态空间$\mathcal{X}$中的开放子集 ${\mathcal{N}}_k\in \mathcal{X}$, for $k=1,\dots ,\tau$。 我们假设这些开放子集的联合覆盖了状态空间，即。

$$\bigcup _{k=1}^{\tau }{\mathcal{N}}_k=\mathcal{X}\text{\hspace{1em}(7.8)}$$

我们可以定义相同的指示函数为

$$1_{{\mathcal{N}}_i}(x):=\{\begin{array}{ll}1,& \text{ if }x\in {\mathcal{N}}_i\\ 0,& \text{ otherwise }\end{array}$$

然后，可以通过以下方式构建一个瓦片编码特征向量

$${\varphi }^{\text{tile }}(x):={\left(1_{{\mathcal{N}}_1}(x),\dots ,1_{{\mathcal{N}}_{\tau }}(x)\right)}^{\top }.$$

有了线性总成本函数近似的构造，然后就可以直接构造经典的DP算法了。

7.1.2 神经网络函数逼近 (Neural Function Approximation)

深度强化学习（DRL）的最新发展表明，神经网络（NNs）在解决具有大型或连续状态空间的挑战性RL问题方面具有卓越的性能。历史上，神经网络（NN）被用来近似总成本函数已经有几十年了。最近NN在解决模式识别、计算机视觉和语音识别等挑战性问题上的成功，进一步引发了人们对NN应用于总成本函数的努力。基于NN的总成本函数近似（NN-VFA）方法已经在许多具有挑战性的领域中证明了其卓越的性能，例如Atari游戏和围棋游戏。尽管取得了这些进展，对基于NN-VFA的算法的充分理解仍然是一个开放的问题，而且对更具挑战性的应用有着巨大的需求。

一个经典的NN由许多连接的基本计算单元组成，称为神经元，见图9，呈层状结构，如多层感知器（MLP），见图9。

图9：单一感知器神经元的图示。

让$\sigma :\mathbb{R}\to \mathbb{R}$ 是一个单位激活函数，传统上它被选择为非常数、有界、连续和单调增长。常见的例子有Sigmoid函数

$$\sigma (x)=\frac{1}{1+e^{-x}},\text{\hspace{1em}(7.9)}$$

和整流线性单元（ReLU）。

$$\sigma \left(x\right)=\{\begin{array}{ll}0,& \text{ for }x\le 0\\ x,& \text{ for }x>0\end{array}\text{\hspace{1em}(7.10)}$$

More precisely, let us use $L$ represents the number of layers in the MLP structure, using$n_l$Represents the llthThe number of processing units in layer$l$$l=1,\dots ,L.$ _ Specifically, use$l=0$ , we refer to the input layer, and set$n_0=K$，$n_L=1$ . Here, we use$p^{\prime }:\mathbb{R}\to R$ is the activation functionσ \sigmaThe first derivative of$\sigma \; .$

For the ( l , k ) (l, k) in the MLP structure$(l,k)$ unit, refers to thellthkthin layer$l$$k$ units, we define the correspondingunit mappingas

$$f_{l,k}\left(w_{l,k},{\varphi }_{l-1}\right):=p\left(w_{l,k}^{\top }{\varphi }_{l-1}-b_{l,k}\right),\text{\hspace{1em}(7.11)}$$

其中，${\varphi }_{l-1}\in {\mathbb{R}}^{n_{l-1}}$ 表示来自第$(l-1)$层的输出，$w_{l,k}\in {\mathbb{R}}^{n_{l-1}}$ 和$b_{l,k}\in \mathbb{R}$分别是与第$(l,k)$单元相关的权重向量和偏置。请注意，一般来说，偏差$b_{l,k}$是一个与虚拟单元相关的自由变量。然而，通过下面的分析，为了表述方便，我们把它固定为公式（7.11）中的一个常数标量。然后，我们可以通过堆叠该层的所有单元映射，简单地定义第$l$层映射为

$$f_l\left(W_l,{\varphi }_{l-1}\right):={\left[f_{l,1}\left(w_{l,1},{\varphi }_{l-1}\right),\dots ,f_{l,n_l}\left(w_{l,n_l},{\varphi }_{l-1}\right)\right]}^{\top }\text{\hspace{1em}(7.12)}$$

其中$W_l:=\left[w_{l,1},\dots ,w_{l,n_l}\right]\in {\mathbb{R}}^{n_{l-1}\times n_l}$It’s the $l$ weight matrices. Specifically, let us use${\varphi }_0\in {\mathbb{R}}^K$ represents input, then the$The\; output\; of\; layer\; l$ is recursively defined as${\varphi }_l:=f_l\left(W_l,{\varphi }_{l-1}\right)$ Note that the last layer of MLP usually uses self-mapping as the activation function, that is,${\varphi }_L:=W_L^{\top }{\varphi }_{L-1}$。最后，用$W:=R^{K\times n_1}\times \dots \times R^{n_{L-1}}$来表示MLP中所有参数矩阵的集合。我们将所有的层间映射组合起来，定义整个MLP网络的映射

$$f:\mathcal{W}\times {\mathbb{R}}^K\to \mathbb{R},(\mathbf{W},x)\mapsto f_L\left(W_L,\cdot \right)\circ \dots \circ f_1\left(W_1,x\right)\text{\hspace{1em}(7.13)}$$

图10：一个有两个隐藏层的MLP。

有了这样的结构，我们可以定义一组参数化的总成本函数，由一个给定的MLP架构规定为
$$\mathcal{F}:=\left\{f(\mathbf{W},\cdot ):{\mathbb{R}}^K\to R\mid \mathbf{W}\in \mathcal{W}\right\}\text{\hspace{1em}(7.14)}$$

More specifically, we use $\mathcal{F}\left(K,n_1,\dots ,n_{L-1},1\right)$ to represent the architecture of MLP, that is, the number of units in each layer. Let us use$F_x(\mathbf{W}):=f(\mathbf{W},x)$ means the status isxxMLP of$x$$\mathbf{W}$的评估，用$F(x,\mathbf{W}):={\left[F_{x_1}(\mathbf{W}),\dots ,F_{x_K}(\mathbf{W})\right]}^{\top }\in {\mathbb{R}}^K$ represents an approximate total cost function, and the approximate total cost function set of MLP is defined as

J n : = { F ( x , W ) ∣ x ∈ X , W ∈ W } ⊂ R K (7.15) \mathcal{J}_{n}:=\{F(x, \mathbf{W}) \mid x \in \mathcal{X}, \mathbf{W} \in \mathcal{W}\} \subset \mathbb{R}^{K} \tag{7.15} Jn​:={ F(x,W)∣x∈X,W∈W}⊂RK(7.15)

7.2 Bellman Residual Minimization

For the particular approximation architecture of the total cost function being chosen, it is important to investigate the quality and methods to determine the best approximation to the optimal total cost function.

Definition 7.1 Direct estimate of optimal total cost function.

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , and a closed subset$\mathcal{J}\subset {\mathbb{R}}^K$ , then the optimal total cost function$J_D$The direct optimal estimate of is

$$J_D\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert J-J^{\ast }\Vert }_{\infty }\text{\hspace{1em}(7.16)}$$

SolutionJD $J_D$The quality of is measured by the true total cost of its associated GIP.

Proposition 7.1 Constraints on optimal direct estimates

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } and a closed subset$J\subset {\mathbb{R}}^K$，$J_D\in J$ is$J^{}$The optimal direct estimate of${}^{\ast }$${Pi}_D$It's about $J_D$greedy strategy, then we have

$${\Vert J^{{Pi}_D}-J^{\ast }\Vert }_{\infty }\le \frac{2(1+c}{1-c}\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\ast }\Vert }_{\infty }.\text{\hspace{1em}(7.17)}$$

---

Proof.
By applying the triangle inequality with infinite norm, we can get

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\mathfrak{g}}J-J\Vert }_{\infty }& \le {\Vert T_{g}J-J^{\ast }\Vert }_{\infty }+{\Vert J^{\ast }-J\Vert }_{\infty }\\ & \le (1+c){\Vert J-J^{\ast }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(7.18)}$$

According to Proposition $4.2$ Yes$\_$

$$\begin{array}{rl}{\Vert J^{{\pi }_D}-J^{\ast }\Vert }_{\infty }& \le \frac{2}{1-\gamma }{\Vert {\mathrm{T}}_{\mathfrak{g}}{J}_D-J_D\Vert }_{\infty }\\ & \le \frac{2(1+\gamma )}{1-\gamma }{\Vert J_D-J^{\ast }\Vert }_{\infty }\\ & =\frac{2(1+\gamma )}{1-\gamma }\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\ast }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(7.19)}$$

即证。

---

很明显，这种对最佳总成本的直接评估只是理论上的讨论。我们需要开发更可行的机制来寻找最佳近似值。定理3.1中所示的最佳贝尔曼算子的充分和必要条件表明，使用最佳贝尔曼算子下的残余误差来估计最佳总成本函数的可能措施如下

定义 7.2 最佳总成本函数的间接估计 (Indirect estimate of optimal total cost function).

给定一个无限范围的MDP $\mathcal{X},\mathcal{U},p,g,\gamma$，和一个封闭子集$\mathcal{J}\subset {\mathbb{R}}^K$，那么最优总成本函数$J_B$的最优间接估计为
$$J_B\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert {\mathrm{T}}_{\mathfrak{g}}J-J\Vert }_{\infty }\text{\hspace{1em}(7.20)}$$

Attention, $J$与 ${\mathrm{T}}_g$The maximum norm of the difference between$J$ is called the Bellman residual.

Proposition 7.2 Constraints on optimal indirect estimates

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , and let$J_B\in J$ is$J^{\ast }$的最优间接估计，${\pi }_B$是关于$J_B$的贪婪策略。那么我们有

$${\Vert J^{{\pi }_B}-J^{\ast }\Vert }_{\infty }\le \frac{2(1+\gamma )}{1-\gamma }\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\ast }\Vert }_{\infty }.\text{\hspace{1em}(7.21)}$$

---

Proof.
通过应用无穷范数的三角不等式，我们可以得到

$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\mathfrak{g}}J-J\Vert }_{\infty }& \le {\Vert T_{g}J-J^{\ast }\Vert }_{\infty }+{\Vert J^{\ast }-J\Vert }_{\infty }\\ & \le (1+c){\Vert J-J^{\ast }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(7.22)}$$
直接的有
$$\begin{array}{rl}{\Vert {\mathrm{T}}_{\mathfrak{g}}{J}_B-J_B\Vert }_{\infty }& =\underset{J\in \mathcal{J}}{\min }{\Vert T_{g}J-J\Vert }_{\infty }\\ & \le (1+c)\underset{J\in J}{\min }{\Vert J-J^{\ast }\Vert }_{\infty }.\end{array}\text{\hspace{1em}(7.23)}$$

该结果由命题4.2得出。

---

图11：参数化总成本近似的误差界限。

有趣的是，如Propositions 7.1和7.2所示，直接和间接方法所提供的误差界限是相等的。因此，利用贝尔曼残差最小化作为计算工具来估计最优总成本是安全的。另一个有趣的观察表明，只有当最优总成本位于给定的近似集时，两个误差界线才会变成零。显然，来自命题4.3的结果确保了当近似空间被适当构建时，GIP ${\pi }_B$可以是最优的。

Corollary 7.1 最优策略${\pi }_B$的充分不必要约束

给定一个无限范围的MDP $\mathcal{X},\mathcal{U},p,g,\gamma$，并让$J_B\in \mathcal{J}$是$J^{\ast }$的最优间接估计， ${\pi }_B$是关于$J_B$的贪婪策略，最优总成本和任何非最优总成本之间的差距由$\rho :={\min }_{\pi \notin {\mathfrak{P}}_{dm}^{\ast }}{\Vert J^{\pi }-J^{\ast }\Vert }_{\infty }>0$定义。 如果满足以下条件。

$$\underset{J\in \mathcal{J}}{\min }{\Vert J-J^{\ast }\Vert }_{\infty }\le \frac{\rho (1-\gamma )}{2(1+\gamma )}\text{\hspace{1em}(7.24)}$$

那么 ${\pi }_B$It's the best.

7.3 Approximate Value Iteration

It is obvious that the above Bellman residual given in equation (7.20) is difficult to optimize, especially since ${\mathrm{T}}_{\mathfrak{g}}$The evaluation involves discrete optimization. A practical solution is to adopt VI's procedure of approximating the one-step optimal Bellman operator in a given approximate architecture.

Let us adopt the linear function approximation architecture, let $J_k={Phi}^{\top }{\theta }_k$is an estimate of the optimal total cost function. Optimal Bellman operator $T_g$Right $J_k$An application of is not necessarily located in the same subspace $J$ , as depicted in Figure 12.

Figure 12: Fitted value iteration algorithm. From the total cost function $J_k$The $Starting\; with\; k$ estimates, apply the optimal Bellman operator$T_g{J}_k$。

In order to convert the result $T_g{J}_k$Bring back $J_k$回到近似空间$\mathcal{J}$，我们需要应用一个与无穷范数相关的适当的正交投影。具体来说，我们有如下更新规则

$$J_{k+1}={\Pi }_{\infty }{\mathrm{T}}_{\mathfrak{g}}{J}_k\text{\hspace{1em}(7.25)}$$

其中${\Pi }_{\infty }$是关于无穷大准则的正交投影

$${\Pi }_{\infty }(J)={\Phi }^{\top }\underset{\theta \in {\mathbb{R}}^m}{\mathrm{argmin}}{\Vert J-{\Phi }^{\top }\theta \Vert }_{\infty }.\text{\hspace{1em}(7.26)}$$

This projected VI algorithm is called the fitted VI algorithm . It can be proven that the fitting VI converges to a unique fixed point. Unfortunately, it is numerically infeasible to solve the minimization problem under infinite gauges. To alleviate this difficulty, we can exploit the equivalence of norms to develop a numerically feasible algorithm.

As we all know, for a given $x\in {\mathbb{R}}^m$ , the following relationship holds

$$\Vert x{\Vert }_{\infty }\le \Vert x{\Vert }_2\text{\hspace{1em}(7.27)}$$

Then, an approximate VI step can be defined as

$$J_{k+1}\in \underset{J\in \mathcal{J}}{\mathrm{argmin}}{\Vert J-{\mathrm{T}}_{\mathfrak{g}}{J}_k\Vert }_2^2.\text{\hspace{1em}(7.28)}$$

由于我们局限于一个LFA架构，所以很容易采用正交投影的解决方案。让我们定义$\Pi :={\Phi }^{\top }{\left(\Phi {\Phi }^{\top }\right)}^{-1}\Phi$，我们可以像Algorithm 9那样直接构建一个具有LFA的近似VI。

需要注意的是，Algorithm 9不能保证收敛性，这是因为组成的操作符$\left(\Pi T_{\mathfrak{g}}\right):{\mathbb{R}}^K\to {\mathbb{R}}^K$对于$|\cdot {\Vert }_2$或$|\cdot {\Vert }_{\infty }$来说都不是一个收缩。这种现象在近似DP的背景下被称为范数不匹配(norm mismatch) 。

在本节的其余部分，我们研究近似VI的通用框架的收敛性。对于一个给定的估计值$J_k$，一个近似的VI步骤被计算为找到一个新的估计值，但要达到一定的误差，如下面的不等式，并对所有$k=0,1,\dots ,\infty$ 进行计算

$${\Vert J_{k+1}-{\mathrm{T}}_{\mathfrak{g}}{J}_k\Vert }_{\infty }\le \delta .\text{\hspace{1em}(7.29)}$$

在这里，可以用范数之间的等价关系来指定误差界限$\delta$，即：

$${\Vert J_{k+1}-{\mathrm{T}}_{\mathfrak{g}}{J}_k\Vert }_{\infty }\le {\Vert J_{k+1}-T_g{J}_k\Vert }_2.\text{\hspace{1em}(7.30)}$$

Figure 13 illustrates the basic idea of ​​a general AVI algorithm.

Figure 13: Approximation iteration algorithm. Here, each disk around the total cost function estimate symbolizes the error margin of the total cost function approximation.

In the following Proposition, we study its performance in terms of the error or difference between the estimate and the optimal total cost function.

Proposition 7.3 Error bounds for approximate VI algorithm

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , by approximatingVI VIJ k J_{k}produced by$V$$I\; algorithm$$J_k$The sequence satisfies the following inequality

$$\underset{k\to \infty }{\lim }{\Vert J_k-J^{\ast }\Vert }_{\infty }\le \frac{}{1-c}\text{\hspace{1em}(7.31)}$$

---

Proof.

Definition $J_0\in {\mathbb{R}}^K$ is the initial estimate of the total cost function. We can apply the triangle inequality of infinite norm to the$k$ output and kkthexact VI operatorThe difference between$k\; outputs,\; as\; shown\; below$

$$\begin{array}{rl}{\Vert J_k-{\mathrm{T}}_{\mathfrak{g}}^k{J}_0\Vert }_{\infty }=& \Vert J_k-{\mathrm{T}}_{\mathfrak{g}}{J}_{k-1}+{\mathrm{T}}_{\mathfrak{g}}{J}_{k-1}-{\mathrm{T}}_{\mathfrak{g}}^2{J}_{k-2}+\dots \\ & \dots -{\mathrm{T}}_{\mathfrak{g}}^{k-1}{J}_1+{\mathrm{T}}_{\mathfrak{g}}^{k-1}{J}_1-{\mathrm{T}}_{\mathfrak{g}}^k{J}_0{\Vert }_{\infty }\\ \le & {\Vert J_k-{\mathrm{T}}_{\mathfrak{g}}{J}_{k-1}\Vert }_{\infty }+{\Vert {\mathrm{T}}_{\mathfrak{g}}{J}_{k-1}-{\mathrm{T}}_{\mathfrak{g}}^2{J}_{k-2}\Vert }_{\infty }+{\Vert {\mathrm{T}}_{\mathfrak{g}}^{k-1}{J}_1-{\mathrm{T}}_{\mathfrak{g}}^k{J}_0\Vert }_{\infty }\\ \le & \delta +\gamma \delta +\dots +{\gamma }^{k-1}\delta ,\end{array}\text{\hspace{1em}(7.32)}$$

其中第二个不等式是由于最佳贝尔曼算子的收缩特性${\mathrm{T}}_{\mathfrak{g}}$。根据几何级数的特性，我们得到

$${\Vert J_k-{\mathrm{T}}_{\mathfrak{g}}^k{J}_0\Vert }_{\infty }\le \frac{1-{\gamma }^k}{1-\gamma }\delta \text{\hspace{1em}(7.33)}$$

让极限$k\to \infty$，结果由Proposition 3.5中的VI算法的收敛性得出，即${\lim }_{k\to \infty }{\mathrm{T}}_{\mathfrak{g}}^k{J}_0=J^{\ast }$。

---

与经典的VI算法类似，就近似VI算法产生的估计总成本函数而言，可以一系列贪婪诱导性策略。通过回顾Theorem 4.1中GIP的直接误差界限，我们直接得出以下属性

$$\begin{array}{rl}\underset{k\to \infty }{\lim }{\Vert J^{{Pi}_k}-J^{\ast }\Vert }_{\infty }& \le \frac{2}{1-c}\underset{k\to \infty }{\lim }{\Vert J_k-J^{\ast }\Vert }_{\infty }\\ & \le \frac{2c}{(1-c{)}^2}\end{array}\text{\hspace{1em}(7.34)}$$

where $J^{{Pi}_k}$is the kth relative to the approximate VI algorithm$GIP\; for\; k$ iterations.

Corollary 7.2 Boundedness of general AVI algorithm

Given an infinite range MDP {X, U, p, g, γ} \{\mathcal{X}, \mathcal{U}, p, g, \gamma\}{ X,U,p,g,γ } , let$J_k$is the sequence of total cost function estimates produced by the approximate VI algorithm. Let us use ${Pi}_k$Express about $J_k$The corresponding greedy induction strategy, then the total cost function of GIP ${Pi}_k$Satisfy the following properties

$$\underset{k\to \infty }{\lim }{\Vert J^{{Pi}_k}-J^{\ast }\Vert }_{\infty }\le \frac{2c}{(1-c{)}^2}\text{\hspace{1em}(7.35)}$$

Remark 7.1 Tolerance $\delta$ and error$\rho$ relationship

As we have already discussed, the above error bounds can be conservative. In order to be able to retrieve an optimal strategy from the results of the AVI algorithm, we review the results of Proposition 4.3. Let $r>0$ is the error between the best total cost function and its closest total cost function, as defined by formula (4.11). It is expected that the error bound at the limit of the AVI algorithm will obey the following inequality

$$\underset{k\to \infty }{\lim }{\Vert J_k-J^{\ast }\Vert }_{\infty }\le \frac{}{1-c}<\frac{r(1-c}{2c}\text{\hspace{1em}(7.36)}$$

As a result, we have

$$d<\frac{r(1-c{)}^2}{2c}\text{\hspace{1em}(7.37)}$$

Obviously, the tolerance δ \deltaThe choice of$\delta$$The\; total\; cost\; of\; \rho$ , which unfortunately is not available in the general case.

7.3 Example: E-Bus

Consider a group of electric buses running round trips 24 hours a day. The task is to identify optimal operating actions at different battery states. The battery’s endurance and charging speed gradually decrease with the increase of battery life. Hence, for different buses, they have different transition probabilities between battery states. The following figure illustrates the state transitions between different states.

• Five states: $H$ - high battery, $E$ - empty battery, $L_1,L_2,L_3$ : three different low level battery statuses
• Two actions: $S$- continue to serve, C - charge
• Numbers on the edges refer to transition probabilities. $p_1=0.4,p_2=0.6$
• Discount factor $\gamma =0.9$ .

We choose the number of unserviced passengers as the local costs:

• In the high battery state, if it keeps the service, the unserviced passenger number is 0 .
• In all the low battery stats, if it keeps the service, the unserviced passenger number is 2 .
• In all the low battery state and empty battery, if it charges the battery, the unserviced passenger number is 5 .

Task 1: Implement Value Iteration (VI) and Approximate Value Iteration (AVI) algorithm with Linear Function approximation (LFA) using the feature matrix

$$Phi=\left[\begin{array}{ccccc}-0.40& -0.44& -0.45& -0.46& -0.48\\ -0.07& 0.73& -0.61& 0.21& -0.23\end{array}\right]$$

Compare their convergence speeds in terms of the difference from each iterate to the corresponding accumulation point ${\Vert J_k-J^{\ast }\Vert }_{\infty }$ against the index of sweep $k$ .

Task 2: Given the gap between the optimal total cost and any non-optimal total cost ($\rho :={\min }_{\pi \notin {\mathfrak{P}}_{dm}^{\ast }}{\Vert J^{\pi }-J^{\ast }\Vert }_{\infty }=0.239$ ), check conditions on optimality in policy according to Eq. (7.24).

Task 3: If we use a random-generated feature matrix $\Phi$ , repeat the aforementioned two tasks, what can we observe? Discussion: how do we choose the feature matrix $\Phi$ .

7.3.1 相关代码

import random
import matplotlib.pyplot as plt
import matplotlib
import numpy as np

matplotlib.rcParams.update(matplotlib.rcParamsDefault)

INIT_J = 0  # define initial total cost
ITER_N = 100  # the number of iterations

# transition probability
p1 = 0.4
p2 = 0.6

# cost of each state-action pair
ghs = 0
gl1s = gl2s = gl3s = 2
gec = gl3c = gl2c = gl1c = 5

# discount factor
gamma = 0.9

##
# using VI to get optimal_J. i.e. J*:
k = 0
jh = jl1 = jl2 = jl3 = je = INIT_J
while k < 200:
    jh_ = p1 * (ghs + gamma * jl1) + p2 * (ghs + gamma * jl2)
    jl1_ = min(p1 * (gl1s + gamma * jl2) + p2 * (gl1s + gamma * jl3), gl1c + gamma * jh)
    jl2_ = min(p1 * (gl2s + gamma * jl3) + p2 * (gl2s + gamma * je),
               p1 * (gl2c + gamma * jl1) + p2 * (gl2c + gamma * jh))
    jl3_ = min(gl3s + gamma * je, p1 * (gl3c + gamma * jl2) + p2 * (gl3c + gamma * jl1))
    je_ = p1 * (gec + gamma * jl3) + p2 * (gec + gamma * jl2)

    jh, jl1, jl2, jl3, je = jh_, jl1_, jl2_, jl3_, je_
    k += 1

jh_optim, jl1_optim, jl2_optim, jl3_optim, je_optim = jh, jl1, jl2, jl3, je
print('After VI we get the optimal costs {:.4f} {:.4f} {:.4f} {:.4f} {:.4f}'.format(jh_optim, jl1_optim, jl2_optim,
                                                                                    jl3_optim, je_optim))


##
# VI
jh = jl1 = jl2 = jl3 = je = INIT_J
vi_converge = []

for k in range(ITER_N):
    jh_ = p1 * (ghs + gamma * jl1) + p2 * (ghs + gamma * jl2)
    jl1_ = min(p1 * (gl1s + gamma * jl2) + p2 * (gl1s + gamma * jl3), gl1c + gamma * jh)
    jl2_ = min(p1 * (gl2s + gamma * jl3) + p2 * (gl2s + gamma * je),
               p1 * (gl2c + gamma * jl1) + p2 * (gl2c + gamma * jh))
    jl3_ = min(gl3s + gamma * je, p1 * (gl3c + gamma * jl2) + p2 * (gl3c + gamma * jl1))
    je_ = p1 * (gec + gamma * jl3) + p2 * (gec + gamma * jl2)

    jh, jl1, jl2, jl3, je = jh_, jl1_, jl2_, jl3_, je_
    k += 1
    # calculate the infinite norm of ||J_k - J*||
    infinity_gap = max(
        abs(jh - jh_optim),
        abs(jl1 - jl1_optim),
        abs(jl2 - jl2_optim),
        abs(jl3 - jl3_optim),
        abs(je - je_optim)
    )
    vi_converge.append(infinity_gap)

plt.plot(range(ITER_N), vi_converge, 'o-', label='VI')

plt.ylabel(r"$\|\| J - J^*\|\|_{\infty}$")
plt.xlabel('Iteration k')

##
# Approximate value iteration using specific features

phi = np.array([[-0.4, -0.44, -0.45, -0.46, -0.48],
                [-0.07, 0.73, -0.61, 0.21, -0.23]])
inv = np.linalg.inv(phi @ phi.T)
PI = phi.T @ inv @ phi
# boundary of optimal indirect estimate
rho = 0.239
boundary = rho * (1 - gamma) / (2 * (1 + gamma))
print('Upper bound  = {}, '.format(boundary))

J_AVI = np.array([0, 0, 0, 0, 0])
J_OPT = np.array([jh_optim, jl1_optim, jl2_optim, jl3_optim, je_optim])
print('\t and the left hand side = {}'.format(min(abs(PI @ J_OPT - J_OPT))))
avi_converge = []
for k in range(ITER_N):
    # TgJk
    jh, jl1, jl2, jl3, je = J_AVI
    jh_ = p1 * (ghs + gamma * jl1) + p2 * (ghs + gamma * jl2)
    jl1_ = min(p1 * (gl1s + gamma * jl2) + p2 * (gl1s + gamma * jl3), gl1c + gamma * jh)
    jl2_ = min(p1 * (gl2s + gamma * jl3) + p2 * (gl2s + gamma * je),
               p1 * (gl2c + gamma * jl1) + p2 * (gl2c + gamma * jh))
    jl3_ = min(gl3s + gamma * je, p1 * (gl3c + gamma * jl2) + p2 * (gl3c + gamma * jl1))
    je_ = p1 * (gec + gamma * jl3) + p2 * (gec + gamma * jl2)
    J_AVI = np.array([jh_, jl1_, jl2_, jl3_, je_])

    # orthogonal projector
    J_AVI = PI @ J_AVI
    if min(abs(J_AVI - J_OPT)) <= boundary:
        print('After {} iterations, the policy is optimal'.format(k))
    avi_converge.append(max(abs(J_AVI - J_OPT)))

plt.plot(range(ITER_N), avi_converge, '^-', label='AVI')

##
# Approximate value iteration using random features

phi = np.random.rand(2, 5)
inv = np.linalg.inv(phi @ phi.T)
PI = phi.T @ inv @ phi
# boundary of optimal indirect estimate
rho = 0.239
boundary = rho * (1 - gamma) / 2 * (1 + gamma)

J_AVI_rand = np.array([0, 0, 0, 0, 0])
avi_converge_rand = []
for k in range(ITER_N):
    # TgJk
    jh, jl1, jl2, jl3, je = J_AVI_rand
    jh_ = p1 * (ghs + gamma * jl1) + p2 * (ghs + gamma * jl2)
    jl1_ = min(p1 * (gl1s + gamma * jl2) + p2 * (gl1s + gamma * jl3), gl1c + gamma * jh)
    jl2_ = min(p1 * (gl2s + gamma * jl3) + p2 * (gl2s + gamma * je),
               p1 * (gl2c + gamma * jl1) + p2 * (gl2c + gamma * jh))
    jl3_ = min(gl3s + gamma * je, p1 * (gl3c + gamma * jl2) + p2 * (gl3c + gamma * jl1))
    je_ = p1 * (gec + gamma * jl3) + p2 * (gec + gamma * jl2)
    J_AVI_rand = np.array([jh_, jl1_, jl2_, jl3_, je_])

    # orthogonal projector
    J_AVI_rand = PI @ J_AVI_rand
    if min(abs(J_AVI_rand - J_OPT)) <= boundary:
        print('After {} iterations, the policy is optimal'.format(k))
    avi_converge_rand.append(max(abs(J_AVI_rand - J_OPT)))

plt.plot(range(ITER_N), avi_converge_rand, '^-', label='AVI')


plt.legend()
plt.show()

7.3.2 输出结果

After VI we get the optimal costs 26.1268 28.5141 29.3736 30.7331 31.9256
Upper bound  = 0.006289473684210525, 
	 and the left hand side = 0.20402763408068125
After 44 iterations, the policy is optimal

7.3.3 Feature matrix $Phi$

• Calculate $J^{\pi }$ for all the possible policies. $J^{Pi}\in {\mathbb{R}}^{8\times 5}$
  So, how to establish a $J^{\pi }$ for all the possible policies?
• Singular value decomposition (SVD): $J^{Pi}={\mathbf{U}\mathbf{\Sigma }\mathbf{V}}^{\ast }$.
• Select the first two component of ${\mathbf{V}}^{\ast }$ as$Phi$