RL-赵-(二)-基于模型：贝尔曼/Bellman公式【用于计算给定π下的StateValue：①线性方程组法、②迭代法】、Action Value【根据状态值求解得到；进而用来评价action优劣】

State Value ：the average Return that an agent can obtain if it follows a given policy/π【给定一个policy/π，所有可能的trajectorys得到的所有return的平均值/期望值：$v_{\pi }(s)\doteq \mathbb{E}[G_t|S_t=s]$】.
Return：the discounted sum of all the rewards collected along a trajectory【return 等于沿着一个特定的 trajectory 收集到的所有 rewards 的折现总和：$G_t\doteq R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots ,$】

$v_{\pi }(s)$：从状态 $s$ 出发，遵循策略 $\pi$ 能够获得的期望回报。

• 从给定状态 $s_i$ 出发基于给定的policy/π所包含的trajectorys之所以有多个，是因为在每个state时agent接下来要采取的action是按该policy下的概率来采样的，如下图：
  如果每一个state处所采取的action是固定的（以箭头表示），则每一个state在该policy中只有一个固定的trajectory；
  

• $v_{\pi }(s)$ depends on $s$：This is because its defnition is a conditional expectation with the condition that the agent starts from $S_t=s.$

• $v_{\pi }(s)$ depends on $\pi$：This is because the trajectories are generated by following the policy $\pi$. For a different policy, the state value may be different.【不同的policy/π得到的trajectories集合是不一样的】

• $v_{\pi }(s)$ does not depend on $t$：If the agent moves in the state space, $t$ represents the current time step. The value of ${\upsilon }_{\pi }(s)$ is determined once the policy is given.

state values和returns之间的关系进一步明确如下：

• When both the policy and the system model are deterministic, starting from a state always leads to the same trajectory. In this case, the return obtained starting from a state is equal to the value of that state. 【当 policy 和 system model 都是确定性的时候，从一个状态开始总是会导致相同的轨迹。在这种情况下，从一个状态开始获得的return等于该状态的state value】
• By contrast, when either the policy or the system model is stochastic, starting from the same state may generate different trajectories. In this case, the returns of different trajectories are different, and the state value is the mean of these returns.【相反，当 policy 或 system model 中的任何一个是随机的时候，从相同的 state 开始可能会产生不同的trajectory。在这种情况下，不同trajectory的return是不同的，而state value是这些return的平均值。】
• The greater the state value is, the better the corresponding policy is.
• State values can be used as a metric to evaluate whether a policy is good or not.

While state values are important, how can we analyze them?

• The answer is the Bellman equation（贝尔曼方程）, which is an important tool for analyzing state values.
• In a nutshell, the Bellman equation describes the relationships between the values of all states.【贝尔曼方程描述了所有状态值之间的关系。】
• By solving the Bellman equation, we can obtain the state values. This process is called policy evaluation, which is a fundamental concept in reinforcement learning. 【通过解决贝尔曼方程，我们可以得到状态值。这个过程被称为策略评估，是强化学习中的基本概念。】

Finally, this chapter introduces another important concept called the action value.

2.1 Motivating example 1: Why are returns important?

The previous chapter introduced the concept of returns. In fact, returns play a fundamental role in reinforcement learning since they can evaluate whether a policy is good or not. This is demonstrated by the following examples.

Consider the three policies shown in Figure 2.2. It can be seen that the three policies are different at s1. Which is the best and which is the worst? Intuitively,

• the leftmost policy is the best because the agent starting from $s_1$ can avoid the forbidden area. 【最左边的策略是最好的，因为从$s_1$开始的agent可以避开禁区】
• The middle policy is intuitively worse because the agent starting from $s_1$ moves to the forbidden area. 【中间的策略直观上更差，因为从$s_1$开始的agent会移动到禁区】
• The rightmost policy is in between the others because it has a probability of 0.5 to go to the forbidden area.【最右边的策略介于其他两者之间，因为它 有0.5的概率进入禁区】

While the above analysis is based on intuition, a question that immediately follows is whether we can use mathematics to describe such intuition. The answer is yes and relies on the return concept. 【尽管上述分析是基于直觉的，但紧随其后的一个问题是我们是否可以使用数学来描述这种直觉。答案是肯定的，这依赖于return的概念】

In particular, suppose that the agent starts from $s_1$.

• Following the first policy, the trajectory is s1 → s3 → s4 → s4 · · · . The corresponding discounted return is
  $$\begin{array}{rl}retur{n}_1& =0+\gamma 1+{\gamma }^{2}1+\dots \\ & =\gamma (1+\gamma +{\gamma }^2+\dots )\\ & =\frac{\gamma }{1-\gamma },\end{array}$$
  where $\gamma \in (0,1)$ is the discount rate.
• Following the second policy, the trajectory is s1 → s2 → s4 → s4 · · · . The discounted return is

$$\begin{array}{rl}retur{n}_2& \begin{array}{r}=-1+\gamma 1+{\gamma }^{2}1+\dots \end{array}\\ & =-1+\gamma (1+\gamma +{\gamma }^2+\dots )\\ & =-1+\frac{\gamma }{1-\gamma }.\end{array}$$

• Following the third policy, two trajectories can possibly be obtained. One is s1 → s3 → s4 → s4 · · · , and the other is s1 → s2 → s4 → s4 · · · . The probability of either of the two trajectories is 0.5. Then, the average return that can be obtained starting from $s_1$ is

$$\begin{array}{rl}retur{n}_3& \begin{array}{r}=0.5\left(-1+\frac{\gamma }{1-\gamma }\right)+0.5\left(\frac{\gamma }{1-\gamma }\right)\end{array}\\ & =-0.5+\frac{\gamma }{1-\gamma }.\end{array}$$

By comparing the returns of the three policies, we notice that：
$$retur{n}_1>retur{n}_3>retur{n}_2$$

for any value of γ. $retur{n}_1>retur{n}_3>retur{n}_2$ suggests that the first policy is the best because its return is the greatest, and the second policy is the worst because its return is the smallest.

This mathematical conclusion is consistent with the aforementioned intuition: the first policy is the best since it can avoid entering the forbidden area, and the second policy is the worst because it leads to the forbidden area.

2.2 Motivating example 2: How to calculate returns?

While we have demonstrated the importance of returns, a question that immediately follows is how to calculate the returns when following a given policy.

There are two ways to calculate returns. 【有两种计算return的方法】

• The first is simply by definition: a return equals the discounted sum of all the rewards collected along a trajectory【return 等于沿着一个 trajectory 收集到的所有奖励的折现总和】. Consider the example in Figure 2.3. Let $v_i$ denote the return obtained by starting from $s_i$ for i = 1, 2, 3, 4. Then, the returns obtained when starting from the four states in Figure 2.3 can be calculated as
  
• The second way, which is more important, is based on the idea of bootstrapping. By observing the expressions of the returns in (2.2), we can rewrite them as
  
  The above equations indicate an interesting phenomenon that the values of the returns rely on each other. More specifically, v1 relies on v2, v2 relies on v3, v3 relies on v4, and v4 relies on v1. This reflects the idea of bootstrapping, which is to obtain the values of some quantities from themselves.

At first glance, bootstrapping is an endless loop because the calculation of an unknown value relies on another unknown value. In fact, bootstrapping is easier to understand if we view it from a mathematical perspective. In particular, the equations in (2.3) can be reformed into a linear matrix-vector equation:

which can be written compactly as
$$v=r+\gamma Pv.$$
Thus, the value of $v$ can be calculated easily as $v=(I-\gamma P{)}^{-1}r$, where $I$ is the identity matrix with appropriate dimensions. One may ask whether $I-\gamma P$ is always invertible. The answer is yes and explained in Section 2.7.1.

In fact, (2.3) is the Bellman equation for this simple example. Although it is simple, (2.3) demonstrates the core idea of the Bellman equation: the return obtained by starting from one state depends on those obtained when starting from other states.

2.3 State values

We mentioned that returns can be used to evaluate policies. However, they are inapplicable to stochastic systems because starting from one state may lead to different returns.【我们提到过returns可以用来评估policies。然而，在随机系统中，returns是不适用的，因为从一个state出发可能会导致不同的returns】

Motivated by this problem, we introduce the concept of state value in this section.

First, we need to introduce some necessary notations.

Consider a sequence of time steps $t=0,1,2,\dots$ At time $t$, the agent is at state $S_t$, and the action taken following a policy $\pi$ is $A_t.$ The next state is $S_{t+1}$, and the immediate reward obtained is $R_{t+1}.$ This process can be expressed concisely as

$$S_t\stackrel{A_t}{\to }{S}_{t+1},R_{t+1}$$

Note that $S_t,S_{t+1},A_t,R_{t+1}$ are all random variables. Moreover, $S_t,S_{t+1}\in \mathcal{S},A_t\in \mathcal{A}(S_t)$, and $R_{t+1}\in \mathcal{R}(S_t,A_t)$.

Starting from $t$, we can obtain a state-action-reward trajectory:

$$\begin{array}{r}{S}_t\stackrel{A_t}{\to }{S}_{t+1},R_{t+1}\stackrel{A_{t+1}}{\to }{S}_{t+2},R_{t+2}\stackrel{A_{t+2}}{\to }{S}_{t+3},R_{t+3}\dots .\end{array}$$

By definition, the discounted return along the trajectory【注：在给定π条件下，可能有多个trajectory】 is
$$G_t\doteq R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots ,$$
where γ ∈ (0, 1) is the discount rate. Note that Gt is a random variable since $\begin{array}{r}{R}_{t+1},R_{t+2},\dots \end{array}$ are all random variables.

Since $G_t$ is a random variable, we can calculate its expected value (also called the expectation or mean):

$$v_{\pi }(s)\doteq \mathbb{E}[G_t|S_t=s]$$

Here, $v_{\pi }(s)$ is called the state-value function or simply the state value of s.

Some important remarks are given below.

• $v_{\pi }(s)$ depends on $s$：This is because its defnition is a conditional expectation with the condition that the agent starts from $S_t=s.$
• $v_{\pi }(s)$ depends on $\pi$：This is because the trajectories are generated by following the policy $\pi$. For a different policy, the state value may be different.【不同的policy/π得到的trajectories集合是不一样的】
• $v_{\pi }(s)$ does not depend on $t$：If the agent moves in the state space, $t$ represents the current time step. The value of ${\upsilon }_{\pi }(s)$ is determined once the policy is given.

state values和returns之间的关系进一步明确如下：

• When both the policy and the system model are deterministic, starting from a state always leads to the same trajectory. In this case, the return obtained starting from a state is equal to the value of that state. 【当 policy 和 system model 都是确定性的时候，从一个状态开始总是会导致相同的轨迹。在这种情况下，从一个状态开始获得的return等于该状态的state value】
• By contrast, when either the policy or the system model is stochastic, starting from the same state may generate different trajectories. In this case, the returns of different trajectories are different, and the state value is the mean of these returns.【相反，当 policy 或 system model 中的任何一个是随机的时候，从相同的 state 开始可能会产生不同的trajectory。在这种情况下，不同trajectory的return是不同的，而state value是这些return的平均值。】

2.4 Bellman equation【贝尔曼公式】

Bellman equation, a mathematical tool for analyzing state val- ues. In a nutshell, the Bellman equation is a set of linear equations that describe the relationships between the values of all the states.【贝尔曼方程，这是一种分析状态值的数学工具。简而言之，贝尔曼方程是一组线性方程，描述了所有状态值之间的关系。】

贝尔曼公式推导：

• First, note that $G_t$ can be rewritten as
  $$\begin{array}{rl}{G}_t& \begin{array}{r}=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots \end{array}\\ & =R_{t+1}+\gamma (R_{t+2}+\gamma R_{t+3}+\dots )\\ & =R_{t+1}+\gamma G_{t+1}\end{array}$$

where $\begin{array}{r}{G}_{t+1}=R_{t+2}+\gamma R_{t+3}+\dots .\end{array}$ This equation establishes the relationship between $G_t$ and $G_{t+1}$.

State Value 表示为(2.4) ：
$$\begin{array}{rl}{\upsilon }_{\pi }(s)& =\mathbb{E}[G_t|S_t=s]\\ & =\mathbb{E}[R_{t+1}+\gamma G_{t+1}|S_t=s]\\ & =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\end{array}$$

• 第一项 $\mathbb{E}[R_{t+1}|S_t=s]$ 表示 the expectation of immediate rewards，根据the law of total expectation，可被改写为(2.5)：
  $$\begin{array}{rl}\mathbb{E}[R_{t+1}|S_t=s]& =\sum _{a\in \mathcal{A}}\pi (a|s)\mathbb{E}[R_{t+1}|S_t=s,A_t=a]\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r\end{array}$$
  • Here, $A$ and $R$ are the sets of possible actions and rewards, respectively.
  • It should be noted that A may be different for different states.
  • In this case, $A$ should be written as $A(s)$. Similarly, $R$ may also depend on $(s,a)$.
  • We drop the dependence on $s$ or $(s,a)$ for the sake of simplicity in this book. 【为了书写简洁】
  • Nevertheless, the conclusions are still valid in the presence of dependence.
• 第二项 $\mathbb{E}[G_{t+1}|S_t=s]$ ,表示 the expectation of the future rewards，可被改写为：
  $$\begin{array}{rl}\mathbb{E}[G_{t+1}|S_t=s]& =\sum _{s^{\prime }\in \mathcal{S}}\mathbb{E}[G_{t+1}|S_t=s,S_{t+1}=s^{\prime }]p(s^{\prime }|s)\\ & =\sum _{s^{\prime }\in \mathcal{S}}\mathbb{E}[G_{t+1}|S_{t+1}=s^{\prime }]p(s^{\prime }|s)\text{(due to the Markov property)}\\ & =\sum _{s^{\prime }\in \mathcal{S}}{v}_{\pi }(s^{\prime })p(s^{\prime }|s)\\ & =\sum _{s^{\prime }\in \mathcal{S}}{v}_{\pi }(s^{\prime })\sum _{a\in \mathcal{A}}p(s^{\prime }|s,a)\pi (a|s)\end{array}$$
  • The above derivation uses the fact that $\mathbb{E}[G_{t+1}|S_t=s,S_{t+1}=s^{\prime }]=\mathbb{E}[G_{t+1}|S_{t+1}=s^{\prime }]$，which is due to the Markov property that the future rewards depend merely on the present state rather than the previous ones.
  • $s^{\prime }$ 表示 $s$ 的下一个时刻的状态；

将(2.5)-(2.6)代入(2.4)，得公式(2.7)：

$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =\underset{\text{mean of immediate rewards}}{\underbrace{\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r}}+\underset{\text{mean of future rewards}}{\underbrace{\gamma \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }),}}\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\right],\text{for all }s\in \mathcal{S}\end{array}$$

This equation is the Bellman equation, which characterizes the relationships of state values. It is a fundamental tool for designing and analyzing reinforcement learning algorithms.【这个方程是贝尔曼方程，它描述了state value之间的关系。它是设计和分析强化学习算法的基本工具。】

• $v_{\pi }(s)$ and $v_{\pi }(s^{\prime })$ are unknown state values to be calculated.
  • the Bellman equation refers to a set of linear equations for all states rather than a single equation.
  • If we put these equations together, it becomes clear how to calculate all the state values.
• $\pi (a|s)$ is a given policy.
  • Since state values can be used to evaluate a policy, solving the state values from the Bellman equation is a $policyevaluation$ process.
• $p(r|s,a)$ and $p(s^{\prime }|s,a)$ represent the system model.
  • We will first show how to calculate the state values $with$ this model and then show how to do that $without$ the model by using model-free algorithms later in this book.

In addition to the expression in (2.7), readers may also encounter other expressions of the Bellman equation in the literature. We next introduce two equivalent expressions.

• First, it follows from the law of total probability that
  $$\begin{array}{rl}p(s^{\prime }|s,a)& =\sum _{r\in \mathcal{R}}p(s^{\prime },r|s,a),\\ p(r|s,a)& =\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime },r|s,a).\end{array}$$
  Then, equation (2.7) can be rewritten as
  $$\begin{array}{r}{v}_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}\sum _{r\in \mathcal{R}}p(s^{\prime },r|s,a)\left[r+\gamma v_{\pi }(s^{\prime })\right]\end{array}$$
• Second, the reward $r$ may depend solely on the next state $s^{\prime }$ in some problems. As a result, we can write the reward as $r(s^{\prime })$ and hence $p(r(s^{\prime })|s,a)=p(s^{\prime }|s,a)$, substituting which into (2.7) gives
  $$\begin{array}{rl}{v}_{\pi }(s)& =\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)\left[r(s^{\prime })+\gamma v_{\pi }(s^{\prime })\right]\end{array}$$

2.5 Examples for illustrating the Bellman equation

We next use two examples to demonstrate how to write out the Bellman equation and calculate the state values step by step. Readers are advised to carefully go through the examples to gain a better understanding of the Bellman equation.【我们接下来使用两个例子来演示如何写出贝尔曼方程并逐步计算状态值。建议读者仔细阅读这些例子，以更好地理解贝尔曼方程。】

If we compare the state values of the two policies in the above examples, it can be seen that：【如果我们比较下述示例中两种策略的状态值，可以看到】
$$v_{{\pi }_1}(s_i)\ge v_{{\pi }_2}(s_i),i=1,2,3,4,$$

• which indicates that the policy in Figure 2.4 is better because it has greater state values.【这表明图2.4中的策略更好，因为它具有更大的状态值。】
• This mathematical conclusion is consistent with the intuition that the first policy is better because it can avoid entering the forbidden area when the agent starts from $s_1$. 【这个数学结论与直觉一致，即第一个policy更好，因为当agent 从$s_1$开始时，它可以避免进入禁区。】
• As a result, the above two examples demonstrate that state values can be used to evaluate policies.【这两个例子证明了state value可以 用来评估policy】

2.5.1 example01【policy是确定性的】

Consider the first example shown in Figure 2.4, where the policy is deterministic【policy是确定性的】.

We next write out the Bellman equation and then solve the state values from it.

First, consider state $s_1.$ Under the policy：

• The probabilities of taking the actions are $\pi (a=a_3|s_1)=1$ and $\pi (a\ne a_3|s_1)=0$.
• The state transition probabilities are $p(s^{\prime }=s_3|s_1,a_3)=1$ and $p(s^{\prime }\ne s_3|s_1,a_3)=0.$
• The reward probabilities are $p(r=0|s_1,a_3)=1$ and $p(r\ne 0|s_1,a_3)=0.$

Substituting these values into
$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =\underset{\text{mean of immediate rewards}}{\underbrace{\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r}}+\underset{\text{mean of future rewards}}{\underbrace{\gamma \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }),}}\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\right],\text{for all }s\in \mathcal{S}\end{array}$$

gives
$$v_{\pi }(s_1)=0+\gamma v_{\pi }(s_3)$$

Similarly, it can be obtained that
$$\begin{array}{rl}& {\upsilon }_{\pi }(s_2)=1+\gamma {\upsilon }_{\pi }(s_4)\\ & v_{\pi }(s_3)=1+\gamma {\upsilon }_{\pi }(s_4)\\ & {\upsilon }_{\pi }(s_4)=1+\gamma {\upsilon }_{\pi }(s_4)\end{array}$$

We can solve the state values from these equations. Since the equations are simple, we can manually solve them. Here, the state values can be solved as：【我们可以从这些方程中解出状态值。由于这些方程很简单，我们可以手动解决它们。在这里，状态值可以被解决为】
$$\begin{array}{c}{\upsilon }_{\pi }(s_4)=\frac{1}{1-\gamma },\\ {\upsilon }_{\pi }(s_3)=\frac{1}{1-\gamma },\\ {\upsilon }_{\pi }(s_2)=\frac{1}{1-\gamma },\\ {\upsilon }_{\pi }(s_1)=\frac{\gamma }{1-\gamma }.\end{array}$$

Furthermore, if we set $\gamma =0.9$, then
$$\begin{array}{c}{\upsilon }_{\pi }(s_4)\begin{array}{r}=\frac{1}{1-0.9}=10,\end{array}\\ {\upsilon }_{\pi }(s_3)=\frac{1}{1-0.9}=10,\\ {\upsilon }_{\pi }(s_2)=\frac{1}{1-0.9}=10,\\ {\upsilon }_{\pi }(s_1)=\frac{0.9}{1-0.9}=9.\end{array}$$

2.5.2 example02【policy是随机的】

Consider the second example shown in Figure 2.5, where the policy is stochastic【policy是随机的】.

We next write out the Bellman equation and then solve the state values from it.

First, consider state $s_1.$ Under the policy：

• At state $s_1$, the probabilities of going right and down equal 0.5. The probabilities of taking the actions are $\pi (a=a_2|s_1)=0.5$ and $\pi (a=a_3|s_1)=\mathrm{0.5.}$
• The state transition probability is deterministic since $p(s^{\prime }=s_3|s_1,a_3)=1$ and $p(s^{\prime }=s_2|s_1,a_2)=1.$
• The reward probability is also deterministic since $p(r=0|s_1,a_3)=1$ and $p(r=-1|s_1,a_2)=1.$

Substituting these values into
$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =\underset{\text{mean of immediate rewards}}{\underbrace{\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r}}+\underset{\text{mean of future rewards}}{\underbrace{\gamma \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }),}}\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\right],\text{for all }s\in \mathcal{S}\end{array}$$

gives
$$\begin{array}{r}{v}_{\pi }(s_1)=0.5[0+\gamma v_{\pi }(s_3)]+0.5[-1+\gamma v_{\pi }(s_2)]\end{array}$$

Similarly, it can be obtained that

$$\begin{array}{rl}& {\upsilon }_{\pi }(s_2)=1+\gamma {\upsilon }_{\pi }(s_4),\\ & {\upsilon }_{\pi }(s_3)=1+\gamma {\upsilon }_{\pi }(s_4),\\ & {\upsilon }_{\pi }(s_4)=1+\gamma {\upsilon }_{\pi }(s_4).\end{array}$$

The state values can be solved from the above equations. Since the equations are simple, we can solve the state values manually and obtain

$$\begin{array}{rl}& v_{\pi }(s_4)=\frac{1}{1-\gamma },\\ & {\upsilon }_{\pi }(s_3)=\frac{1}{1-\gamma },\\ & {\upsilon }_{\pi }(s_2)=\frac{1}{1-\gamma },\\ & v_{\pi }(s_1)=0.5[0+\gamma v_{\pi }(s_3)]+0.5[-1+\gamma v_{\pi }(s_2)]=-0.5+\frac{\gamma }{1-\gamma }\end{array}$$

Furthermore, if we set $\gamma =0.9$, then

$$\begin{array}{rl}& {\upsilon }_{\pi }(s_4)=10,\\ & {\upsilon }_{\pi }(s_3)=10,\\ & v_{\pi }(s_2)=10,\\ & v_{\pi }(s_1)=-0.5+9=\mathrm{8.5.}\end{array}$$

2.6 Matrix-vector form of the Bellman equation【贝尔曼方程的矩阵-向量形式】

The Bellman equation in (2.7) is in an element-wise form. Since it is valid for every state, we can combine all these equations and write them concisely in a matrix-vector form, which will be frequently used to analyze the Bellman equation.【贝尔曼方程(2.7)是以元素形式表示的。由于它对于每个状态都有效，我们可以将所有这些方程组合起来，并以矩阵-向量形式简洁地写出来，将经常用于分析贝尔曼方程。】

To derive the matrix-vector form, we first rewrite the Bellman equation【为了得到矩阵-向量形式，我们首先将贝尔曼方程重写】

$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =\underset{\text{mean of immediate rewards}}{\underbrace{\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r}}+\underset{\text{mean of future rewards}}{\underbrace{\gamma \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }),}}\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\right],\text{for all }s\in \mathcal{S}\end{array}$$

重写为：

$$\begin{array}{r}{v}_{\pi }(s)=r_{\pi }(s)+\gamma \sum _{s^{\prime }\in \mathcal{S}}{p}_{\pi }(s^{\prime }|s)v_{\pi }(s^{\prime })(2.8)\end{array}$$

其中：
$$\begin{array}{rl}& r_{\pi }(s)\doteq \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r\\ & p_{\pi }(s^{\prime }|s)\dot{=}\sum _{a\in \mathcal{A}}\pi (a|s)p(s^{\prime }|s,a)\end{array}$$

• $r_{\pi }(s)$ denotes the mean of the immediate rewards；
• $p_{\pi }(s^{\prime }|s)$ is the probability of transitioning from $s$ to $s^{\prime }$ under policy $\pi .$

Suppose that the states are indexed as $s_i$ with $i=1,\dots ,n,$ where $n=|S|.$ For state $s_i$, ( 2.8) can be written as
$$\begin{array}{rl}{v}_{\pi }(s_i)=r_{\pi }(s_i)+\gamma \sum _{s_j\in \mathcal{S}}{p}_{\pi }(s_j|s_i)v_{\pi }(s_j)& (2.9)\end{array}$$

令：$v_{\pi }=[v_{\pi }(s_1),\dots ,v_{\pi }(s_n){]}^T\in {\mathbb{R}}^n$，$r_{\pi }=[r_{\pi }(s_1),\dots ,r_{\pi }(s_n){]}^T\in {\mathbb{R}}^n$，and $P_{\pi }\in {\mathbb{R}}^{n\times n}$ with $[P_{\pi }{]}_{ij}=p_{\pi }(s_j|s_i)$。Then, (2.9) can be written in the following matrix-vector form:

$$\begin{array}{rl}{v}_{\pi }& =r_{\pi }+\gamma P_{\pi }{v}_{\pi },\end{array}(2.10)$$

where $v_{\pi }$ is the unknown to be solved, and $r_{\pi }$, $P_{\pi }$ are known.

The matrix $P_{\pi }$ has some interesting properties.

• First, it is a nonnegative matrix meaning that all its elements are equal to or greater than zero. This property is denoted as $P_{\pi }\ge 0$, where 0 denotes a zero matrix with appropriate dimensions. In this book, $\ge$ or $\le$ represents an elementwise comparison operation.
• Second, $P_{\pi }$ is a stochastic matrix meaning that the sum of the values in every row is equal to one. This property is denoted as $P_{\pi }1=1$, where $1=[1,\dots ,1{]}^T$ has appropriate dimensions.

Consider the example shown in Figure 2.6. The matrix-vector form of the Bellman equation is

$$\underset{v_{\pi }}{\underbrace{\left[\begin{array}{c}{v}_{\pi }(s_1)\\ v_{\pi }(s_2)\\ v_{\pi }(s_3)\\ v_{\pi }(s_4)\end{array}\right]}}=\underset{r_{\pi }}{\underbrace{\left[\begin{array}{c}{r}_{\pi }(s_1)\\ r_{\pi }(s_2)\\ r_{\pi }(s_3)\\ r_{\pi }(s_4)\end{array}\right]}}+\gamma \underset{P_{\pi }}{\underbrace{\left[\begin{array}{cccc}{p}_{\pi }(s_1|s_1)& p_{\pi }(s_2|s_1)& p_{\pi }(s_3|s_1)& p_{\pi }(s_4|s_1)\\ p_{\pi }(s_1|s_2)& p_{\pi }(s_2|s_2)& p_{\pi }(s_3|s_2)& p_{\pi }(s_4|s_2)\\ p_{\pi }(s_1|s_3)& p_{\pi }(s_2|s_3)& p_{\pi }(s_3|s_4)& p_{\pi }(s_4|s_4)\\ p_{\pi }(s_1|s_4)& p_{\pi }(s_2|s_4)& p_{\pi }(s_3|s_4)& p_{\pi }(s_4|s_4)\end{array}\right]}}\underset{v_{\pi }}{\underbrace{\left[\begin{array}{c}{v}_{\pi }(s_1)\\ v_{\pi }(s_2)\\ v_{\pi }(s_3)\\ v_{\pi }(s_4)\end{array}\right]}}$$

Substituting the specific values into the above equation gives

$$[\begin{array}{c}{v}_{\pi }(s_1)\\ v_{\pi }(s_2)\\ v_{\pi }(s_3)\\ v_{\pi }(s_4)\end{array}]=\left[\begin{array}{c}0.5(0)+0.5(-1)\\ 1\\ 1\\ 1\end{array}\right]+\gamma \left[\begin{array}{cccc}0& 0.5& 0.5& 0\\ 0& 0& 0& 1\\ 0& 0& 0& 1\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{v}_{\pi }(s_1)\\ v_{\pi }(s_2)\\ v_{\pi }(s_3)\\ v_{\pi }(s_4)\end{array}\right]$$

It can be seen that $P_{\pi }$ satisfies $P_{\pi }1=1$.

2.7 从贝尔曼方程求解 State Values【Solving state values from the Bellman equation】

Calculating the state values of a given policy is a fundamental problem in reinforcement learning. This problem is often referred to as policy evaluation. 【计算给定策略的状态值是强化学习中的一个基本问题。这个问题通常被称为策略评估。】

In this section, we present two methods for calculating state values from the Bellman equation.【介绍两种从贝尔曼方程计算state values的方法】

2.7.1 Closed-form solution

$\mathrm{Since}{v}_{\pi }=r_{\pi }+\gamma P_{\pi }{v}_{\pi }$ is a simple linear equation, its closed-form solution can be easily obtained as

$$v_{\pi }=(I-\gamma P_{\pi }{)}^{-1}{r}_{\pi }.$$

Some properties of $(I-\gamma P_{\pi }{)}^{-1}$ are given below.

• $I-\gamma P_{\pi }$ is invertible. The proof is as follows. According to the Gershgorin circle theorem [4], every eigenvalue of $I-\gamma P_{\pi }$ lies within at least one of the Gershgorin circles. The $i$th Gershgorin circle has a center at $[I-\gamma P_{\pi }{]}_{ii}=1-\gamma p_{\pi }(s_i|s_i)$ and a radius equal to ${\sum }_{j\ne i}[I-\gamma P_{\pi }{]}_{ij}=-{\sum }_{j\ne i}\gamma p_{\pi }(s_j|s_i).$ Since $\gamma <1$, we know that the radius is less than the magnitude of the center: ${\sum }_{j\ne i}\gamma p_{\pi }(s_j|s_i)<1-\gamma p_{\pi }(s_i|s_i)$ Therefore, all Gershgorin circles do not encircle the origin, and hence, no eigenvalue of $I-\gamma P_{\pi }$ is zero.

• $(I-\gamma P_{\pi }{)}^{-1}\ge I$, meaning that every element of $(I-\gamma P_{\pi }{)}^{-1}$ is nonnegative and, more specifically, no less than that of the identity matrix. This is because $P_{\pi }$ has nonnegative entries, and hence, $(I-\gamma P_{\pi }{)}^{-1}=I+\gamma P_{\pi }+{\gamma }^2{P}_{\pi }^2+\cdots \ge I\ge 0.$

• For any vector $r\ge 0$, it holds that $(I-\gamma P_{\pi }{)}^{-1}r\ge r\ge 0$ This property follows from the second property because $[(I-\gamma P_{\pi }{)}^{-1}-I]r\ge 0.$ As a consequence, if $r_1\ge r_2$, we $\begin{array}{r}\text{have }(I-\gamma P_{\pi }{)}^{-1}{r}_1\ge (I-\gamma P_{\pi }{)}^{-1}{r}_2.\end{array}$

2.7.2 Iterative solution

Although the closed-form solution is useful for theoretical analysis purposes, it is not applicable in practice because it involves a matrix inversion operation, which still needs to be calculated by other numerical algorithms. In fact, we can directly solve the Bellman equation using the following iterative algorithm:【尽管闭式解对于理论分析目的很有用，但在实践中不适用，因为它涉及矩阵求逆运算，仍需要
通过其他数值算法进行计算。实际上，我们可以使用以下迭代算法直接解决贝尔曼方程：】

$$\begin{array}{r}{v}_{k+1}=r_{\pi }+\gamma P_{\pi }{v}_k,k=0,1,2,\dots \end{array}(2.11)$$

This algorithm generates a sequence of values { v 0 , v 1 , v 2 , … } \{ v_0, v_1, v_2, \ldots \} { v0​,v1​,v2​,…} , where $v_0\in {\mathbb{R}}^n$ is an initial guess of $v_{\pi }$.

It holds that【大家认为】

$$\begin{array}{rl}{v}_k\to v_{\pi }& =(I-\gamma P_{\pi }{)}^{-1}{r}_{\pi },\text{ as }k\to \infty (2.12)\end{array}$$

2.7.3 Illustrative examples

We next apply the algorithm in (2.11) to solve the state values of some examples.

The examples are shown in Figure 2.7. The orange cells represent forbidden areas. The blue cell represents the target area. The reward settings are $r_{\text{boundary}}=r_{\text{forbidden}}=-1$，and $r_{\text{target}}=1$. Here, the discount rate is $\gamma =0.9$.

• Figure 2.7(a) shows two “good” policies and their corresponding state values obtained by (2.11). The two policies have the same state values but differ at the top two states in the fourth column. Therefore, we know that different policies may have the same state values.
• Figure 2.7(b) shows two “bad” policies and their corresponding state values. These two policies are bad because the actions of many states are intuitively unreasonable. Such intuition is supported by the obtained state values. As can be seen, the state values of these two policies are negative and much smaller than those of the good policies in Figure 2.7(a).

2.8 From state value to action value

While we have been discussing state values thus far in this chapter, we now turn to the action value, which indicates the “value” of taking an action at a state. While the concept of action value is important, the reason why it is introduced in the last section of this chapter is that it heavily relies on the concept of state values. It is important to understand state values well first before studying action values.

State Value：
$$\begin{array}{r}{v}_{\pi }(s)\doteq \mathbb{E}[G_t|S_t=s]\end{array}$$

Action Value：
$$\begin{array}{r}{q}_{\pi }(s,a)\doteq \mathbb{E}[G_t|S_t=s,A_t=a]\end{array}$$

As can be seen, the action value is defined as the expected return that can be obtained after taking an action at astate. 【动作值被定义为在某个状态下采取某个动作后可以获得的预期回报】

It must be noted that $q_{\pi }(s,a)$ depends on a state-action pair $(s,a)$ rather than an action alone. It may be more rigorous to call this value a state-action value, but it is conventionally called an action value for simplicity.【必须注意的是，$q(s,a)$取决于state-action对$(s,a)$,而不仅仅是一个action 。为了简单起见，通常将这个值称为 action value，尽管更严谨的说法应该是state-action value】

What is the relationship between action values and state values? 【行动值和状态值之间的关系是什么？】

• First, it follows from the properties of conditional expectation that
  $$\begin{array}{rl}\underset{v_{\pi }(s)}{\underbrace{\mathbb{E}[G_t|S_t=s]}}& =\sum _{a\in \mathcal{A}}\underset{q_{\pi }(s,a)}{\underbrace{\mathbb{E}[G_t|S_t=s,A_t=a]}}\pi (a|s)\end{array}$$
  It then follows that
  $$\begin{array}{rlr}{v}_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a|s)q_{\pi }(s,a).& & (2.13)\end{array}$$
  As a result, a state value is the expectation of the action values associated with that state. 【State Value是与该state相关联的Action Value的期望值】
• Second, since the state value is given by
  $$\begin{array}{r}{v}_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a|s)[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })],\end{array}$$
  comparing it with (2.13) leads to
  $$\begin{array}{r}{q}_{\pi }(s,a)=\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }).(2.14)\end{array}$$
  It can be seen that the action value consists of two terms.
  • The first term is the mean of the immediate rewards；
  • the second term is the mean of the future rewards.

Both (2.13) and (2.14) describe the relationship between state values and action values. They are the two sides of the same coin:

• (2.13) shows how to obtain state values from action values【action values---->state values】；
• (2.14) shows how to obtain action values from state values【state values---->action values】；

2.8.1 Illustrative examples

We next present an example to illustrate the process of calculating action values and discuss a common mistake that beginners may make.

Consider the stochastic policy shown in Figure 2.8. We next only examine the actions of $s_1$. The other states can be examined similarly. The action value of $(s_1,a_2)$ is

$$q_{\pi }(s_1,a_2)=-1+\gamma v_{\pi }(s_2)$$

where $s_2$ is the next state. Similarly, it can be obtained that

$$\begin{array}{r}{q}_{\pi }(s_1,a_3)=0+\gamma v_{\pi }(s_3)\end{array}$$

A common mistake that beginners may make is about the values of the actions that the given policy does not select. For example, the policy in Figure 2.8 can only select $a_2$ or $a_3$ and cannot select $a_1,a_4,a_5.$ One may argue that since the policy does not select $a_1,a_4,a_5,$ we do not need to calculate their action values, or we can simply set $q_{\pi }(s_1,a_1)=q_{\pi }(s_1,a_4)=q_{\pi }(s_1,a_5)=0.$ This is wrong.

• First, even if an action would not be selected by a policy, it still has an action value In this example, although policy $\pi$ does not take $a_1$at$s_1, $ we can still calculate its action value by observing what we would obtain after taking this action. Specificallyy after taking $a_1$, the agent is bounced back to $s_1$(hence, the immediate reward is -1) and then continues moving in the state space starting from $s_1$ by following $\pi$ (hence, the future reward is $\gamma v_{\pi }(s_1))$. As a result, the action value of $(s_1,a_1)$ is
  $$q_{\pi }(s_1,a_1)=-1+\gamma v_{\pi }(s_1)$$
  Similarly, for $a_4$ and $a_5$, which cannot be possibly selected by the given policy either, we have

$$\begin{array}{rl}{q}_{\pi }(s_1,a_4)& =-1+\gamma v_{\pi }(s_1),\\ q_{\pi }(s_1,a_5)& =0+\gamma v_{\pi }(s_1).\end{array}$$

• Second, why do we care about the actions that the given policy would not select?
  Although some actions cannot be possibly selected by a given policy, this does not mean that these actions are not good. It is possible that the given policy is not good, so it cannot select the best action. The purpose of reinforcement learning is to find optimal policies. To that end, we must keep exploring all actions to determine better actions for each state.【其次，为什么我们关心给定策略不会选择的行动？尽管某些行动不可能被给定策略选择，但这并不意味着这些行动不好。有可能给定的策略不好，因此无法选择最佳行动。强化学习的目的是找到最优策略。为此，我们必须不断探索所有行动，以确定每个状态的更好行动。】

Finally, after computing the action values, we can also calculate the state value ac- cording to (2.14):
$$\begin{array}{rl}{\upsilon }_{\pi }(s_1)& \begin{array}{r}=0.5q_{\pi }(s_1,a_2)+0.5q_{\pi }(s_1,a_3),\end{array}\\ & \begin{array}{r}=0.5[0+\gamma v_{\pi }(s_3)]+0.5[-1+\gamma v_{\pi }(s_2)].\end{array}\end{array}$$

2.8.2 The Bellman equation in terms of action values

The Bellman equation that we previously introduced was defined based on state values. In fact, it can also be expressed in terms of action values.

In particular, substituting (2.13) into (2.14) yields

$$\begin{array}{r}{q}_{\pi }(s,a)=\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)\sum _{a^{\prime }\in \mathcal{A}(s^{\prime })}\pi (a^{\prime }|s^{\prime })q_{\pi }(s^{\prime },a^{\prime }),\end{array}$$

which is an equation of action values. The above equation is valid for every state-action pair. If we put all these equations together, their matrix-vector form is

$$\begin{array}{rl}{q}_{\pi }& =\tilde{r}+\gamma P\Pi q_{\pi },(2.15)\end{array}$$

where $q_{\pi }$ is the action value vector indexed by the state-action pairs: its $(s,a)$th element is $[q_{\pi }{]}_{(s,a)}=q_{\pi }(s,a).\tilde{r}$ is the immediate reward vector indexed by the state-action pairs: $[\tilde{r}{]}_{(s,a)}={\sum }_{r\in \mathcal{R}}p(r|s,a)r.$ The matrix $P$ is the probability transition matrix, whose row is indexed by the state-action pairs and whose column is indexed by the states: $[P{]}_{(s,a),s^{\prime }}=p(s^{\prime }|s,a).$ Moreover, $\Pi$ is a block diagonal matrix in which each block is a $1\times |\mathcal{A}|$ vector: ${\Pi }_{s^{\prime },(s^{\prime },a^{\prime })}=\pi (a^{\prime }|s^{\prime })$ and the other entries of $\Pi$ are zero.

Compared to the Bellman equation defined in terms of state values, the equation defined in terms of action values has some unique features. For example, $\tilde{r}$ and $P$ are independent of the policy and are merely determined by the system model. The policy is embedded in $\Pi$. It can be verified that (2.15) is also a contraction mapping and has a unique solution that can be iteratively solved. More details can be found in [5].

2.9 Summary

The most important concept introduced in this chapter is the state value. Mathematically, a state value is the expected return that the agent can obtain by starting from a state.【本章介绍的最重要的概念是state value。在数学上，状态值是 agent 从一个state开始可以获得的expected return】

The values of different states are related to each other. That is, the value of state s relies on the values of some other states, which may further rely on the value of state s itself. 【不同状态的值彼此相关。也就是说，状态 $s$ 的值依赖于其他一些states的值，而这些states
的值可能进一步依赖于状态$s$ 本身的值】

This phenomenon might be the most confusing part of this chapter for beginners.

It is related to an important concept called bootstrapping, which involves calculating something from itself. Although bootstrapping may be intuitively confusing, it is clear if we examine the matrix-vector form of the Bellman equation. 【如果我们考虑贝尔曼方程的矩阵-向量形式，就会变得清晰】

In particular, the Bellman equation is a set of linear equations that describe the relationships between the values of all states.【特别是，贝尔曼方程是一组线性方程，描述了所有状态值之间的关系】

Since state values can be used to evaluate whether a policy is good or not, the process of solving the state values of a policy from the Bellman equation is called policy evaluation. 【由于状态值可以用来评估一个policy的好坏，从贝尔曼方程中解决策略的state values的过程被称为策略评估（policy evaluation）】

As we will see later in this book, policy evaluation is an important step in many reinforcement learning algorithms.

Another important concept, action value, was introduced to describe the value of taking one action at a state. 【另一个重要的概念，action value，被引入来描述在某个state下采取某个 action 的价值】

As we will see later in this book, action values play a more direct role than state values when we attempt to find optimal policies. 【正如我们将在本书后面看到的那样，当我们试图寻找最优policy时，action values比state value起到更直接的作用】

Finally, the Bellman equation is not restricted to the reinforcement learning field. Instead, it widely exists in many fields such as control theories and operation research. In different fields, the Bellman equation may have different expressions. In this book, the Bellman equation is studied under discrete Markov decision processes. More information about this topic can be found in [2].【最
后，贝尔曼方程不仅限于强化学习领域，而是广泛存在于许多领域，如控制理论和运筹学。在不同的领域中，贝尔曼方程可能有不同的表达方式。在本书中，贝尔曼方程是在离散马尔可夫决策过程下进行研究的】

2.10 Q&A

Q: What is the relationship between state values and returns?
A: The value of a state is the mean of the returns that can be obtained if the agent starts from that state.【一个状态的价值是指如果代理从该状态开始，可以获得的回报的平均值。】

Q: Why do we care about state values?
A: State values can be used to evaluate policies. In fact, optimal policies are defined based on state values. 【状态值可以用来评估策略。事实上，最优策略是基于状态值来定义的。】This point will become clearer in the next chapter.

Q: Why do we care about the Bellman equation?
A: The Bellman equation describes the relationships among the values of all states.【贝尔曼方程描述了所有状态的值之间的关系。】
It is the tool for analyzing state values.

Q: Why is the process of solving the Bellman equation called policy evaluation?
A: Solving the Bellman equation yields state values. Since state values can be used to evaluate a policy, solving the Bellman equation can be interpreted as evaluating the corresponding policy. 【解决贝尔曼方程可以得到状态值。由于状态值可以用来评估策略，所以解决贝尔曼方程可以被解释为评估相应的策略。】

Q: Why do we need to study the matrix-vector form of the Bellman equation?
A: The Bellman equation refers to a set of linear equations established for all the states. To solve state values, we must put all the linear equations together. The matrix-vector form is a concise expression of these linear equations. 【贝尔曼方程是为所有状态建立的一组线性方程。要解决状态值，我们必须将所有线性方程放在一起。矩阵-向量形式是这些线性方程的简洁表达。】

Q: What is the relationship between state values and action values?
A: On the one hand, a state value is the mean of the action values for that state. On the other hand, an action value relies on the values of the next states that the agent may transition to after taking the action. 【一方面，状态值是该状态的动作值的平均值。另一方面，动作值依赖于agent在采取行动后可能转移到的下一个状态的值】

Q: Why do we care about the values of the actions that a given policy cannot select?
A: Although a given policy cannot select some actions, this does not mean that these actions are not good. On the contrary, it is possible that the given policy is not good and misses the best action. To find better policies, we must keep exploring different actions even though some of them may not be selected by the given policy.【尽管某个policy不能选择某些行动，但这并不意味着这些actions不好。相反，可能是这个policy本身不好，错过了最佳actions。为了找到更好的policy，我们必须继续探索不同的actions，即使其中一些actions可能不被当前的policy所选择】

---

---

---

参考资料：
怎么理解强化学习中状态价值和动作价值？