RL-Zhao-(9)-Policy-Based02: Selection of objective function/Metrics [①average state value; ②average one-step reward], gradient calculation of objective function

1. Selection of objective functions (Metrics to define optimal policies) [Category 2]

There are two types of objective functions/metrics:

• The average state value
• Average one-step reward

1、The average state value

2-dimensional equations:
$$\begin{array}{rl}{\overline{in}}_{\pi }& =\sum _{s\in \mathcal{S}}d(s)v_{\pi }(s)\\ & \doteq AND\left[\sum _{t=0}^{\infty }{c}^t{R}_{t+1}\right]\end{array}$$

The first category isthe average state value, or simply average value. This metric is defined as follows:
$${\overline{in}}_{\pi }=\sum _{s\in S}d\left(s\right)v_{\pi }\left(s\right)$$

• Naka ${\overline{in}}_{\pi }$is the weighted average of state values;
• $d(s)\ge 0$This is the situation$s$'s weight
• 因为 ${\sum }_{s\in S}d\left(s\right)=1$, I can do it $d\left(s\right)$ is interpreted as a probability distribution (probability distribution). Then, this melnic can be written as
  $${\overline{in}}_{\pi }=\mathbb{E}[v_{\pi }(S)]$$
  inside$S\sim d$。

昛RAN ${\overline{in}}_{\pi }$ Correct strategy $\pi$ function, dissimilar strategy $\pi$ corresponds to different values, so we can optimize and find an optimal strategy$\pi$ Let this value reach the maximum. This is actually a very natural way to choose metric.

Vector-product(infinitesimal) function:
$${\overline{in}}_{\pi }=\sum _{s\in S}d\left(s\right)v_{\pi }\left(s\right)=d^T{v}_{\pi }$$
in

• ${in}_{\pi }=[...,{in}_{\pi }(s),...{]}^T\in {\mathbb{R}}^{|S|}$ , each element in this area ${in}_{\pi }\left(s\right)$ 对应的情况$s$ 的state value；
• $d=[...,d(s),...{]}^T\in R^{|S|}$, each element in this area $d\left(s\right)$ /span>$The\; weight\; or\; probability\; of\; s$;

This form is very helpful for analyzing its gradient later.

How to choose distribution $d$ ? There are two situations here:

• The first case is $d$ Sum strategy $\pi$ The dead series.
  • This case is relatively simple because the gradient of the metric is easy to calculate. If this $d$ sum $\pi$ Naga-ku of the non-existential system ${in}_{\pi }$When the gradient of , this $d$ does not involve any gradient, so I will only find one ${in}_{\pi }$gradient of (if $d$ sum $\pi$ 有关于那我求 ${in}_{\pi }$ When using gradient, of course this is also required $d$ 关于这个 $\pi$ gradient, so it is relatively more troublesome).
  • In this case, in order to show that $d$ sum $\pi$ 無有关体，General$d$Photograph$d_0$，${\overline{in}}_{\pi }$ Photo ${\overline{in}}_{Pi}^0$
  • How do you choose$d_0?$
    • A simple way is to treat all states equally, that is, choose$d_0(s)=\frac{1}{|S|}$, so this is actually a uniform distribution.
    • Another important situation is that we only have a specific state $s_0$Interested. For example, episodes always start from the same state $s_0$ Start [For example, at the beginning of some games, it always starts from that screen, which actually corresponds to a specific state$s_0$, then I hope that the reward I can get is as big as possible. In this case, I cannot treat all states equally, so I may have preferences for some states, then In this extreme case, I only care about $s_0$，从$s_0$Starting from , the bigger the reward I get, the better], then, we only focus on starting from $s_0$Long-term return starting with . In this case
      $$\begin{array}{rl}& d_0(s_0)=1\\ & d_0(s\ne s_0)=0\end{array}$$
      The current situation is below ${\overline{in}}_{\pi }$ You have completed the change ${\overline{in}}_{Pi}^0$, let me maximize this${\overline{in}}_{\pi }$, actually maximizing from $s_0$How much return can I get from setting off?
• The second case is $d$ dependence on strategy $\pi$。
  • This is a common choice, choose $d$ One $d_{\pi }(s)$, 即 stationany distribution under $\pi$[Simply put, I have a strategy, and then I follow that strategy to constantly interact with the environment. When I execute that strategy many, many times, I can The prediction of being in a certain state is the probability that the agent is in that state, that is, it will gradually reach a stable state at that time, and this probability can actually be directly passed through this$d_{Pi}^T{P}_{\pi }=d_{Pi}^T$The formula is calculated].
    • $d_{p}A\; basic\; property\; of\; i$ is that it satisfies
      $$d_{Pi}^T{P}_{\pi }=d_{Pi}^T$$
      Inside$P_{\pi }$Is the state transition probability matrix (state transition matrix).
  • 选择$d_{\pi }$explanation of:
    • If a state is frequently visited in a long run, it is more important and should be given more weight;
    • If a state is rarely visited, its weight will naturally be less;

2、average one-step reward

2definitely defined:
$$\begin{array}{rl}{\bar{r}}_{\pi }& \doteq \sum _{s\in \mathcal{S}}{d}_{\pi }\left(s\right)r_{\pi }\left(s\right)\\ & \doteq \underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}\right]\end{array}$$

The second category is average one-step reward, or average reward for short. Specifically ${}_{\prime }$ the metrics是
$${\bar{r}}_{\pi }\doteq \sum _{s\in S}{d}_{\pi }\left(s\right)r_{\pi }\left(s\right)=\mathbb{E}[r_{\pi }\left(S\right)]$$
Among them:

• $d_{\pi }(s)$ 是 $The\; weight\; corresponding\; to\; s$ is actually a stationary distribution, which depends on this strategy $\pi$的；
• $S\sim d_{\pi }$。
• $\begin{array}{r}{r}_{\pi }\doteq \sum _{a\in \mathcal{A}}\pi (a|s)r(s,a)\end{array}$ is here $s$ Starting from an average value of the immediate reward of the single step I got, and
  $$r(s,a)=\mathbb{E}[R|s,a]=\sum _{r}rp(r|s,a)$$

As its name suggests, ${\bar{r}}_{\pi }$ is a weighted average of one-step immediate reward (${\bar{r}}_{\pi }$The horizontal line above also represents an average meaning).

The second form of average reward above:

• Suppose an agent generates a trajectory along a given policy, and its rewards are$(R_{t+1},R_{t+2},...)$；
• 沿着trajectony的average single-step reward是
  $$\begin{array}{rl}& \begin{array}{r}\underset{n\to \infty }{\lim }\frac{1}{n}\mathbb{E}[R_{t+1}+R_{t+2}+\cdots +R_{t+n}|S_t=s_0]\end{array}=\underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}|S_t=s_0\right]\end{array}$$
  inside $s_0$ 是the starting state of the trajectory。

Let us divide the equation into one:
$$\begin{array}{rl}\underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}|S_t=s_0\right]& =\underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}\right]\\ & =\sum _s{d}_{\pi }(s)r_{\pi }(s)\\ & ={\bar{r}}_{\pi }\end{array}$$
Landlord $s_0$ No more Why? Because $s_0$ does not work. It means that after you have run an infinite number of steps, it no longer matters where you started.
Note:

• When n approaches infinity, starting state $s_0$It doesn't matter anymore.
• These two are about ${\bar{r}}_{\pi }$The equations are equal.

This formula is one that you may often see in papers.

3、Remarks

I would like to emphasize a few points on the above two metrics:

• Remark1
  • These metrics are strategies $\pi$ function;
  • Cause strategy$\pi$Koreyu$\theta$ is parameterized, so these metrics are$\theta$ function;
  • In other words, different $\theta$ can generate different metric values;
  • Therefore, we can search for the optimal $\theta$ and then maximize these metrics;
• Remark2
  • These metrics are complex and divided into two cases. The first is the discounted case, where$c\in [0,1)$; This is an undiscounted case in$c=1$。
  • Here we only consider the discounted case.
• Remark3
  • Directly,${\bar{r}}_{\pi }$ is short-sighted because it rarely considers the immediate rewards, while ${\overline{in}}_{\pi }$考虑 the total reward overall steps。【$\times$】
  • However, in fact these two metrics are equivalent, specifically, in the discounted case, when$c<1$,有
    $${\bar{r}}_{\pi }=(1-\gamma ){\overline{in}}_{\pi }$$

4. Exercise (another form of objective function)

Answer: First, analyze and understand such a metric.

• 它从$S_0\sim d$ Start, end $A_0,R_1,S_1,A_1,R_2,S_2,......$
• $A_t\sim \pi (S_t)$，并且 $R_{t+1},S_{t+1}\sim p(R_{t+1}|S_t,A_t)$，$p(S_{t+1}|S_t,A_t)$

Then, we know that this metric is the same as the average value, because
$$\begin{array}{rl}J(\theta )=AND\left[\sum _{t=0}^{\infty }{c}^t{R}_{t+1}\right]& =\sum _{s\in \mathcal{S}}d(s)\mathbb{E}\left[\sum _{t=0}^{\infty }{c}^t{R}_{t+1}|S_0=s\right]\\ & =\sum _{s\in S}d(s)v_{\pi }(s)\\ & ={\overline{in}}_{\pi }\end{array}$$

2. Gradient of the objective function (Gradients of the metrics)

Given a metric, then

• derive its gradient
• Then, apply gradient-based methods to optimize this metric.

The calculation of gradient is the most complex calculation part in policy gradient methods!

• First, we need to distinguish between different objective functions/metrics:
  • $\begin{array}{rl}{\overline{in}}_{\pi }& =\sum _{s\in S}d\left(s\right)v_{\pi }\left(s\right)\doteq AND\left[\sum _{t=0}^{\infty }{c}^t{R}_{t+1}\right]\end{array}$
  • ${\overline{in}}_{Pi}^0$；
  • $\begin{array}{r}{\bar{r}}_{\pi }\doteq \sum _{s\in S}{d}_{\pi }\left(s\right)r_{\pi }\left(s\right)\doteq \underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}\right]\end{array}$
• Second, we need to distinguish between the discounted and undiscounted cases

Gradients/Speed ​​Ratio Equations:
$${\nabla }_{\theta }J\left(\theta \right)=\sum _{s\in S}\eta (s)\sum _{a\in \mathcal{A}}{\nabla }_{\theta }\pi (a|s,\theta )q_{\pi }(s,a)$$
inside

• Topic function $J(\theta )$可以是：
  • $\begin{array}{rl}{\overline{in}}_{\pi }& =\sum _{s\in S}d\left(s\right)v_{\pi }\left(s\right)\doteq AND\left[\sum _{t=0}^{\infty }{c}^t{R}_{t+1}\right]\end{array}$
  • ${\overline{in}}_{Pi}^0$；
  • $\begin{array}{r}{\bar{r}}_{\pi }\doteq \sum _{s\in S}{d}_{\pi }\left(s\right)r_{\pi }\left(s\right)\doteq \underset{n\to \infty }{\lim }\frac{1}{n}AND\left[\sum _{k=1}^n{R}_{t+k}\right]\end{array}$
• $=$ can mean strict equality (=), approximate (≈) or proportional equality (∝);
• $\begin{array}{r}\sum _{s\in S}\end{array}$: indicates pair $s$ ぐ和；
• $\eta$ is a distribution or weight under states. Each state has a weight$\eta \left(s\right)$；$\eta$ For different objective functions, it will show different distribution under different circumstances;
• $\begin{array}{r}\sum _{a\in \mathcal{A}}{\nabla }_{\theta }\pi (a|s,\theta )q_{\pi }(s,a)\end{array}$: represents each state $s$ Independent action function ${\nabla }_{\theta }\pi (a|s,\theta \left)q_{\pi }\right(s,a)$'s addition;
  • ${\nabla }_{\theta }\pi (a|s,\theta )$: Display strategy $\pi$ level;
  • $q_{\pi }(s,a)$：表示$(s,a)$ 的action value；

In short, the results obtained in all these casesgradient are very similar, so this formula is used to express them. For most students, this formula is enough. Unless you need to research and innovate new algorithms, then you can read the detailed content in the book.

Separate information ${\bar{r}}_{\pi }$、${\overline{in}}_{\pi }$、${\overline{in}}_{Pi}^0$, if you want to have a slightly larger range(Details are not given here. Interested readers can read my book)::
$$\begin{array}{rl}& {\nabla }_{\theta }{\bar{r}}_{\pi }\simeq \sum _s{d}_{\pi }(s)\sum _a{\nabla }_{\theta }\pi (a|s,\theta \left)q_{\pi }\right(s,a),\\ & {\nabla }_{\theta }{\overline{in}}_{\pi }=\frac{1}{1-c}{\nabla }_{\theta }{\bar{r}}_{\pi }\\ & {\nabla }_{\theta }{\overline{in}}_{Pi}^0=\sum _{s\in \mathcal{S}}{r}_{\pi }\left(s\right)\sum _{a\in A}{\nabla }_{\theta }\pi (a|s,\theta \left)q_{\pi }\right(s,a)\end{array}$$

1. Analysis of gradient formula

Slope/Gradient
$${\nabla }_{\theta }J(\theta )=\sum _{s\in S}\eta (s)\sum _{a\in A}{\nabla }_{\theta }\pi (a|s,\theta \left)q_{\pi }\right(s,a)$$
can be written in a compact and useful form:

$$\begin{array}{rl}{\nabla }_{\theta }J\left(\theta \right)& =\sum _{s\in S}\eta \left(s\right)\sum _{a\in A}{\nabla }_{\theta }\pi (a|s,\theta \left)q_{\pi }\right(s,a)\\ & =\mathbb{E}[{\nabla }_{\theta }\ln \pi (A|S,\theta )q_{\pi }(S,A)]\end{array}$$

Inside:$S\sim \eta$，$A\sim \pi (A|S,\theta )$。

• possession $\sum$ Remove all and write it in the form of an expectation: $\mathbb{E}[]$，
• 这り边的$S$、$A$ The city is the amount of change, $S\sim \eta$，$A\sim \pi (A|S,\theta )$。
  • $S$ is full $\eta$ distribution;
  • $A$is full $\pi (A|S,Such\; a\; distribution\; of\; \theta )$;

Only one ${\nabla }_{\theta }J(\theta )=\mathbb{E}[{\nabla }_{\theta }\ln \pi (A|S,\theta )q_{\pi }(S,A)]$ Such an expression?

Because we can use sampling to approximate this gradient!

三、Gradient-ascent algorithm

1、REINFORCE algorithm

---

---

---

Reference materials:
Introduction to Reinforcement Learning (13) - Policy Gradient Method