[Reinforcement Learning Theory] Derivation of State Value Function and Action Value Function Series Formulas

Because the definition difference and formula relationship between the state value function and the action value function are often confused, this time it will be sorted out and recorded.

To understand formula derivation, you need to understand several concepts in the basic definition.

basic definition

reward function

There are two notations for the reward function .

① Denote as $r\left(s\right)$ , represents a certain state$The\; reward\; of\; s$ refers tothe expectation, namely:
$$r(s)=E[R_t|S_t=s]$$

Why use $R_t$The expectation instead of directly using $R_t$To represent the reward for this state?

Because for the same state, it takes different actions, the reward $R_t$may be different.

②Described as $r(s,a)$ means a statesstake actionaa$under\; s$$The\; reward\; of\; a$ refers tothe expectationof the reward that can be obtained whentransferring to this state and taking this action, namely: r ( s , a ) = E [ R t ∣ S t = s , A t = a ] r(s, a) = \mathbb{E}[R_t | S_t = s, A_t = a]
$$r(s,a)=E[R_t|S_t=s,A_t=a]$$

Why use $R_t$The expectation instead of directly using $R_t$To represent the reward for taking this action in this state?

Because for the same state, even if the action taken is the same, the next state $s^{\prime }$ may also be different, and the reward obtained afterwards$R_{t+1}$It may also be different, and the final return $G_t$Nature is also different.

return

Return (Return) , denoted as $G_t$, referring to from ttState st s_tat time$t$$s_t$From the beginning to the end state, the sum of the decay of all rewards is:
$$\begin{array}{rl}{G}_t=& R_t+\gamma R_{t+1}+c^2{R}_{t+2}+...\\ =& \sum _{k=0}^{\infty }{c}^k{R}_{t+k}\end{array}$$

value

Value (value) is a state-based concept. The value of a certain state refers to: from a certain state $s$ , until the terminal state, the cumulative reward (that is, the reward$G_t$) expectations .

1. Why use ** $G_t$The expectation** and cannot directly use $G_t$？

Because for the same starting state $s$ , its$G_t$Can be different . In order to objectively evaluate the value of a state, it is necessary to consider as much as possible the different rewards it can bring.

2. Why $G_t$Can it be different?

Because in the process of interaction, the initial state $s_t$may go to a different state $s^{\prime }$ , the obtained$R_t$Also different, the final return $G_t$Naturally it is different.

value function

Value function (value function) , denoted as $V\left(s\right)$ can be understood as a mapping relationship, and the input is the state$s$ ​, the output is the value of the state, namely:
$$V(s)=E[G_t|S_t=s]$$

What is the difference between a reward function and a value function?

According to my own understanding, the reward function only focuses on the benefits that the state can currently obtain, while the value function focuses on all benefits from the current state to the end of the future.

state transition matrix

The state transition matrix can be written as $P(s^{\prime }|s)$, indicating the state$s$ transitions to state$s^{\text{'}}$ probability.

In the case of a finite set of states, it can be represented by a matrix; if the set of states is not limited, it is called a state transition function.

Strategy

Policy (policy) , denoted as $\pi$ . The strategy can be understood as: in the known input state$In\; the\; case\; of\; s$ , take action$a$ approximation, i.e.:
$$\pi (s,a)=\pi (a|s)=P(A_t=a|S_t=s)$$
strategy$\pi$ (the value) is only related to the current state$s$ dependent, independent of the state preceding the current state.

for the same state $s$ , using the strategy$\pi$ is different, the action taken$a$ is different, the value value is also different.

state transition function

The state transition function can be written as $P(s^{\prime }|s,a)$ , means in state$s$ performs action$After\; a$ reaches state$s^{\text{'}}$ probability.

In contrast to state transition matrices, state transition functions can represent situations where the set of states is not finite.

There are two forms of state transition, one is $P(s^{\prime }|s)$, the other is$P(s^{\prime }|s,a)$ , the bridge connecting the two is the strategy$\pi$，即：
$$P(s^{\prime }|s)=\sum _{a\in A}\pi (a|s)P(s^{\prime }|s,a)$$

state value function

The state-value function can be denoted as $V^{\pi }\left(s\right)$, means: in the Markov decision process, the agent starts from state$s$ , follow the strategyπ \piThe return G t obtained by$\pi$$G_t$The expectation of , namely:
$$V^{\pi }\left(s\right)={\mathbb{E}}_{}[G_t|S_t=t]$$

It looks similar to the value function , except that the policy is not emphasized in the value function .

The following two questions and their answers are similar to the two questions and answers asked when understanding value .

1. Why is the return $G_t$instead of just using the return $G_t$OK?

Because for the same state $s$ and a given strategy$\pi$ , whose$G_t$may be different . To objectively evaluate the value of a state under a given policy, it is necessary to consider as much as possible the different rewards it can bring.

2. Why $G_t$Can it be different?

Because for the same state $s$ and a given strategy$\pi$ , the action aataken by the agent in the current state$a$ may be different (especially when the strategy used is a randomness strategy), the resulting$R_t$different, the final $G_t$It may also be different.

action value function

The action-value function can be written as $Q^{\pi }(s,a)$ , means: in the Markov decision process, the agent starts from state$s$ , according to the strategy$\pi$ performs action$a$ , the final reward$G_t$的期望，即：
$$Q^{\pi }(s,a)=E[G_t|S_t=s,A_t=a]$$

It looks very similar to the state-value function , except that the action is not emphasized in the state-value function .

The following two questions and their answers are similar to the two questions and answers asked when understanding the state-value function .

1. Why is the return $G_t$instead of just using the return $G_t$OK?

Because for the same state $s$ , a given strategy$\pi$ and a given action$a$ , whose$G_t$may be different . To objectively evaluate the value of a state under a given policy, it is necessary to consider as much as possible the different rewards it can bring.

2. Why $G_t$Can it be different?

Because for the same state $s$ , a given strategy$\pi$ and a given action$a$ , its next state$s^{\prime }$ may be different (because the environment may have changed differently), the action$a$ may be different (especially when the strategy used is a randomness strategy), the resulting$R_t$different, the final $G_t$It may also be different.

The relationship between the state-value function and the action-value function

relationship 1

$$V^{\pi }\left(s\right)=\sum _{a\in A}\pi (a|s\left)Q^{\pi }\right(s,a)$$

The derivation basis of relation 1: ①Definition of state-value function; ②Definition of action-value function. The derivation process is as follows:
$$\begin{array}{rl}{V}^{\pi }\left(s\right)& ={\mathbb{E}}_{}[G_t|S_t=s]\\ & =\sum _{a\in A}\pi (a|s)E_{}[G_t|S_t=s,A_t=a]\\ & =\sum _{a\in A}\pi (a|s\left)Q^{\pi }\right(s,a)\end{array}$$
Line 1 uses the definition of the state value function ;

The understanding that transitions from line 2 to line 3 uses the definition of the action-value function .

Here I explain why ${\mathbb{E}}_{}[G_t|S_t=s]$转换成${\sum }_{a\in A}\pi (a|s)E_{}[G_t|S_t=s,A_t=a]$，而不是${\sum }_{a\in A}\pi (a|s)G_t$(The difference between these two ideas is only whether there is a pair of $G_t$Seek expectations) to give their own explanation: although the latter seems to be more in line with the formula form of mathematical expectations: state $s$ according to the probability$\pi (a|s)$ selects action$After\; a$ , the object multiplied later should be the corresponding state$s$ takes actionaaThe reward obtained by$a$$G_t$. However, when explaining the definition of the action value function , it was mentioned above : For the same state $s$ , a given strategy$\pi$ and a given action$a$ , whose$G_t$may be different . That is, $(s,a)$ and$G_t$There is not a one-to-one correspondence, so they cannot be directly multiplied. At this time, it is necessary to find a method that can be compared with $(s,a)$ one-to-one correspondence, and generalization$(s,a)$ The concept of earnings. At this time,action valuebecomes a good choice. It is for$(s,a)$ Can correspond to multiple$G_t$Calculate the mathematical expectation and approximate $(s,a)$ The overall income (from this point of view, in fact, the second line can also be omitted, and the third line can be directly introduced).

relationship 2

$$Q^{\pi }(s,a)=r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a)V^{\pi }(s^{\prime })$$

The derivation basis of relation 2: ①Definition of action-value function; ②Definition of reward; ③Definition of reward function; ④Definition of state-value function. The derivation process is as follows:
$$\begin{array}{rl}{Q}^{\pi }(s,a)& ={\mathbb{E}}_{}[G_t|S_t=s,A_t=a]\\ & =E_{}[R_t+\gamma R_{t+1}+c^2{R}_{t+2}+...|S_t=s,A_t=a]\\ & =E_{}[R_t+c(R_{t+1}+\gamma R_{t+2}+...)|S_t=s,A_t=a]\\ & =E_{}[R_t|S_t=s,A_t=a]+c{E}_{}[G_{t+1}|S_t=s,A_t=a]\\ & =r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)E_{}\right[G_{t+1}|S_{t+1}=s^{\prime }]\\ & =r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a)V^{\pi }(s^{\prime })\end{array}$$
During the derivation, the equation in line 1 uses the definition of the action-value function ;

The conversion from line 1 to line 2 and the conversion from line 3 to line 4 use the definition of Return ;

The transformation from line 4 to line 5 uses the definition of the reward function ;

The transformation from line 5 to line 6 uses the definition of the state-value function .

Here I explain why the second sub-item in line 5 is ${\mathbb{E}}_{}[G_{t+1}|S_t=s,A_t=a]$拆解成${\sum }_{s^{\prime }\in S}P(s^{\prime }|s,a)E_{}[G_{t+1}|S_{t+1}=s^{\prime }]$, instead of${\sum }_{s^{\prime }\in S}P(s^{\prime }|s,a)G_{t+1}$(The difference between these two ideas is only whether there is a pair of $G_{t+1}$Seek expectations) to give their own explanation: although the latter seems to be more in line with the formula form of mathematical expectations: state $s$ , action$a$ According to the probability$P(s^{\prime }|s,a)$ Transition to state$s^{\prime }$ , the object multiplied later should be the corresponding state$s^{\prime }$ return$G_{t+1}$. But the above mentioned when explaining the definition of value : for the same initial state $s$ , its$G_t$Can be different . That is, $s^{\prime }$ and$G_{t+1}$There is not a one-to-one correspondence, so they cannot be directly multiplied. At this time, it is necessary to find a method that can be compatible with $s^{\prime }$ One-to-one correspondence, and can generalize state$s^{\text{'}}$ The concept of earnings. At this time,valuebecomes a good choice. It is for$s^{\prime }$ can correspond to multiple$G_{t+1}$Calculate the mathematical expectation and approximate the state $s^{’}$ (In this way, line 5 can also be omitted, and line 6 can be derived directly).

另外，特别提醒：
$$\begin{array}{rl}{Q}^{\pi }(s,a)& ={\mathbb{E}}_{}[G_t|S_t=s,A_t=a]\\ & =\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)E\right[G_{t+1}|S_{t+1}=s^{\prime }]\end{array}$$
Can't turn like this! Because this does not take into account $R_t$, from $G_t$Goes directly to $G_{t+1}$。

Bellman Equation

The following formula isBellman equation.
$$V(s)=r(s)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s)V(s^{\prime })$$
Compared with the original formula of the value function, the Bellman equation allows the usercalculatethe analytical solution ofthe value functionwhenthe reward functionandthe state transition matrix.

The derivation basis of the Bellman equation: ① definition of return; ② definition of reward function; ③ definition of value function. The derivation process is as follows:
$$\begin{array}{rl}V(s)& =E[G_t|S_t=s]\\ & =E[R_t+\gamma R_{t+1}+c^2{R}_{t+2}+...|S_t=s]\\ & =E[R_t+c(R_{t+1}+\gamma R_{t+2}+...)|S_t=s]\\ & =E[R_t+c{G}_{t+1}|S_t=s]\\ & =E[R_t|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =r(s)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s\left)E\right[G_{t+1}|S_{t+1}=s^{\prime }]\\ & =r\left(s\right)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s)V(s^{\prime })\end{array}$$
During the derivation process, the conversion from line 1 to line 2 and the conversion from line 3 to line 4 use the definition of Return ;

The transformation from line 5 to line 6 uses the definition of the reward function ;

The transformation from line 6 to line 7 uses the definition of the value function .

Here I explain why the second sub-item in line 6 is $E[G_{t+1}|S_t=s]$ into${\sum }_{s^{\prime }\in S}P(s^{\prime }|s\left)E\right[G_{t+1}|S_{t+1}=s^{\prime }]$, instead of${\sum }_{s^{\prime }\in S}P(s^{\prime }|s)G_{t+1}$(The difference between these two ideas is only whether there is a pair of $G_{t+1}$Seek expectations) to give their own explanation: although the latter seems to be more in line with the formula form of mathematical expectations: state $s$ according to the probability$P(s^{\prime }|s)$transition to state$s^{\prime }$ , the object multiplied later should be the corresponding state$s^{\prime }$ return$G_{t+1}$. But the above mentioned when explaining the definition of value : for the same initial state $s$ , its$G_t$Can be different . That is, $s^{\prime }$ and$G_{t+1}$There is not a one-to-one correspondence, so they cannot be directly multiplied. At this time, it is necessary to find a method that can be compatible with $s^{\prime }$ One-to-one correspondence, and can generalize state$s^{\text{'}}$ The concept of earnings. At this time,valuebecomes a good choice. It is for$s^{\prime }$ can correspond to multiple$G_{t+1}$Calculate the mathematical expectation and approximate the state $s^{’}$ (In this way, line 6 can also be omitted, and line 7 can be derived directly).

Bellman Expectation Equation

In fact, the Bellman expectation equation is the previous Bellman equation, just introducing the action $a$ is more complete after that.

According to the two relations of the state value function and the action value function, the Bellman expectation equation of the two value functions can be deduced.

Equation 1

$$V^{\pi }\left(s\right)=\sum _{a\in A}\pi (a|s)\left[r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a)V^{\pi }(s^{\prime })\right]$$
Subject to the following: Level 2 and Level 1, given:
$$\begin{array}{rl}{V}^{\pi }\left(s\right)& =\sum _{a\in A}\pi (a|s\left)Q^{\pi }\right(s,a)\\ & =\sum _{a\in A}\pi (a|s)\left[r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a)V^{\pi }(s^{\prime })\right]\end{array}$$

Equation 2

$$Q^{\pi }(s,a)=r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)\sum _{a^{\prime }\in A}\pi \right(a^{\prime }|s^{\prime })Q^{\pi }(s^{\prime },a^{\prime })$$

推导过程如下：将关系1式代入关系2式，得：
$$\begin{array}{rl}{Q}^{\pi }(s,a)& =r(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)V^{\pi }(s^{\prime })\\ & =r(s,a)+c\sum _{s^{\prime }\in S}P(s^{\prime }|s,a\left)\sum _{a^{\prime }\in A}\pi \right(a^{\prime }|s^{\prime })Q^{\pi }(s^{\prime },a^{\prime })\end{array}$$

Bellman Optimal Equation

To be added……