(2) Deep reinforcement learning foundation [value learning]

Value-Based Reinforcement Learning

review：

Definition: Discounted return(cumulative discounted future reward)
$\cdot$ $U_t=R_t+\gamma R_{t+1}+c^2{R}_{t+2}+c^3{R}_{t+3}+...$

$\cdot$ The return depends on action $A_t,A_{t+1},A_{t+2},...$ and states $S_t,S_{t+1},S_{t+2},...$
$\cdot$ Actions are random: $P[A=a|S=s]=\pi (a|s).$        \;\;\;(Policy function)
$\cdot$ States are random: $P[S^{{}^{\prime }}=s^{{}^{\prime }}|S=s,A=a]=p(s^{{}^{\prime }}|s,a).$        \;\;\; (State transition)

Definition: Action-value function for policy $\pi .$
$\cdot$ $Q_{}(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t].$

$\cdot$ Taken w.r.t actions $A_{t+1},A_{t+2},A_{t+3},...$ and states $S_{t+1},S_{t+2},S_{t+3},...$
$\cdot$ Integrate out everything except for the observations: $A_t=a_t$ and $S_t=s_t.$

Definition: Optimal action-value function
$\cdot$ $Q^{\ast }(s_t,a_t)=\underset{Pi}{\max }{Q}_{}(s_t,a_t).$
$\cdot$ Whatever policy function $\pi$ is used, the result of taking $a_t$ at state $s_t$ cannot be better than $Q^{\ast }(s_t,a_t).$

1. Deep Q-Network(DQN)

Goal: Win the game($\approx$ maximize the total reward.)

Question: If we know $Q^{\ast }(s,a)$, what is the best action?
$\cdot$ Obviously, the best action is $a^{\ast }=arg\underset{a}{\max }{Q}^{\ast }(s,a).$
$(Q^{\ast }$ is an indication for how good it is for an agent to pick action $a$ while being in state $s$).
$Q^{\ast }$ is a prophet who can always guide us to make actions. But in fact, it is impossible to approximate an omnipotent prophet.

Challenge: We do not know $Q^{\ast }(s,a).$
$\cdot$ Solution: Deep Q Network(DQN)
$\cdot$ Use neural network $Q^{\ast }(s,a,w)$ to approximate $Q^{\ast }(s,a)$.

$w$ is the parameter of neural network, $s$ is the input, and the output of neural network is many values, which are the possible scores of all actions. We train the network through rewards, and the scoring of this network will gradually improve and become better and better.

Deep Q Network:
$\cdot$ Input shape: size of the screenshot.
$\cdot$ Output shape: dimension of action space(scoring of each action).

Question: Based on the predictions, what should be the action?
Answer: If the score of that action is high, which action should be used.

2. Temporal Difference (TD) Learning

The most commonly used method for training DQN is TD algorithm.

Example

$\cdot$ I want to drive from NYC to Atlanta.
$\cdot$ Model Q($w$) estimates the time cost, e.g., 1000 minutes.

Qestion: How do I update the model?

$\cdot$ Make a prediction: $q=Q(w),e.g.,q=1000.$

$\cdot$ Finish the trip and get the target$ y, e.g., y = 860.$

$\cdot$ Loss: $L=\frac{1}{2}(q-y{)}^2.$

$\cdot$ Gradient: $\frac{\partial L}{\partial w}=\frac{\partial q}{\partial w}\cdot \frac{\partial L}{\partial q}=(q-y)\cdot \frac{\partial Q(w)}{\partial w}.$

$\cdot$ Gradient descent: $w_{t+1}=w_t-a\cdot \frac{\partial L}{\partial w}{\mid }_{w=w_t}.$

$\cdot$ Can I update the model before finishing the trip?
$\cdot$ Can I get a better $w$ as soon as I arrived at DC?

Temporal Difference (TD) Learning

$\cdot$ Model’s estimate:

                       \;\;\;\;\;\;\;\;\;\;\; NYC to Atlanta: 1000 minutes (estimate).

$\cdot$ I arrived to DC; actual time cost:

                       \;\;\;\;\;\;\;\;\;\;\;NYC to DC: 300 minutes (actual).

$\cdot$ Model now updates its estimate:
                       \;\;\;\;\;\;\;\;\;\;\; DC to Atlanta: 600 minutes (estimate)

$\cdot$ Model’s estimate: $Q(w)=1000minutes$

$\cdot$ Updated estimate: $300+600=900minutes(TDtarget).$

$\cdot$ TD target $y=900$ is a more reliable estimate than $1000$.

$\cdot$ Loss: $L=\frac{1}{2}$ $\underset{\text{TD error}}{\underbrace{(Q(w)-y)}}$${}^2.$

$\cdot$ Gradient: $\frac{\partial L}{\partial w}=\underset{\text{TD error}}{\underbrace{(1000-900)}}\cdot \frac{\partial Q(w)}{\partial w}.$

$\cdot$ Gradient descent: $w_{t+1}=w_t-a\cdot \frac{\partial L}{\partial w}{\mid }_{w=w_t}.$

3. Why does TD learning work?

$\cdot$ Model’s estimates:
           \;\;\;\;\; NYC to Atlanta: $1000$ minutes.
           \;\;\;\;\; DC to Atlanta: $600$ minutes.
           \;\;\;\;\; $\Rightarrow$ NYC to DC: $400$ minutes.

$\cdot$ Ground truth:
           \;\;\;\;\; NYC to DC: $300$ minutes.

$\cdot$ TD error:$d=400-300=100$

4. How to apply TD learning to DQN?

$\cdot$ In the “driving time” example, we have the equation:
$\underset{\text{Model’s estimate}}{\underbrace{T_{NYC\to ATL}}}\approx \underset{\text{Actual time}}{\underbrace{T_{NYC\to DC}}}+\underset{\text{Model’s estimate}}{\underbrace{T_{DC\to ATL}}}.$

The above is the form of TD algorithm.

$\cdot$ In deep reinforcement learning:
$Q(s_t,a_t,w)\approx r_t+c\cdot Q(s_{t+1},a_{t+1};w).$

Prove



  \,
  \,

5. Summary

Definition: Optimal action-value function.

$\cdot$ $Q^{\ast }(s_t,a_t)=\underset{Pi}{\max }E[U_t\mid S_t=s_t,A_t=a_t].$

The $Q^{\ast }$ function can score all actions based on the current state, and the score can reflect the quality of each state. As long as there is a $Q^{\ast }$ function, it can be used to control the movement of the agent. At each moment, the agent only needs to select the action with the highest score to execute this action. However, we don’t have $Q^{\ast }$ function. The purpose of value learning is to learn a function to approximate $Q^{\ast }$ function, so we have $DQN$ .$\_$

DQN: Approximately $Q^{\ast }$ (s,a) using a neural network(DQN).

$\cdot$ $Q^{\ast }(s,a;w)$ is a neural network parameterized by $w$.

$\cdot$ Input: observed state $s$.

$\cdot$ Output: scores for all the action $a\in A.$