Reinforcement Learning [Getting Started]

Reinforcement Learning [Getting Started]

concept

Action : action

State : state

Reward : Reward

$\pi (a|s)$ : Policy function, which is a probability density function, which refers to the probability of each action in the current state.

S'~P(*|a, s): State transition function, which is also a probability density function. According to the current state and current action, it returns the probabilities of the next state (because mobs are random).

观察s1--->根据策略函数获得a1--->生成下一个状态s2--->返回一个奖励r1
再将新的状态s2作为输入，根据策略函数获得a2……
轨迹：
s1,a1,r1,s2,a2,r2,…,sT,aT,rT

Return : return. Future accumulated rewards
$$U_t=R_t+R_{t+1}+R_{t+2}+\dots$$
$R_t$and $R_{t+1}$are not equally important, obviously $R_t$more important.

折扣回报：
$$U_t=R_t+\gamma R_{t+1}+c^2{R}_{t+2}+c^3{R}_{t+3}+\dots$$
The discount rate is a hyperparameter and needs to be tuned yourself.

Action-value function:

With the discount reward, we can know whether the game is about to win or lose, and the bigger the future reward, the better.

But the discount report is a random variable, it depends on at, st, at+1, etc., how to evaluate the discount return? Use expectations! Both future states and actions are eliminated using points. It is also related to the policy function, because there can be different policy functions.
$$Q_{}(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t].$$
Optimal action-value function:

How to remove the Policy function? Use it to the max!
$$Q^{\star }(s_t,a_t)=\underset{Pi}{\max }{Q}_{}(s_t,a_t).$$
This function can evaluate the current action a, and can tell us whether the current action is good or not.

State value function (state-value function):

离散情况：
$$V_{}\left(s_t\right)={\mathbb{E}}_A\left[Q_{}(s_t,A)\right]=\sum _a\pi (a|s_t)\cdot Q_{}(s_t,a)$$
连续情况：
$$V_{}\left(s_t\right)=E_A\left[Q_{}(s_t,A)\right]=\int \pi (a|s_t)\cdot Q_{}(s_t,a)da$$
takes the action A as a random variable, and seeks to eliminate A.

The state value function can evaluate the current situation and judge whether we are about to win or lose.

The state value function can also evaluate the policy function $\pi$ is good or bad.

How to control Agent?

1. Based on the Policy function $\pi (a|s)$
   • Observation state $s_t$
   • Get the random variable $a_t$~ $\pi (\ast |s_t)$
2. Based on $Q^{\star }(s,a)$ function
   • Based on the observed state s
   • Input each a to get the a that maximizes Q: $a_t={\mathrm{argmax}}_a{Q}^{\star }(s_t,a)$

**Reinforcement learning library:**OpenAI Gym

Reinforcement learning is mainly to learn the Policy function $\pi (a|s)$ gives$Q^{\star }(s,a)$ With one of the two functions, the Agent can be controlled.

value learning

Deep Q Network（DQN）

Use a neural network to approximate $Q^{\star }(s,a)$ function

Goal: The larger the sum of the rewards obtained at the end of the game, the better.

$Q^{\star }(s,a)$ Each action can be evaluated, and for the expectation of future value, the action with greater value expectation is better.

Using a neural network $Q(s,a;w)$ Approximate$Q^{\star }(s,a)$ function, the parameter of the neural network is w.

How to train DQN?

Temporal Difference（TD） Learning

If I want the model to predict the time it takes from point A to point B, the model will give a predicted value. If my real value is only from A to C (C is a point between A and B), How should I train? TD algorithm! The model can predict the usage time from C to B, plus the actual usage time from A to C, and use this time (TD target) for gradient descent.

Therefore, the TD algorithm can also be used to update the parameters without finishing the game.

The TD algorithm must have a formula similar to the above in the figure above. The left side is the predicted value of the model, and the right side is the sum of a part of the real value and a part of the predicted value.

In deep reinforcement learning, there happens to be such a formula:
$$Q(s_t,a_t;w)\approx r_t+c\cdot Q(s_{t+1},a_{t+1};w)$$
公式看$U_t=R_t+r\cdot U_{t+1}$

And Q is an estimate of U, so there is the above formula.

• Prediction: At time t, the model makes a prediction $Q(s_t,a_t;{\mathbf{w}}_t)$.
• TD target: At time t+1, the real return $R_t$, and the state S t + 1 S_{t+1} at time t+1$S_{t+1}$, then you can calculate $a_{t+1}$, so that the TD target can be calculated.

$$\begin{array}{c}{y}_t=r_t+c\cdot Q(s_{t+1},a_{t+1};{\mathbf{w}}_t)\\ =r_t+c\cdot {\max }_{a}Q(s_{t+1},a;w_t).\end{array}$$

• Loss：
  $$L_t=\frac{1}{2}[Q(s_t,a_t;\mathbf{w})-y_t{]}^2$$

strategy learning

Approximate the policy function using a neural network.

• ${\sum }_{a\in \mathcal{A}}\pi (a|s;i)=1$

Just replaced the policy function with a neural network, then the state value function
$$V_{}(s_t)={\mathbb{E}}_A\left[Q_{}(s_t,A)\right]=\sum _a\pi (a|s_t)\cdot Q_{}(s_t,a)$$
Independently
$$V(s_t;i)=\sum \pi (a|s_t;i)\cdot Q_{}(s_t,a).$$
V can evaluate the current state and the quality of the strategy function. How to make the strategy function better and better?

Change the neural network parameters $\theta$ to make this value bigger and bigger.

Define the objective function as $J(\theta )={\mathbb{E}}_S[V(S;\theta )]$ , remove the state S as a random variable, so that only the evaluation of the policy network remains.

• Observation status s
• Derivation, so that the gradient of the above objective function rises.

This is a stochastic gradient, where the randomness comes from s, which is obtained using random sampling.

We have the following equations:
$$\frac{\partial V(s;i}{\partial \theta }=\sum _a\frac{\partial \pi (a|s;i}{\partial \theta }\cdot Q_{}(s,a).$$

$$\frac{\partial V(s;i}{\partial \theta }={\mathbb{E}}_{A\sim \pi (\cdot |s;\theta {)}_c}[\frac{\partial \log \pi (A|s,i}{\partial \theta }\cdot Q_{}(s,A)]$$

If the actions are discrete, then use the above formula to calculate each action separately and add them together.

If the action is continuous, use the following formula, the neural network integral is not easy to calculate. Monte Carlo approximation! Take one or more samples to approximate.

How to calculate Q? Two ways.

1. You can first use the strategy function to play the game from beginning to end, and record each state, action, and reward. These recorded data are then used for approximation.
2. Approximated using a neural network.

Actor-Critic method

Actor: Policy network, used to control agent movement (Policy Network)

Critic: Value Network, used to score actions (Value Network)
$$V_{}(s)=\sum _a\pi (a|s)\cdot Q_{}(s,a)$$
Use two neural networks respectively, one approximating$\pi$ (known as a policy network, also known as an Actor), an approximation to$Q_{}$(Value Network, value network), the value network is to score actions.

Thus, $V_{}$In general:
$$V_{}(s)=\sum _a\pi (a|s)\cdot Q_{}(s,a)\approx \sum _a\pi (a|s;i)\cdot q(s,a;w)$$
The product of the policy network and the value network.

Policy network:

Value Network:

network training

$$V(s;i,\mathbf{w})=\sum _a\pi (a|s;i)\cdot q(s,a;w)$$

Training: Update parameters $\theta andw$ .

• The purpose of updating the strategy network is to increase the value of V, make the strategy better and better, and make the score of q higher and higher.
• The purpose of updating the value network q is to make the scoring value more accurate, so as to better estimate the sum of future values.

Five-step update:

1. Observed state $S_t$。
2. Randomly sample actions at a_t according to the policy network$a_t$。
3. Execute action $a_t$, update state $S_{t+1}$, get the reward $r_t$
4. Use the TD algorithm to update the parameter w of the value network.
5. Use the policy gradient algorithm to update the parameters $\theta $ of the policy network, using the parameters of the value network.

Algorithm steps:

1. Observed state $S_t$And randomly sample $a_t$。
2. Execute action $a_t$, the environment gives a new state $S_{t+1}$and reward $r_t$。
3. According to $S_{t+1}$And policy network, random sampling to obtain a new action ${\tilde{a}}_{t+1}$。
4. Evaluate qt = q ( st , at ; wt ) according to the value network q_{t}=q(s_{t},a_{t};\mathbf{w}_{t})$q_t=q(s_t,a_t;{\mathbf{w}}_t)$和$q_{t+1}=q(s_{t+1},{\tilde{a}}_{t+1};w_t)$
5. Use the TD algorithm to calculate the loss $d_t=q_t-(r_t+c\cdot q_{t+1})$
6. Deriving the value network to calculate the gradient ${\mathbf{d}}_{\mathbf{w},t}=\frac{\partial q(s_t,a_t;\mathbf{w})}{\partial \mathbf{w}}{\mid }_{\mathbf{w}={\mathbf{w}}_t}$。
7. 更新网络${\mathbf{w}}_{t+1}=w_t-a\cdot d_t\cdot d_{w,t}$。
8. Let the exponential function ${\mathbf{d}}_{i,}=\frac{\partial \log \pi (a_t|s_t,i}{\partial \theta }{\mid }_{\theta ={\theta }_t}$。
9. The functional derivatives are given by $i_{t+1}=i_t+b\cdot d_t\cdot d_{i,}$。

reference:

https://www.bilibili.com/video/BV1We4y1w7Us?p=6&spm_id_from=pageDriver&vd_source=77cb10b9cc158f4815e8d992103d448b