Deep Reinforcement Learning - Chapter 6~8 Q-Learning

Reference notes and columns:

1. Tianjin Baozi stuffing's Zhihu deep reinforcement learning column
2. Datawhale deep reinforcement learning notes

1. Q-Learning concept

The Q-learning method is a different-strategy time-difference method.
Among them, the different strategy refers to the strategy that the Agent chooses Action, corresponding to ϵ \epsilon in line 5 of the pseudocode$ϵ$ Greedy policy
Note that the policy is not evaluated (line 6).

The time difference methodrefers to using the time difference target to update the current state-action value function Q. The time difference target is$r_t+\gamma maxQ(S_{t+1},a)$

What Q-Learning needs to learn is a Policy Evaluation Function (Policy Evaluation Function) that evaluates the Agent's choice of Action strategy.

2. Q-Learning Function

2.1 State Value Function Estimation$V^{\pi }\left(s\right)$

Given a certain state, Q-Learing continuously assumes that the next interacting actor is $\pi$ , the expected value V π ( S ) V^{\pi}(S)of reward accumulated until the end of the interaction$V^{\pi }\left(S\right)$.

$V^{}$The INPUTof${}^{\pi }$$($$S$$)$is a state s, andthe OUTPUTis a scalar (that is, it is continuously assumed that the next interacting actor is$\pi$ , the reward accumulated until the end of the interaction)

1. Approximate with Monte-Carlo(MC) based method

   Assume that the parameters in the neural network are the weights $\theta$ , then the state-action value function is$Q(S_{t+1},a)$ , the cumulative reward function is$G$(当 input $state{s}_a$, the correct output should be $G_a$

2. Use Temporal-difference (timing difference) method to approximate
   a given $state$ $s_t$, take $action$ $a_t$, get reward $r_t$, jump to $state$ $s_{t+1}$, you can apply TD:

   ​ $V^{\pi }(s_t)$ $=V^{\pi }(s_{t+1})+r_t$

   put $s_t$Throw it into the network and you will get $V^{\pi }\left(s_t\right)$ , put$s_{t+1}$Throwing it into the network will get $V^{\pi }\left(s_{t+1}\right)$ . Through training, update$V^{\pi }\left(s\right)$parameter, when the result of their two subtraction and$r_t$The closer, $V^{\pi }\left(s\right)$is learned.

Because the MS method is subject to $G_a$The impact of randomness is greater than that of the TD method, so the TD method is more common.

2.2 State-action Value Function $Q^{\pi }(s,a)$

$Q^{\pi }(s,a)$ TheINPUTis a state s and action a, andthe OUTPUTis the expected value of accumulated reward.

There are two ways to write Q-function:

• input is state and action, output is a scalar;
• The input is a state s, and the output is several values.

for example:

Suppose we have 3 actions, and the 3 actions are stay still, up, and down.

Assuming it is in the first state, no matter which action is taken, the expected reward obtained at the end of the game is actually the same. Because the ball is here, even if you go down, you should actually come for first aid next, so no matter which action you take today, it won't make much difference.

Assume that in the second state, the table tennis ball has bounced very close to the edge. At this time, if you take the upward direction, you can get the positive reward and catch the ball. If you stand still or go down, you will miss the ball next. The reward you get will be negative.

Suppose in the third state, the ball is very close, so it will go up.

Suppose in the fourth state, the ball is bounced back