Reinforcement study notes: Q-learning

This article is organized in the reinforcement learning tutorial of datawhalechina.github.io and the reinforcement learning tutorial of Mofan python

This article should be short, if it is longer, it will take a few hours at every turn. I really can't stand it.

0x01 Introduction

In reinforcement learning, $Q^{\pi }(s,a)$ ,$Q(s,a)$ , which means that in state$Under\; s$ , the current step adopts the strategy$a$ , all subsequent steps are based on the given policy$\pi$ to decide the expected value of the reward to get until the end of the round.

In order to maximize the reward, here the agent's strategy $\pi$ can be considered to be in the current state$Under\; s$ , the mindless choice makes$Q^{\pi }(s,a)$ the largest$a$。

The above figure is the algorithm flow chart of Q-learning. the policy of a given agent $\pi$ , we need to update Q π ( s , a )through such an algorithm$Q^{\pi }(s,a)$ ,$Q(s,a)$ . In order to maximize the benefits, the agent is in each state$s$ will mindlessly choose$Q^{\pi }(s,a)$ Action$a$。

Suppose the environment state is $s$ , the approach of Q-learning is:

First, according to the current $s$ select an action$a$ , this actionaaThe selection strategy of a is$ϵ$$ϵ-greedy$ , i.e. a high probability selection such that$Q(s,a)$ value (then$Q(s,a)$ has not been trained yet) reaches the maximum$a^{\ast }$ , but there will be$ϵ$ probability to randomly choose an action$a$。

Next, we choose action $After\; a$ , you can take$a$ , then get reward$r$ , and reach the next state$s^{\prime }$。

Next, update $Q(s,a)$：

$$Q(s,a)=(1-a)Q(s,a)+\alpha (r+c\underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime }))$$

where $\alpha$ is the learning rate,$\gamma$ is the discount factor. Suppose we initialize Q ( s , a ) Q(s, a)completely randomly$Q(s,a)$ value, but we gradually introduced the return$r$ . And, because the learning rate$The\; existence\; of\; \alpha$ , the previous initial term will be slowly diluted byrrQ ( s , a ) Q(s, a)composed of$r$$Q(s,a)$ More and more values ​​are retained.

Finally, repeat the above action, $Q(s,a)$ The value gradually converges.