Reinforcement Learning: Basic Concepts

  Taking grid-world as an example, the basic concepts of reinforcement learning are introduced. As shown in the figure, the robot is in a grid world, and different grids represent different functions. White represents that the robot can enter freely, yellow represents a trap (once the robot enters, it will be forced to return to the starting point), and blue represents the end point. , and now the robot is required to learn by itself and walk the shortest path from the starting point to the ending point.

state state

State: the possible state $S_i$, the state shown in the figure has $s1$,$s2$, ···,$with9$

State collection: the collection of all possible states of the system at each stage $S$=$\{\begin{array}{l}{S}_i\end{array}\}$

behavior action

Behavior: possible actions $a_i$. In the grid-world, there are 5 behaviors of robots:
  $a1$ : move up;
  $a2$ : move right;
  $a3$ : move down;
  $a4$ : move left;
  $a5$ : Keep the original position unchanged;

Behavior set: the set of all possible actions $A(S_i)$=$\{\begin{array}{l}{a}_i\end{array}\}$

state transition

State transition: A state will transfer to another state after taking a behavior $S_1—a2—>S_2$

State Transition Probability: Describe state transitions with probabilities!
       $p(s_j|s_i,a_n)$ : means in state$s_i$take action $a_n$reach state $s_j$The probability of
       such as: $p(s_2|s_1,a_2)=1$

strategypolicy

Strategy: Based on the ultimate purpose, guide what behavior should be taken in the current state. Each state corresponds to a strategy.
   For example, if a strategy is given now (the green arrow indicates the strategy), as shown in the figure below, different starting points will be obtained based on the given strategy The path to reach the destination.

In specific problems, the conditional probability representation strategy in mathematics is often used, and π π    is habitually used in reinforcement learning$\pi$ represents the strategy,$\pi (a_n|s_i)$ means that in state$s_i$Take action under the condition $a_n$The probability.
   in state $Taking\; s1$ as an example, the mathematical expression of the strategy given the arrow above is as follows (deterministic probability):
$$\pi (a_1|s_1)=0$$$$\pi (a_2|s_1)=1$$$$\pi (a_3|s_1)=0$$$$\pi (a_4|s_1)=0$$$$\pi (a_5|s_1)=0$$
   to state$s1$ as an example, the strategy of uncertainty probability:

$$\pi (a_1|s_1)=0$$$$\pi (a_2|s_1)=0.5$$$$\pi (a_3|s_1)=0.5$$$$\pi (a_4|s_1)=0$$$$\pi (a_5|s_1)=0$$

In programming, policies are usually expressed in the form of an array (matrix):

reward reward

   In reinforcement learning, the reward is a real number (scalar) that is obtained after taking an action. There are positive and negative rewards. If the reward is positive, it means that we encourage this kind of behavior; if the reward is negative, it means that we don’t want to take this kind of behavior, which is essentially a punishment for taking this kind of behavior.
   Rewards can be understood as a means of interaction between humans and robots. Through rewards, robots can be guided to act according to our expectations and achieve our goals. Rewards must depend on the current state and the actions taken, and are given based on the actions taken.

   In grid-world, the reward rules are as follows:

	1、如果机械人试图走出边界，则奖励reward=-1		
	2、如果机械人试图进入禁止的单元格，则奖励reward=-1	
	3、如果机械人到达目标单元格，则奖励reward=+1	
	4、其他情况，奖励reward=0	

   In reinforcement learning, the conditional probability is used to express the reward obtained by taking an action.
$$p(r|s_i,a_n):\; means\; in\; state{s}_{i}take\; action\; aunder\; the\; condition{}_{n}The\; probability\; of\; getting\; a\; reward$$
   is in the state$s_1$As an example (since we have designed the reward rules, the rewards obtained are deterministic, but in actual situations the rewards obtained are not deterministic): p ( $p(r=-1|s_1,a_1)=1$

state-action-return trajectory

   In reinforcement learning, trajectory is a chain that records the behavior taken from the starting point to the end point, the reward obtained, and the state transition, including three changes of state, reward, and behavior. Sampling (return) is a very important probability in the trajectory. It is for a trajectory, and its function is to add up all the rewards obtained along a trajectory.

.

   As shown in the figure above, different strategies result in different trajectories. So how do you evaluate the pros and cons of different strategies? In reinforcement learning, return is usually used to evaluate the pros and cons of a strategy.

Attenuation coefficient γ γThe introduction of$\gamma$

   is shown in the figure above. When the robot reaches the end point, the strategy is still in progress, and it will continue to enter the end point repeatedly, which will make the return diverge. To solve this problem, we introduce the discount rate$c\in [0,1)$ After introducing the attenuation coefficient, you will get the discount return

and you can find$The\; role\; of\; \gamma$ :
   1. The guaranteed reward is limited
   2. It can balance the long-term and short-term rewards
    if$If\; the\; \gamma$ value is small, it means that we will pay more attention to the recent rewards, that is, the final reward is determined by the initial reward
    if$If\; the\; \gamma$ value is too large, it means that we pay more attention to long-term rewards, making the robot far-sighted

episode

   When the robot interacts with the environment according to the strategy, the robot will stop when it reaches the end point, and the resulting trajectory is called an episode.
   In a grid world, should you stop when you reach your goal? In fact, we can do it with a unified math.

Markov decision process MDP (Markov dwcision process)

   A Markov decision process consists of three elements:
1. Set:
$$State\; set:S$$$$Behavior\; set:A\left(s\right)$$$$Reward\; set:R(s,a)$$

2. Probability:
$$State\; transition\; probability:p(s^{\prime }|s,a)$$ $$Reward\; probability:p(r|s,a)$$

3. Strategy:
$$Strategy:\pi (a|s)$$

  Markov property : No aftereffect, which means that the current decision is only related to the current state and goals, and has nothing to do with the past.

The grid-world grid world can be abstracted into a more general model, the Markov process. Circles represent states, links with arrows represent state transitions, and a Markov decision process becomes a Markov chain once a policy is given.