[Reinforcement Learning] 01 - Introduction to Reinforcement Learning

Two types of machine learning

The relationship between supervised learning/unsupervised learning/reinforcement learning/machine learning

• predict
  • Predict the desired output based on the data (supervised learning) $P(y|x)$
  • Generate data instances (unsupervised learning) $P(x,y)$
• decision making
  • Taking action in dynamic environments (reinforcement learning)
    • transition to new state
    • Get instant rewards
    • Maximize cumulative rewards over time

The difference between prediction and decision-making: whether actions will cause the environment to change.

The difference between reinforcement learning and other machine learning :

• No supervision, only reward signals;
• feedback delay;
• Time series, there are correlations or dependencies between different data (Non iid data);
• The actions of agents will affect the data sequence they receive.

In reinforcement learning, data is obtained during the interaction between the agent and the environment. If the agent does not take a certain decision-making action, the data corresponding to the action can never be observed, so the training data of the current agent comes from the decision-making results of previous agents. Therefore, the data distribution generated by the agent's interaction with the environment is different depending on the agent's strategy.



There is a concept about data distribution in reinforcement learning, called occupancy measure . The normalized occupancy measure is used to measure a specific state action sampled during the interaction between an agent's decision-making and a dynamic environment. Probability distribution of state-action pairs.

Occupancy metrics have a very important property: given two policies and two occupancy metrics obtained by interacting with a dynamic environment, then the two policies are the same if and only if the two occupancy metrics are the same. That is, if an agent's strategy changes, the occupancy measure obtained from its interaction with the environment will also change accordingly.

reinforcement learning definition

Reinforcement learning uses the concept of agent to represent a machine that makes decisions. Compared with the "model" in supervised learning, the "agent" in reinforcement learning emphasizes that the machine can not only perceive the surrounding environment information, but also directly change the environment by making decisions, rather than just giving some prediction signals .

Reinforcement learning: Computational methods for achieving goals by learning from interactions

• Perception: Being aware of one’s surroundings to some extent
• Action: Taking action to affect a state or achieve a goal
• Goal: Maximize rewards over time

Reinforcement learning interaction process

Every step of Agent $t$:

• Get observations $O_t$
• Get reward $R_t$
• Execute action $A_t$

Every step of the environmenttt$t$:

• Get action $A_t$
• Given the observation $O_{t+1}$
• Give reward $R_{t+1}$

In the environment step $t=t+1$

Reinforcement learning system elements

History

Past $O_i,R_i,A_i$的序列
$$H_t=O_1,R_1,A_1,...,A_{t-1},O_t,R_t$$

• Until ttAll observable variables at time$t$
• Determine the next step based on history: (Agent: $A_i$; Env: $O_{i+1},R_{i+1}$)

State

Used to determine what will happen next ( $O,R,A$)

• is a function about history $S_t=f(H_t)$ $f\left(H_t\right)$ is difficult to obtain directly in some cases (POMDP)

Policy

• The behavior of the agent, mapping from state to action
• Deterministic policy: $a=\pi \left(s\right)$
• 随机策略(Stochastic policy): $\pi (a|s)=\mathbb{P}[A_t=a|S_t=s]$

Reward

• Define the scalar of the reinforcement learning goal and evaluate the quality of the state

Value Function

• Forecast of future cumulative rewards
• Used to evaluate the quality of the state under a given strategy
  $$v_{}(s)={\mathbb{E}}_{}\left[R_{t+1}+\gamma R_{t+2}+c^2{R}_{t+3}+...\mid S_t=s\right]$$

Model

• Used to predict what the environment will do next
• Predict the next state ${\mathcal{P}}_{s{s}^{\prime }}^a=\mathbb{P}[S_{t+1}=s^{\prime }\mid S_t=s,A_t=a]$
• Predict the next (immediate) reward ${\mathcal{R}}_s^a=\mathbb{E}\left[R_{t+1}|S_t=s,A_t=a\right]$

Maze example

find the shortest path

• Reward $R$ : -1 for each step;
• Movement $A$: N, E, S, W；
• Status $S$ : Agent’s location;
• The arrow represents the strategy $\pi \left(s\right)$ ;
• The number represents the value of each step $v_{}\left(s\right)$ (number of cells from Goal);

Reinforcement learning agent classification

• Model-based reinforcement learning
  • Strategy (and/or) Value Function
  • environment model
  • Example: Maze, Go
• Model-agnostic reinforcement learning (usually we don’t know the model of the environment accurately)
  • Strategy (and/or) Value Function
  • No environment model
  • Atari Example

Atari Example

• Rules unknown
• Learning from interaction (the environment is a black box)
• Select actions on the joystick and view scores and pixel graphics

---

Other types

• value based

  • No strategy (implied)
  • value function

• Based on strategy

  • Strategy
  • no value function

• Actor-Critic

  • Strategy
  • value function

relationship between types

---

The essential way of thinking of reinforcement learning:

The strategy of reinforcement learning will be continuously updated during training, and its corresponding data distribution (i.e., occupancy measure) will also change accordingly. Therefore, a major difficulty in reinforcement learning is that the data distribution seen by the agent constantly changes as the agent learns.

Since rewards are based on state-action pairs, the value corresponding to a strategy is actually the expectation of the corresponding reward under an occupancy metric. Therefore, finding the optimal strategy corresponds to finding the optimal occupancy metric.

Reinforcement learning focuses on finding an agent strategy that produces an optimal data distribution in the process of interacting with a dynamic environment, that is, maximizing the expectation of a given reward function under the distribution.

reference

[1] Boyu AI
[2] https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver
[3] Hands-on reinforcement learning