Model-Based Reinforcement Learning and Model-Free Reinforcement Learning are two different reinforcement learning methods.
Modeled reinforcement learning means that the agent can model the environment during the learning process, that is, the dynamic transfer function of the learning environment (also known as the state transfer function or transfer model). Based on the environment model, the agent can predict the future state and rewards, and then formulate better decision-making strategies. In modeled reinforcement learning, the agent needs to go through two processes: learning a model of the environment and making decisions based on the model. Since model-based reinforcement learning requires learning a model of the environment, more training data and computing resources may be required.
Model-based reinforcement learning methods usually include the following types of algorithms:
Dynamic programming-based algorithms: such as Value Iteration (Value Iteration) and Policy Iteration (Policy Iteration) and other algorithms. These algorithms use dynamic programming methods to solve optimal decision-making strategies by establishing state transition models and reward function models.
Algorithms based on model prediction: such as Model Predictive Control (Model Predictive Control, MPC) and Model-Based Reinforcement Learning (Model-Based Reinforcement Learning) and other algorithms. These algorithms develop better decision-making strategies by learning a model of the environment, predicting future states and rewards.
Algorithms based on gradient descent: such as policy-based gradient descent (Policy Gradient) and Actor-Critic algorithms. These algorithms use the gradient descent method to solve the optimal decision-making strategy by establishing a state transition model and a reward function model.
Model-free reinforcement learning means that the agent does not need to model the environment during the learning process. Instead, the agent learns a decision policy by interacting with the environment, directly learning the mapping between states and rewards. In model-free reinforcement learning, the agent only needs to go through one process: making a decision based on the current state and the reward. Since model-free RL does not require learning a model of the environment, it is generally easier to implement and run than model-based RL.
Application of model-based reinforcement learning:
Board games: Model-based reinforcement learning algorithms (such as MCTS) are widely used in board games, such as Go, chess, etc. AlphaGo and AlphaZero are typical examples of using MCTS
Path planning: Model reinforcement learning algorithms (such as dynamic programming) can be used for path planning problems, such as robot navigation, UAV path planning, etc.
Resource scheduling: Model-based reinforcement learning algorithms can be used to optimize resource scheduling problems, such as task scheduling in data centers, path planning for logistics distribution, etc.
Application of model-free reinforcement learning:
Game AI: Model-free reinforcement learning algorithms (such as DQN) are widely used to train game agents, such as Atari games, Flappy Bird, etc.
Autonomous Driving: Model-free reinforcement learning algorithms can be used to train control policies for self-driving cars for safe and efficient driving.
Robot Control: Model-free reinforcement learning algorithms can be used to train robots to perform various tasks such as navigation, grasping, flying, etc.
Image reference from:
Mushroom Book EasyRL: datawhalechina.github.io/easy-rl/#/chapter3/chapter3?id=_313-%e6%9c%89%e6%a8%a1%e5%9e%8b%e4% b8%8e%e5%85%8d%e6%a8%a1%e5%9e%8b%e7%9a%84%e5%8c%ba%e5%88%ab