1. Human intervention reinforcement learning
Deep reinforcement learning has made amazing progress in some board games, video games, and navigation and control tasks in realistic 3D environments. These achievements were achieved in a simulated environment. Can deep reinforcement learning translate this success to real-world tasks?
There are two main problems faced here. The first is that deep reinforcement learning requires a large number of observations (obtaining these observations in real-world tasks is slow and expensive). The second obstacle to practical applications of reinforcement learning is safety. Model-free reinforcement learning agents can only learn through trial and error. In order to learn to avoid disaster, they first need to cause it. In a game, it is okay for a reinforcement learning agent to die numerous times during training. However, in real-world tasks, once is never too many.
Self-driving cars hit pedestrians, expensive robots break their own hardware, news feed algorithms display fake stories or share users’ private information, chatbots say offensive things to customers, and the list goes on.
Current deep reinforcement learning agents must take these disastrous actions many times before they learn to avoid them. Additionally, they may “forget” that these behaviors are bad and only remember them by trying them again, which is known as the Sisyphus Curse of deep reinforcement learning.
How do artificial intelligence systems learn safely in the real world? Self-driving cars are equipped with safety drivers who sit in the driver's seat and constantly monitor the road, ready to take control measures if an accident is about to occur. Can reinforcement learning systems also learn safely with human supervisors? One big problem is that learning a simple video game can require an RL system to process hundreds of millions of frames. If one person had to watch and check every frame, the training time would take more than a year.
2. Security reinforcement learning program
HIRL (Human Intervention Reinforcement Learning) applies human supervision to reinforcement learning agents to achieve safe learning. At the beginning of training, agents are supervised by disaster prevention personnel. Then train supervised learning