Strengthening the construction of the learning environment gymnasium

0. Preface

Gym is currently one of the most commonly used tools for reinforcement learning and has been iteratively upgraded. In 2021, the gym library will no longer be updated, and gymnasium will be launched as an ^alternative1 .

The main difference between gymnasium and gym is that the number of return parameters of resetand has changed. For specific changes, see version changes .step

The environment used in this article is:

• python: 3.9
• pipenv: 2023.3.20
• gymnasium: 0.28.1
• Operating system Windows 10 and LinuxMint 20.3 (Ubuntu 20.04)

1. Environment construction

This article uses pipenv as a virtual environment solution to build the environment, mainly including the configuration of gymnasium (including atari). After building a complete reinforcement learning environment, libraries such as CUDA, PyTorch, and opencv need to be installed.

1.1 Virtual environment configuration

Install pipenv:

pip install pipenv -i https://pypi.tuna.tsinghua.edu.cn/simple/

Pay attention to version compatibility issues here. According to the official statement, pipenv is suitable for python 3.7 and ^above2 .

1.2 Install gymnasium

In reinforcement learning, Atari games are one of the classic experimental environments. Gymnasium does not include atari by default. This article will install the gymnasium version that includes atari.

Create a new virtual environment and enter:

mkdir -p ~/rl
cd ~/rl
pipenv shell	# 在当前工作目录进入虚拟环境

PipfileAt this point, you will be prompted to create a new virtual environment, and you can find more virtual environment configuration files named in this directory .

Next install gymnasium:

pipenv install gymnasium[atari] gymnasium[accept-rom-license]

gymnasium[atari]Note that both must be installed here gymnasium[accept-rom-license], otherwise the atari environment cannot be successfully established.

2. Code testing

The list of Atari included in gymnasium can be found in the official website documentation . Here we use the game Pong as an example, and the version uses NoFrameskip-v4.

Build gym_test.py:

import gymnasium as gym

env_name= 'PongNoFrameskip-v4'
print(f'gymnasium version: {
      
      gym.__version__}')
env = gym.make(env_name)

Run the script, if a result similar to the following appears, it means that gymnasium including atari is installed successfully.

(rl) aa@bb:~/rl$ python env_test.py
gymnasium version: 0.28.1
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]

3. Version changes

3.1 resetand stepmethod

• In the old version, reset()the method only returns the reset environment observation value, while the new version reset()method returns the environment observation value and some information:

gymnasium.Env.reset(self, *, seed: int | None = None, options: dict[str, Any] | None = None) → tuple[ObsType, dict[str, Any]]

• In the old version, step()the method returns the next observation, immediate reward, whether it is over and additional information, while the new version of the step()method returns the next observation, immediate reward, whether it is over, whether it is timed out and additional information, and there is an additional return value of whether it is timed out:

gymnasium.Env.step(self, action: ActType) → tuple[ObsType, SupportsFloat, bool, bool, dict[str, Any]]

3.2 wrappers.Monitor

gymnasium.wrappers.Monitorhas been removed, direct calls will report the following error:

AttributeError: module 'gymnasium.wrappers' has no attribute 'Monitor'

gymnasium provides gymnasium.wrappers.RecordVideoto provide video recording function:

class gymnasium.wrappers.RecordVideo(env: Env, video_folder: str, 
									episode_trigger: Callable[[int], bool] | None = None, 
									step_trigger: Callable[[int], bool] | None = None, 
									video_length: int = 0, name_prefix: str = 'rl-video', 
									disable_logger: bool = False)

reference link

1. Reinforcement Learning Environment Upgrade - From gym to Gymnasium
2. Gym Atari: Gym no longer distributes ROMs.
3. Atari
4. AttributeError: module 'gym.wrappers' has no attribute 'Monitor'_tooony_'s Blog-CSDN Blog

---

1. github/gym ↩︎

2. pypa/pipenv: Python Development Workflow for Humans. ↩︎