Based on deep reinforcement learning training "Street Fighter II: Champion Special Edition" to clear the bottom BOSS-intelligent AI agent project to get started

Introduction to SFighterAI project

An interesting project was reproduced on this machine - based on deep reinforcement learning, an intelligent AI agent was trained for clearing the bottom boss of "Street Fighter II Special Champion Edition".

Project warehouse address: SFighterAI
This project has trained an intelligent AI agent for clearing the bottom boss of "Street Fighter II Special Champion Edition" based on deep reinforcement learning. The intelligent agent makes decisions based entirely on the game screen (RGB pixel value). In the first round of the last level in the given archive of the project, it can achieve a 100% winning rate (in fact, there is an "overfitting" phenomenon, see Discussion in the Running Tests section).

Realize the software environment

Use the official version of virtualbox7.0+ubuntu-22.04.2-desktop-amd64, Python3.8.10, and the environment uses conda version 23.3.1.
py file debugging uses jupyter notebook (according to personal usage habits, not necessary).
AI agent (completely adopts the original author's plan):

	gym==0.21.0
	gym-retro==0.8.0
	stable-baselines3==1.7.0
	tensorboard==2.12.1

The initial attempt to test under the windows11 system was unsuccessful.
Please note that
1. The gcc installed by default in the ubuntu environment has compatibility issues in the conda23.3.1 environment. In the Python environment, it needs to be updated to a gcc compatible with the conda platform. Use the command to update:
bash conda install -c conda-forge gcc
2. Path problem
The path in the test file test.py needs to be adjusted with the actual project path.
Involved files: ./main/test.py
3. Packages to be added: chardet and tensorflow
To avoid compatibility issues, it is recommended to install conda install.
conda install chardet -y
conda install tensorflow -y

Project file structure

├───data
├───main
│   ├───logs
│   ├───trained_models
│   └───scripts
├───utils
│   └───scripts

Game configuration files are stored under data/the folder ; the project's main code folder is main/. Among them, logs/contains the terminal text and data curves that record the training process (use Tensorboard to view); trained_models/contains the model weight files of different stages, which can be used to run tests test.pyin and watch the battle strategy learned by the intelligent agent in different training stages Effect.

run guide

This project is based on the Python programming language and mainly uses standard code libraries such as OpenAI Gym Retro and Stable-Baselines3 . The Python version used by the program is 3.8.10. It is recommended to use Anaconda to configure the Python environment. The original author used the Windows 11 system configuration test to pass, but repeated attempts to reproduce it failed, and various strange errors occurred, so he tried the Ubuntu system to implement it, and it has been successfully reproduced.

Environment configuration

For the detailed process of VirtualBox, conda installation and domestic source replacement, and jupyter notebook installation and debugging, please refer to the blog post:
The following is the main command record of the console/terminal (Console/Terminal/Shell) used in the process.

# 更新conda
# conda update -n base -c defaults conda
conda update --name base conda
# 创建 conda 环境，将其命名为 StreetFighterAI，Python 版本 3.8.10
conda create -n StreetFighterAI python=3.8.10
conda activate StreetFighterAI
# conda deactivate
# conda env remove -n StreetFighterAI

# 安装 Python 代码库
g:&&cd G:\bsp\street-fighter-ai\main
# bash Anaconda3-5.3.1-Linux-x86_64.sh
# conda info
# cd [项目上级文件夹]/street-fighter-ai/main
# 如果提示权限不够，执行：sudo chmod -R 777 ./street-fighter-ai/
# conda activate StreetFighterAI
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple  -r requirements.txt
pip install -r requirements.txt
安装OpenGL库：
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple ffmpeg pyvirtualdisplay retro pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pyglet==1.5.11
sudo apt-get install python-opengl
sudo apt install xvfb
# wheels路径：c:\users\86131\appdata\local\pip\cache\wheels
# pip install setuptools==57.5.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
# requirements.txt:
    gym==0.21.0 
    gym-retro==0.8.0
    stable-baselines3==1.7.0
    tensorboard==2.12.1

Verify and adjust the gym environment:

First add pycharm to configure the conda virtual environment
to strengthen the learning notes: Getting started with Gym – from installation to the first complete code example
Check the available environments for gym:

import gym
from gym import envs
env_list = envs.registry.all()
env_ids = [env_item.id for env_item in env_list]
print('There are {0} envs in gym'.format(len(env_ids)))
env_ids

Test code:

import gym
import time

# 生成环境
env = gym.make('CartPole-v1', render_mode='human')
# 环境初始化
state = env.reset()
# 循环交互

while True:
    # 渲染画面
    # env.render()
    # 从动作空间随机获取一个动作
    action = env.action_space.sample()
    # agent与环境进行一步交互
    state, reward, terminated, truncated, info = env.step(action)
    print('state = {0}; reward = {1}'.format(state, reward))
    # 判断当前episode 是否完成
    if terminated:
        print('terminated')
        break
    time.sleep(0.1)

# 环境结束
# env.close()

gym-retro game folder

Run the program script to locate the gym-retro game folder location
windows CMD:
cd … && python .\utils\print_game_lib_folder.py
in Ubuntu:
python ./utils/print_game_lib_folder.py
/home/testlinux/anaconda3/envs/StreetFighterAI/lib/python3 .8/site-packages/retro/data/stable/StreetFighterIISpecialChampionEdition-Genesis

After the console outputs the folder path, copy it to the file explorer and jump to the corresponding path. This folder is the game data folder of "Street Fighter II: Champion Special Edition" under gym-retro, which contains the game ROM files and data configuration files. Copy the four files , , , under data/the folder in this project to this folder, and overwrite the original files (administrator permissions may be required). Among them, the files are the game archives of the last level of difficulty 4 of "Street Fighter II: Champion Special Edition", and the three files are gym-retro configuration files, which store the memory address of the game information (this project only uses the [agent_hp ] and [enemey_hp], used to read the life value of game characters in real time).Champion.Level12.RyuVsBison.statedata.jsonmetadata.jsonscenario.json.state.json

The game ROM file (which can be understood as the game program itself) of "Street Fighter II Special Champion Edition" is also required to run the program. gym-retro itself does not provide the ROM file of the game, you need to obtain it through legal means. You can refer to this link .

After obtaining the game ROM file through legal channels, copy it to the aforementioned gym-retro game data folder and rename it to rom.md. At this point, the environment configuration preparations are completed.

Note: If you want to record the battle video of the smart agent, you also need to install ffmpeg .

conda install ffmpeg

Error prompt and solution

Could not initialize NNPACK!

Detailed error content: [W NNPACK.cpp:64] Could not initialize NNPACK! Reason: Unsupported hardware.
Reference: Link to the original text
This error is usually caused by your computer not supporting the hardware instruction set required by the NNPACK library. NNPACK is an efficient computer vision library for optimizing the computation of neural networks.
If your computer does not support the instruction set required by NNPACK, you can try other computer vision libraries such as OpenCV or PyTorch.
If you want to continue using NNPACK, please make sure your computer meets NNPACK's requirements. NNPACK requires the computer to support the AVX2 instruction set and the FMA instruction. You can check if your CPU supports these instruction sets to see if NNPACK is available.
There is no need to solve it, and it will not affect the operation of the project, but the calculation speed is limited, especially under the condition of virtual machine, it is difficult to meet this condition.

Error message: libGL error: MESA-LOADER: failed to open swrast

The gcc installed by default in the ubuntu environment has compatibility problems in the conda23.3.1 environment. In the Python environment, it needs to be updated to the gcc compatible with the conda platform. Use the command to update:

    conda install -c conda-forge gcc

run test

After the environment configuration is complete, you main/can test.pyrun the test under the folder to experience the performance of the intelligent agent in different training stages.

cd G:\bsp\street-fighter-ai\main
# cd ~/street-fighter-ai/main
# cd [项目上级文件夹]/street-fighter-ai/main
python test.py

Model weight files are main/trained_models/stored under the folder. Among them ppo_ryu_2500000_steps_updated.zipis test.pythe model file used by default. This model has good generalization and has the ability to pass the last level of "Street Fighter II: Champion Special Edition". If you want to watch the performance of other models, you test.pycan model_pathmodify the variable in to the path of other model files. The observations about the actual performance of the model at each training stage are described as follows:

• ppo_ryu_2000000_steps_updated: Over-fitting phenomenon appeared at the beginning, with generalization ability but not very strong.
• ppo_ryu_2500000_steps_updated: Close to the final overfitting state, unable to fully dominate in the first round of the last level, but has a certain generalization ability. There is a higher chance of winning in the last three rounds of the level.
• ppo_ryu_3000000_steps_updated: Close to the final overfitting state, it can almost dominate the first round of the last level, the winning rate is close to 100%, but the generalization ability is weak.
• ppo_ryu_7000000_steps_updated: Overfitting, completely dominant in the first round of the last level, with a winning rate of 100%, but poor generalization ability.
  

training model

If you want to train your own model, you can run it under main/the folder train.py.

cd [项目上级文件夹]/street-fighter-ai/main
python train.py

view curve

The project contains the Tensorboard graph of the training process, and you can use Tensorboard to view the detailed data. It is recommended to use the Tensorboard plug-in integrated with VSCode to view directly (I love you VSCode!). The traditional viewing methods are listed below:

cd [项目上级文件夹]/street-fighter-ai/main
tensorboard --logdir=logs/

Open the default URL of the Tensorboard service in the browser http://localhost:6006/ to view the interactive graph of the training process.

Tips: Replace the system source

sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak #Backup
sudo gedit /etc/apt/sources.list #Modify

# 阿里云源
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse


# 清华大学源
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse

Perform an update: sudo apt-get update

Tips: Replace the pip source

main command

mkdir ~/.pip
create conf file
sudo gedit ~/.pip/pip.conf
write content:

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
index-index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host =
    pypi.tuna.tsinghua.edu.cn
    mirrors.aliyun.com

List of commonly used domestic sources

Aliyun [http://mirrors.aliyun.com/pypi/simple/]

University of Science and Technology of China [https://pypi.mirrors.ustc.edu.cn/simple/]

Douban [http://pypi.douban.com/simple/]

Tsinghua University [https://pypi.tuna.tsinghua.edu.cn/simple/]

University of Science and Technology of China [http://pypi.mirrors.ustc.edu.cn/simple/]

Huazhong University of Science and Technology: [http://pypi.hustunique.com/]

Shandong University of Technology: [http://pypi.sdutlinux.org/]

Change pip source under windows

1. Create a folder

    win+R 打开用户目录%HOMEPATH%，在此目录下创建 pip 文件夹，在 pip 目录下创建 pip.ini 文件

2. Copy the following content

[global]
timeout = 6000
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
trusted-host = pypi.tuna.tsinghua.edu.cn

3. Refer to the original link: https://blog.csdn.net/weixin_39715012/article/details/120635192

jupyter notebook

1.2 Install jupyter notebook
Execute four commands in turn, the third command is to set the password
conda install jupyter notebook
jupyter notebook --generate-config
jupyter notebook password
cd ~/.jupyter/
password:123
[NotebookPasswordApp] Wrote hashed password to /home/testlinux /.jupyter/jupyter_notebook_config.json
edit jupyter_notebook_config.py
gedit jupyter_notebook_config.py
conda install ipykernel
python -m ipykernel install --user --name StreetFighterAI --display-name StreetFighterAI

Anaconda operation guide

Supplementary information: build jupyter notebook in ubuntu server, and install numpy, scipy, matplotlibm, pandas, sklearn///tensorflow

Ubuntu installation Anaconda detailed steps (Ubuntu21.10, Anaconda3)
ubuntu Anaconda installation, mirror source change and python virtual environment configuration details
Getting started with Anaconda

1. Create an environment

conda create -n <env_name>

Create an environment named py36 based on python3.6
conda create --name py36 python=3.6

2. Activate the environment: conda activate <env_name>
3. Exit the environment: conda deactivate <env_name>
4. Check the installed environment information: conda env list
5. Copy the environment: conda create -n <new_env_name> --clone <origin_env_name>

Duplicate and delete environments

Create a copy called py36_bak by cloning py36:
conda create -n py36_bak --clone py36
1
2
6. Delete the environment: conda env remove -n <env_name>
7. Save the environment information to the environment.yaml file: conda env export > environment.yaml
8. Create a file through the environment.yaml environment file: conda env create -f environment.yaml
9. View installed packages: conda list
10. Search packages: conda search <package_name1>
11. Install packages: conda install <package_name1> <package_name2>
12. Uninstall package: conda remove <package_name>

Anaconda uninstall:

1. Delete the Anaconda3 folder:

rm -rf ~/anaconda3

2. Delete related hidden files:

rm -rf ~/.condarc ~/.conda ~/.continuous

3. Delete anaconda in the environment variable:
open ~/.bashrc (for example: vim ~/.bashrc), find the ones related to conda, and comment them out:

4. Update environment variables:

source ~/.bashrc