Object detection, instance and panoptic segmentation using MMDetection

MMDetection is an open source toolbox for object detection based on PyTorch, which is part of the OpenMMLab project. Contains the following main features:

• Supports three tasks
  • Object Detection refers to the task of classifying and locating objects in pictures
  • Instance Segmentation (Instance Segmentation) refers to the task of classifying and segmenting image objects
  • Panoptic Segmentation (Panoptic Segmentation) is a detection task that unifies semantic segmentation (classifying each pixel of an image) and instance segmentation (detecting object instances and segmenting them)
• Modular design to flexibly support 6 datasets, 57 different algorithms and rich data augmentation, providing 450+ pre-trained models
• Support the deployment of dozens of algorithm models

Install (object detection)

Use the following command to quickly generate a virtual environment:

$ python -m venv venv
# Windows 下进入虚拟环境
$ venv/Scripts/activate

Check the matching version numbers of PyTorch + MMDetection + MMCV in MMDetection and MMCV version compatibility in advance. For example, the version requirements I see now are:

• PyTorch: 1.3+
• MMDetection: 2.28.1
• MMCV: >=1.3.17, <1.8.0

MMDetection is a detection framework based on PyTorchtorch , first install the library:

$ pip install torch
$ pip install opencv-python
$ pip install torchvision

MMDetection includes MMDetection and MMCV. The two are integrated. MMCV needs to be installed. Follow the instructions in Installing MMCV - Using mim to install using mim (the package management tool of the OpenMMLab project):

$ pip install openmim
$ mim install mmcv-full==1.7.1

Then install MMDetection, (the important thing is said twice) Check the matching version number of PyTorch + MMDetection + MMCV in MMDetection and MMCV version compatibility in advance , choose the appropriate version to install, it is recommended to use the mimquick installation directly:

$ mim install mmdet==2.28.1

Under the Windows system, if the following exceptions are encountered during the installation process:

$       error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
$       [end of output]
$ 
$   note: This error originates from a subprocess, and is likely not a problem with pip.
$   ERROR: Failed building wheel for pycocotools
$ Failed to build pycocotools
$ ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects

It can be solved according to the solution provided in "Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools" solution" .

After the installation is complete, execute the following Python code (demo.py) to verify that MMDetection and the required environment are installed correctly:

import mmcv
from mmdet.apis import init_detector, inference_detector

# 一、指定模型的配置文件和 checkpoint 文件路径
# 下载 https://github.com/open-mmlab/mmdetection 项目并解压
# 把 mmdetection-master/configs/_base_ 文件夹复制到当前项目 configs/ 目录下
# 把 mmdetection-master/configs/faster_rcnn 文件夹复制到当前项目 configs/ 目录下
config_file = 'configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
# 从 model zoo 下载 checkpoint 并放在 `checkpoints/faster_rcnn/` 文件下
# 网址为: http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
checkpoint_file = 'checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth'

# 二、根据配置文件和 checkpoint 文件构建模型
# 有 GPU 时使用 device = 'cuda:0'
device = 'cpu'
# 初始化检测器
model = init_detector(config_file, checkpoint_file, device=device)

# 三、测试单张图片并展示结果
# 图像地址: https://github.com/open-mmlab/mmdetection/blob/master/demo/demo.jpg
img = mmcv.imread('demo/demo.jpg')
# 推理演示图像, 结果是一个 `numpy.ndarray` 列表
result = inference_detector(model, img)
# 将结果可视化
model.show_result(img, result)
# 将可视化结果保存为图片
model.show_result(img, result, out_file='demo/demo_result.jpg')

The project directory structure at this time is as follows:

If MMDetection is successfully installed, the above code can run completely and generate the following demo/demo_result.jpgfiles successfully:

instance segmentation

Instance Segmentation refers to the task of classifying and segmenting image objects. Open MMDetection - Model Library/MMDetection and select Instance SegmentationTask Type:

Choose an algorithm with a high AP (average precision) value and a relatively new year. For example, the algorithm name you see now is SCNet (algorithm RF-Next requires a GPU to run), mmdetection-master/configs/find the corresponding scnetfolder and its dependent htcfolders in the previously downloaded directory, and copy them to the current project configs/directory. Click the algorithm name to enter the algorithm details page:

Similarly, select a model with a higher AP (average precision) value , first copy the text of the model name here , and then configs/scnet/metafile.ymlsearch for this text in the file:

After the search is completed, two configurations and parameters can be obtained:

• Config: the configuration file path of the model ( config_file = 'xxx.py')
• Weights: The download URL of the model, download the model file through this address ( checkpoint_file = 'xxx.pth')

After completing the above preparations, execute the following Python code (demo.py) to verify whether it is possible to classify images and segment objects:

import mmcv
from mmdet.apis import init_detector, inference_detector

print('指定模型的配置文件和checkpoint文件路径')
config_file = 'configs/scnet/scnet_r50_fpn_1x_coco.py'
checkpoint_file = 'checkpoints/scnet/scnet_r50_fpn_1x_coco-c3f09857.pth'
print('根据配置文件和checkpoint文件构建模型')
device = 'cpu'
model = init_detector(config_file, checkpoint_file, device=device)
print('测试单张图片并展示结果')
img = mmcv.imread('demo/demo.jpg')
result = inference_detector(model, img)
model.show_result(img, result)
model.show_result(img, result, out_file='demo/demo_result.jpg')

The project directory structure at this time is as follows:

If the instance segmentation task of MMDetection in the above code can run completely, the following demo/demo_result.jpgfiles will be generated smoothly:

panoramic segmentation

Panoptic Segmentation (Panoptic Segmentation) is a detection task that unifies semantic segmentation (classifying each pixel of an image) and instance segmentation (detecting object instances and segmenting them). As before, open MMDetection - Model Library/MMDetection and select Panoptic SegmentationTask Type:

There is currently only one algorithm here, so you can only choose the PanopticFPNmmdetection-master/configs/ algorithm, or find the corresponding panoptic_fpnfolder in the previously downloaded directory, and copy it to the current project configs/directory. Click the algorithm name to enter the algorithm details page:

Then select a model, copy the text of the model name , and configs/panoptic_fpn/metafile.ymlsearch in the file:

After the search is completed, you can get Config ( config_file = 'xxx.py') and Weights downloaded file path ( checkpoint_file = 'xxx.pth') configuration parameters. Then execute the following Python code (demo.py) to verify whether the image can be classified for each pixel while detecting object instances and performing segmentation tasks:

import mmcv
from mmdet.apis import init_detector, inference_detector

# 上面代码没有变，就下面两个变量的值改一下
config_file = 'configs/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco.py'
checkpoint_file = 'checkpoints/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco_20210821_101153-9668fd13.pth'
# 下面代码也没有变
device = 'cpu'
model = init_detector(config_file, checkpoint_file, device=device)
img = mmcv.imread('demo/demo.jpg')
result = inference_detector(model, img)
model.show_result(img, result)
model.show_result(img, result, out_file='demo/demo_result.jpg')

At this time, the directory structure of the project is as follows:

If the panorama segmentation task in the above code can run smoothly, the following demo/demo_result.jpgfiles will be generated:

After the practice is completed here, you will find that the calling code is actually the same, just change the configuration file and model file to achieve different functions!

Next, you can also read "Python calls MMDetection to realize AI matting to remove the background" to learn more about the practical application.