Use OpenMMLab to easily build a backbone network, and cover a variety of visual tasks in one go

The OpenMMLab framework covers almost all aspects of deep learning vision tasks. For each specific task, we provide a corresponding algorithm library, such as MMClassification for classification tasks, MMDetection for detection and MMSegmentation for segmentation tasks, and so on.

A question that follows is, if I want to apply the research method to multiple tasks and evaluate it on multiple tasks, should I fork each algorithm library? Further, if I want to open source my code that relies on the OpenMMLab framework, what should I do?

Here, taking our most common cross-task research - building a backbone network as an example, we will introduce how to quickly develop a new backbone network, and use mmcls, mmdet and mmseg to easily implement evaluation on classification, detection and segmentation tasks.

In order to make it easy for everyone to get started, we provide an example repository: github.com/mzr1996/bac…

In the example repository, we use ConvNeXt as an example to show the files needed to build the backbone network and benchmark it. So let's take this example warehouse as an example to learn step by step how to build your own backbone network.

Configuration Environment

First, we need to configure the Python and  PyTorch  environments. I believe that everyone who is familiar with PyTorch must have prepared these environments. It should be noted that we require Python >= 3.6, PyTorch >= 1.5.0, and newer versions are of course no problem.

In addition to this, we also need the following OpenMMLab repositories:

• MIM : a command line tool for managing OpenMMLab packages and experiments
• MMCV : OpenMMLab Computer Vision Basic Library
• MMClassification : OpenMMLab Image Classification Toolbox and Benchmarks. Besides classification tasks, it is also used to provide diverse backbone networks
• MMDetection : OpenMMLab detection toolbox and benchmarks
• MMSegmentation : OpenMMLab Semantic Segmentation Toolbox and Benchmarks

Assuming you have prepared the Python and PyTorch environments, you only need the following two lines to configure the software environment.

pip install openmim mmcls mmdet mmsegmentation 
mim install mmcv-full 
复制代码

data preparation

Suppose we want to benchmark the classification task on ImageNet, the semantic segmentation task on ADE20K, and the detection task on COCO, then the data files need to be organized in the following format.

data/ 
├── imagenet 
│   ├── train 
│   ├── val 
│   └── meta 
│       ├── train.txt 
│       └── val.txt 
├── ade 
│   └── ADEChallengeData2016 
│       ├── annotations 
│       └── images 
└── coco 
    ├── annotations 
    │   ├── instance_train2017.json 
    │   └── instance_val2017.json 
    ├── train2017 
    └── val2017 
复制代码

Here, we only list the necessary files for training and validation. If you want to benchmark on more datasets or tasks, such as panoptic segmentation using MMDetection, just organize the corresponding datasets according to the needs of MMDetection. Additionally, MMSegmentation provides a tutorial on how to organize datasets related to semantic segmentation.

Implement your backbone

The preparations are all ready, let's start to implement and train our backbone network step by step~

The first is to implement:

1.  models Create your backbone network file under the folder. For example in the example repository  models/convnext.py. In this file, you only need to use PyTorch to implement your backbone network. There are only two additional points to note:

• Backbone and main modules need to be inherited  mmcv.runner.BaseModule. This  BaseModule is  torch.nn.Module a subclass that behaves almost identically, except that it additionally supports  init_cfg specifying initialization methods including pretrained models with parameters.
• Register your backbone class with the registrar using the following decorator structure  mmcls.models.BACKBONES .

from mmcv.runner import BaseModule 
from mmcls.models import BACKBONES 
 
@BACKBONES.register_module(force=True) 
class MyBackbone(BaseModule): 
    ... 
复制代码

注册器是什么？看看这个！

2.[可选] 如果你希望为某些特定的任务增加一些额外的模块、组件，可以参照 models/det/layer_decay_optimizer_constructor.py 添加至对应的文件夹中。

3.将你添加的主干网络、自定义组件添加至 models/__init__.py 文件中。

添加配置文件

如果你还不太清楚配置文件是怎么回事，这篇教程应该可以帮到你。

通常来说，我们使用若干基础配置文件，通过继承的方式来组织一个实验的配置文件。这些基础配置文件一般包括模型、数据集、优化策略和运行配置。你也可以在配置文件中对基础配置文件进行覆盖，或者不使用基础配置文件，全部写到一个配置文件里。

在本示例库中，我们提供了一套比较常用的基础配置文件，你可以在从以下位置访问链接找到更多有用的基础配置文件：MMClassification，MMDetection， MMSegmentation。

对每个不同的任务，对应的配置文件可以放在放在 configs 目录中不同的子文件夹。需要注意的是，在配置文件的 model 部分，为了能够使 MMDetection 和 MMSegmentation 能够调用注册在 mmcls.models.BACKBONES 的主干网络，我们需要在 det 和 seg 的配置文件中额外指定主干网络注册器的位置，这只需要在 type 中添加 mmcls.前缀即可，如：

model = dict( 
    backbone=dict(type='mmcls.MyBackbone', ...), 
    ... 
) 
复制代码

想要了解跨仓库调用模块的更多内容，可以参见《MMDet居然能用MMCls的Backbone？论配置文件的打开方式》

训练和测试

Relying on MIM , a unified experiment management tool provided by OpenMMLab  , after implementing the model and writing the configuration file, we can train and test different tasks without writing any Python scripts.

First, we need to add the directory of the current warehouse to the  PYTHONPATH environment variable, so that Python can find our model file, run the following command in the root directory of the warehouse:

export PYTHONPATH=`pwd`:$PYTHONPATH  
复制代码

Single machine single GPU

# 训练分类模型 
mim train mmcls $CONFIG --work-dir $WORK_DIR 
# 测试分类模型 
mim test mmcls $CONFIG -C $CHECKPOINT --metrics accuracy --metric-options "topk=(1, 5)" 
# 训练目标检测/实例分割模型 
mim train mmdet $CONFIG --work-dir $WORK_DIR 
# 测试目标检测/实例分割模型 
mim test mmdet $CONFIG -C $CHECKPOINT --eval bbox segm 
# 训练语义分割模型 
mim train mmseg $CONFIG --work-dir $WORK_DIR 
# 测试语义分割模型 
mim test mmseg $CONFIG -C $CHECKPOINT --eval mIoU 
复制代码

Description of some parameters:

• $CONFIG: configs/ Configuration files in the folder.
• $WORK_DIR: Folder for saving log and weight files
• $CHECKPOINT: weights file path

Single machine with multiple GPUs (take 4 GPUs as an example)

# 训练分类模型 
mim train mmcls $CONFIG --work-dir $WORK_DIR --launcher pytorch --gpus 4 
# 测试分类模型 
mim test mmcls $CONFIG -C $CHECKPOINT --metrics accuracy --metric-options "topk=(1, 5)" --launcher pytorch --gpus 4 
# 训练目标检测/实例分割模型 
mim train mmdet $CONFIG --work-dir $WORK_DIR --launcher pytorch --gpus 4 
# 测试目标检测/实例分割模型 
mim test mmdet $CONFIG -C $CHECKPOINT --eval bbox segm --launcher pytorch --gpus 4 
# 训练语义分割模型 
mim train mmseg $CONFIG --work-dir $WORK_DIR --launcher pytorch --gpus 4  
# 测试语义分割模型 
mim test mmseg $CONFIG -C $CHECKPOINT --eval mIoU --launcher pytorch --gpus 4 
 
复制代码

Epilogue

OpenMMLab  has been committed to providing everyone with a toolbox for various tasks of visual deep learning. At the same time, the algorithm libraries for various tasks are organically unified. This article introduces a practical way of using multiple algorithm libraries in combination. You are welcome to download and try our sample warehouse . In addition, we will  continue to update various examples of using  MIM to complete various tasks in github.com/open-mmlab/…  . Welcome to pay attention~