When using Pytorch to build a new algorithm, it usually includes the following steps:
Register data set: CustomDataset is a re-encapsulation of MMDetection based on the original Dataset. Its __getitem__() method will redirect to the prepare_train_img() and prepare_test_img() functions according to the training and test modes respectively. When users build their own data sets by inheriting the CustomDataset class, they need to rewrite the load_annotations() and get_ann_info() functions to define the loading and traversal methods of data and labels. After completing the definition of the data set class, you also need to use DATASETS.register_module() to register the module.
Register the model: The method of model construction is similar to Pytorch, which is to create a subclass of Module and then rewrite the forward() function. The only difference is that MMDetection needs to inherit BaseModule instead of Module. BaseModule is a subclass of Module, and any model in MMLab must inherit this class. In addition, MMDetection splits a complete model into three parts: backbone, neck and head for management, so users need to disassemble the algorithm model into three classes in this way, using BACKBONES.register_module() and NECKS.register_module respectively. () and HEADS.register_module() to complete module registration.
Build configuration file: The configuration file is used to configure the operating parameters of each component of the algorithm. It can generally contain four parts: datasets, models, schedules and runtime. After completing the definition and registration of the corresponding module, configure the corresponding operating parameters in the configuration file, and then MMDetection will read and parse the configuration file through the Registry class to complete the instantiation of the module. In addition, the configuration file can implement the inheritance function through the _base_ field to improve code reuse.
Training and verification: After completing the code implementation of each module, registration of the module, and writing of the configuration file, you can use ./tools/train.py and ./tools/test.py to train and verify the model without the need for users Write additional code.
Training process
According to the data flow process, the training process can be simply summarized as:
1. Given any data set, you first need to build a Dataset class for iteratively outputting data.
2. When iteratively outputting data, it is necessary to perform various processing on the data through the data Pipeline. The most typical processing flow is data enhancement operations in training, data preprocessing in testing, etc.
3. The data sequence output by the Dataset can be controlled through the Sampler sampler. The most commonly used is the random sampler RandomSampler. Since the images output in the Dataset are of different sizes, in order to minimize the number of pixels in the pad when forming a subsequent batch, MMDetection introduces the group samplers GroupSampler and DistributedGroupSampler, which is equivalent to adding an additional feature based on the aspect ratio of the image on the basis of RandomSampler. group function
4. Input both the Sampler and the Dataset to the DataLoader, and then output the data composed of the batch through the DataLoader as the input of the Model.
5. For any Model, in order to facilitate the processing of data flow and distributed requirements, MMDetection introduces the upper-layer encapsulation of two Models: the stand-alone version MMDataParallel, and the distributed (single machine with multiple cards or multiple machines with multiple cards) version MMDistributedDataParallel
After the Model is run, loss and other information will be output, which will be saved or visualized through the logger.
6. In order to better decouple, easily obtain dependencies between various components and flexibly expand, MMDetection introduces the Runner class for full life cycle management, and can easily obtain, modify and intercept any life cycle data flow through Hook, and the expansion is very Convenient
Config class file
MMDetection uses the Config class in the MMCV library to complete the parsing of the configuration file. The Config class is used to operate configuration files. It supports loading configurations from multiple file formats, including python, json and yaml. It provides a dictionary object-like interface to get and set values.
Config implements the configuration file to the model, which requires two steps:
Import the .py configuration file into dict, 2) Construct the class through dict
Read configuration file
Generally, Config.fromfile(filename) is used to read the configuration file (you can also directly pass in a dict) and return a Config class:
from mmcv import Config
cfg = Config.fromfile('../configs/test_config.py')
The source code of the fromfile() function is as follows, and its core function is _file2dict(). _file2dict() will parse the configuration file in the key = value format according to text order and obtain a dictionary named cfg_dict. If the _base_ field exists, _file2dict will be called again for each file path included in _base_ () function adds the configuration parameters contained in the file to cfg_dict to realize the inheritance function of the configuration file. It should be noted that _file2dict() will internally verify the key values contained in different files in _base_. Duplicate key values are not allowed in different base files, otherwise MMCV does not know which base file shall prevail.
@staticmethod
def fromfile(filename,
use_predefined_variables=True,
import_custom_modules=True):
cfg_dict, cfg_text = Config._file2dict(filename,
use_predefined_variables)
# import_modules_from_strings()是根据字符串列表对应的模块
if import_custom_modules and cfg_dict.get('custom_imports', None):
import_modules_from_strings(**cfg_dict['custom_imports'])
return Config(cfg_dict, cfg_text=cfg_text, filename=filename)
There are two other points that need to be added. One is that when constructing the Config object, the python dict data type will be converted into the ConfigDict type for processing. ConfigDict is a subclass of Dict in the third-party library addict, because Python's native dict type does not support the attribute access method, especially when there are multiple layers of dicts nested inside the dict. If you follow the key access method, the code will be very difficult to write. Inefficient, and the Dict class implements attribute access by overriding __getattr__(). Therefore, ConfigDict, which inherits Dict, also supports accessing each member value in the dictionary using . attributes.
from mmcv import ConfigDict
model = ConfigDict(dict(backbone=dict(type='ResNet', depth=50)))
print(model.backbone.type) # 输出 'ResNet'
Modify configuration parameters
After knowing the internal logic of MMCV parsing configuration files, it is naturally clear how to modify the values of configuration parameters. Because _file2dict() builds a dictionary based on text order, the key value written later can overwrite the original value (if the variable type is a list, the entire list will be replaced, and a certain item cannot be modified). Taking the modified optimizer as an example, the original inherited optimizer is SGD with a learning rate of 0.02:
# 原来继承的优化器
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
Now if you want to adjust the learning rate inherited from _base_ to 0.001, you can directly add a line to the current configuration file:
# 修改学习率
optimizer = dict(lr=0.001)This will only modify the lr parameter in the optimizer key value , and other parameters will not be affected. The current optimizer configuration becomes:
# 修改学习率后的SGD
optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0001)If you want to change to a new optimizer now, but the parameters of the two optimizers are incompatible, you need to delete the original key values and replace them with a new set of key values. This can be achieved by configuring _delete_ = True :
# 将原来的SGD替换成AdamW
optimizer = dict(_delete_=True, type='AdamW', lr=0.0001, weight_decay=0.0001)Then the replacement of the optimizer is completed. The optimizer parameters of the current configuration file are as follows:
# 当前优化器变为AdamW
optimizer = dict(type='AdamW', lr=0.0001, weight_decay=0.0001)View configuration file
After calling Config.fromfile() , a Config object will be returned, which in addition to the _cfg_dict of the ConfigDict type obtained by parsing the configuration file mentioned above , also contains the two member variables text and pretty_text .
text stores the original text information in each configuration file (including files inherited from _base_), and identifies the path to the configuration file.
pretty_text is the formatted text of the _cfg_dict dictionary content. MMCV internally uses Google's YAPF library to format the dictionary object so that its output conforms to people's reading habits. You can directly print (cfg.pretty_text) to view the complete configuration file information. , the effect is the same as mmdetection/tools/misc/print_config.py of MMDetection.
In addition, pretty_text can be stored in a file through cfg.dump(filepath) for easy viewing.
Registry class file ( MMCV core component analysis (5): Registry - Zhihu (zhihu.com) )
Registry functions and usage
In OpenMMLab, the Registry class can provide a completely similar external decoration function to manage the construction of different components, such as backbones, heads, necks, etc. Internally, the Registry class actually maintains a global key-value pair. Through the Registry class, users can instantiate any desired module through strings.
To put it simply, the registration mechanism is to maintain several query tables. The key is the name of the module and the value is the handle of the module. Each query table manages a batch of different modules with similar functions. Every time we create a new module, we must save the corresponding key-value query pair into the corresponding query table according to the functions implemented by the module. This saving process is called "registration". When we want to call a module, we only need to find the corresponding module handle from the query table based on the module name, and then we can complete operations such as module initialization or method invocation. MMCV implements the mapping from string (key) to class (value) through the Registry class.
The registration of the module is implemented through the member function register_module() of the Registry. Register_module() will call another private function _register_module() internally. The core function of module registration is actually implemented in _register_module(). The core code is also very simple, which is to save the incoming module_name and module_class into the dictionary self._module_dict.
#class Register中主要使用的函数，用于注册
def register_module(self, name=None, force=False, module=None):
"""Register a module.
A record will be added to `self._module_dict`, whose key is the class
name or the specified name, and value is the class itself.
It can be used as a decorator or a normal function.
Example:
>>> backbones = Registry('backbone')
>>> @backbones.register_module()
>>> class ResNet:
>>> pass
>>> backbones = Registry('backbone')
>>> @backbones.register_module(name='mnet')
>>> class MobileNet:
>>> pass
>>> backbones = Registry('backbone')
>>> class ResNet:
>>> pass
>>> backbones.register_module(ResNet)
Args:
name (str | None): The module name to be registered. If not
specified, the class name will be used.
force (bool, optional): Whether to override an existing class with
the same name. Default: False.
module (type): Module class or function to be registered.
"""
if not isinstance(force, bool):
raise TypeError(f'force must be a boolean, but got {type(force)}')
# NOTE: This is a walkaround to be compatible with the old api,
# while it may introduce unexpected bugs.
if isinstance(name, type):
return self.deprecated_register_module(name, force=force)
# raise the error ahead of time
if not (name is None or isinstance(name, str) or is_seq_of(name, str)):
raise TypeError(
'name must be either of None, an instance of str or a sequence'
f' of str, but got {type(name)}')
# use it as a normal method: x.register_module(module=SomeClass)
if module is not None:
self._register_module(module=module, module_name=name, force=force)
return module
# use it as a decorator: @x.register_module()
def _register(module):
self._register_module(module=module, module_name=name, force=force)
return module
return _register
def _register_module(self, module_class, module_name=None, force=False):
if not inspect.isclass(module_class):
raise TypeError('module must be a class, '
f'but got {type(module_class)}')
if module_name is None:
module_name = module_class.__name__
if not force and module_name in self._module_dict:
raise KeyError(f'{module_name} is already registered '
f'in {self.name}')
self._module_dict[module_name] = module_class
It is worth noting that the core of the above function implementation is the use of decorators in Python .
After we obtain the handle of a module through a string, we can instantiate the module through the self.build_func function handle. build_func can be specified manually or inherited from the parent class. Generally speaking, the build_from_cfg() function is used by default, that is, the configuration parameter cfg is used to initialize the module. The configuration parameter cfg is a dictionary, the type field inside is the string of the module name, and the other fields correspond to the input parameters of the module constructor.
def build_from_cfg(cfg, registry, default_args=None):
args = cfg.copy()
# 将cfg以外的外部传入参数也合并到args中
if default_args is not None:
for name, value in default_args.items():
args.setdefault(name, value)
# 获取模块名称
obj_type = args.pop('type')
if isinstance(obj_type, str):
# get函数返回registry._module_dict中obj_type对应的模块句柄
obj_cls = registry.get(obj_type)
if obj_cls is None:
raise KeyError(f'{obj_type} is not in the {registry.name} registry')
elif inspect.isclass(obj_type):
# type值是模块本身
obj_cls = obj_type
else:
raise TypeError(f'type must be a str or valid type, but got {type(obj_type)}')
# 模块初始化, 返回模块实例
try:
return obj_cls(**args)
except Exception as e:
raise type(e)(f'{obj_cls.__name__}: {e}')
Considering that the registry parameter needs to point to the current register itself, we generally call the build() method of the Registry class instead of self.build_func .
def build(self, *args, **kwargs):
return self.build_func(*args, **kwargs, registry=self)
The following is a small example that simulates the registration and calling process of the network model. Note that when we print the Registry object, we actually print the values in self._module_dict .
# 实例化一个注册器用来管理模型
MODELS = Registry('myModels')
# 方式1: 在类的创建过程中, 使用函数装饰器进行注册(推荐)
@MODELS.register_module()
class ResNet(object):
def __init__(self, depth):
self.depth = depth
print('Initialize ResNet{}'.format(depth))
# 方式2: 完成类的创建后, 再显式调用register_module进行注册(不推荐)
class FPN(object):
def __init__(self, in_channel):
self.in_channel= in_channel
print('Initialize FPN{}'.format(in_channel))
MODELS.register_module(name='FPN', module=FPN)
print(MODELS)
""" 打印结果为:
Registry(name=myModels, items={'ResNet': <class '__main__.ResNet'>, 'FPN': <class '__main__.FPN'>})
"""
# 配置参数, 一般cfg从配置文件中获取
backbone_cfg = dict(type='ResNet', depth=101)
neck_cfg = dict(type='FPN', in_channel=256)
# 实例化模型(将配置参数传给模型的构造函数), 得到实例化对象
my_backbone = MODELS.build(backbone_cfg)
my_neck = MODELS.build(neck_cfg)
print(my_backbone, my_neck)
""" 打印结果为:
Initialize ResNet101
Initialize FPN256
<__main__.ResNet object at 0x000001E68E99E198> <__main__.FPN object at 0x000001E695044B38>
"""Runner and Hook mechanism
Runner, also known as executor, is responsible for scheduling the model training process. Its main purpose is to allow users to start training with less code and in a flexible and configurable way. In other words, MMCV encapsulates the entire training process and uses Runner for management and configuration. Although a high degree of encapsulation reduces the amount of code, how to make customized modifications to the internal process (such as dynamically adjusting the learning rate, etc.)? At this time, you need to use the Hook mechanism.
Hook is a general term for a technology that can change the execution flow of a program. In layman's terms, Hook can be understood as a trigger that executes a predefined function at a predefined location in the program. MMCV has reserved interface functions (called callback functions) in several commonly used locations, as shown in the figure below. MMCV has implemented some commonly used Hook functions, and users can also add their own Hook functions, which is very convenient. When the program executes to the specified position, it will enter the callback function, perform the corresponding function, and then continue to execute the main process after the execution is completed.
Corresponding to the training code:
# 开始运行时调用
before_run()
while self.epoch < self._max_epochs:
# 开始 epoch 迭代前调用
before_train_epoch()
for i, data_batch in enumerate(self.data_loader):
# 开始 iter 迭代前调用
before_train_iter()
self.model.train_step()
# 经过一次迭代后调用
after_train_iter()
# 经过一个 epoch 迭代后调用
after_train_epoch()
# 运行完成前调用
after_run()
Runner encapsulates the training and verification process of each framework under the OpenMMLab system, and is responsible for managing the entire life cycle of the training/verification process; through predefined callback functions, users can insert customized Hooks to achieve various customized needs.
Runner class
Runner is divided into EpochBasedRunner and IterBasedRunner. As the name suggests, the former manages the process in an epoch manner, and the latter manages the process in an iter manner. They are both subclasses of BaseRunner. Any subclass of BaseRunner needs to implement the four methods run(), train(), val() and save_checkpoint(), which are also the core methods of Runner. Here, EpochBasedRunner is used as an example to analyze the above four functions. In order to make the code structure look clearer, the code that has nothing to do with the core functions has been deleted.
Constructor
EpochBasedRunner and IterBasedRunner are both subclasses of BaseRunner and inherit the constructor of BaseRunner. By default, runner calls train_step() and val_step() in the model class for training and verification. If batch_processor is specified, batch_processor will be called to process the data in data_loader.
class BaseRunner(metaclass=ABCMeta):
def __init__(self,
model, # [torch.nn.Module] 要运行的模型
batch_processor=None, # 过时用法, 通过实现模型中的train_step()和val_step()方法替代
optimizer=None, # [torch.optim.Optimizer] 优化器, 可以是一个也可以是一组通过dict配置的优化器
work_dir=None, # [str] 保存检查点和Log的目录
logger=None, # [logging.Logger] 训练中使用的日志记录器
meta=None, # [dict] 一些信息, 这些信息会在logger hook中记录
max_iters=None, # [int] 训练epoch数
max_epochs=None): # [int] 训练迭代次数run() function
run()是runner类的主调函数，会根据workflow指定的工作流，对data_loaders中的数据进行处理。目前MMCV支持训练和验证两种工作流，对于EpochBasedRunner而言，workflow配置为[('train', 2)，('val', 1)]表示先训练2个epoch，然后验证一个epoch；[('train', 1)]表示只进行训练，不进行验证。如果是IterBasedRunner，[('train', 2)，('val', 1)]则表示先训练2个iter，然后验证一个iter。
def run(self, data_loaders, workflow, max_epochs=None, **kwargs):
while self.epoch < self._max_epochs:
for i, flow in enumerate(workflow):
mode, epochs = flow
# 根据工作流确定当前是运行train()还是val(), getattr返回对应的函数句柄
epoch_runner = getattr(self, mode)
for _ in range(epochs):
if mode == 'train' and self.epoch >= self._max_epochs:
break
# 运行train()或val()
epoch_runner(data_loaders[i], **kwargs)train()和val()函数
train()和val()函数循环调用run_iter()完成一个epoch流程。函数开头的self.model.train()和self.model.eval()实际上调用的是torch.nn.module.Module的成员函数，将当前模块设置为训练模式或验证模式，两种不同模式下batchnorm、dropout等层的操作会有区别。然后由于测试过程不需要梯度回传，所以val函数加了一个装饰器@torch.no_grad()。
def train(self, data_loader, **kwargs):
# 将模块设置为训练模式
self.model.train()
self.mode = 'train'
self.data_loader = data_loader
self._max_iters = self._max_epochs * len(self.data_loader)
for i, data_batch in enumerate(self.data_loader):
self.run_iter(data_batch, train_mode=True, **kwargs)
self._iter += 1
self._epoch += 1
@torch.no_grad()
def val(self, data_loader, **kwargs):
# 将模块设置为验证模式
self.model.eval()
self.mode = 'val'
self.data_loader = data_loader
for i, data_batch in enumerate(self.data_loader):
self.run_iter(data_batch, train_mode=False)train()和val()的核心函数是run_iter()，根据train_mode参数调用model.train_step()或model.val_step()，这两个函数最终会执行我们自己模型的forward()函数，返回loss值。
def run_iter(self, data_batch, train_mode, **kwargs):
if self.batch_processor is not None:
outputs = self.batch_processor(self.model, data_batch, train_mode=train_mode, **kwargs)
elif train_mode:
outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
else:
outputs = self.model.val_step(data_batch, self.optimizer, **kwargs)
self.outputs = outputssave_checkpoint()函数
save_checkpoint()函数调用torch.save将检查点以下列格式保存。
checkpoint = {
'meta': dict(), # 环境信息(比如epoch_num, iter_num)
'state_dict': dict(), # 模型的state_dict()
'optimizer': dict()) # 优化器的state_dict()
}Hook类
MMCV在./mmcv/runner/hooks/hook.py中定义了Hook的基类以及Hook的注册器HOOKS。作为基类，Hook本身没有实现具体的函数，只是提供了before_run、after_run等6个接口函数，其他所有的Hooks都通过继承Hook类并重写相应的函数完整指定功能。
from mmcv.utils import Registry
HOOKS = Registry('hook')
class Hook:
def before_run(self, runner):
pass
def after_run(self, runner):
pass
def before_epoch(self, runner):
pass
def after_epoch(self, runner):
pass
def before_iter(self, runner):
pass
def after_iter(self, runner):
passMMCV已经实现了部分常用的Hooks，如下图所示。默认Hook不需要用户自行注册，通过配置文件配置对应的参数即可；定制Hook则需要用户手动注册进去。
Hook也是一个模块，使用时需要定义、注册、调用3个步骤。
定义
MMCV实现的Hook都在./mmcv/runner/hooks目录下，这里以CheckpointHook为例介绍一下怎么新建一个Hook。
首先从hook.py中导入注册器HOOKS以及基类Hook。然后新建一个名为CheckpointHook类继承Hook基类，由于Hook基类没有定义构造函数，这里首先必须自己定义__init__函数，然后根据Hook需要实现的功能，重写Hook基类中的一种或几种方法。比如MMCV会在每次训练开始前打印checkpoint的保存路径，会在每次循环结束后或每个epoch执行完成后保存checkpoint，因此CheckpointHook类重写了before_run、after_train_iter和after_train_epoch这3个方法。
from .hook import HOOKS, Hook
@HOOKS.register_module()
class CheckpointHook(Hook):
def __init__(self,
interval=-1,
by_epoch=True,
save_optimizer=True,
out_dir=None,
max_keep_ckpts=-1,
save_last=True,
sync_buffer=False,
file_client_args=None,
**kwargs):
...
def before_run(self, runner):
...
def after_train_iter(self, runner):
...
def after_train_epoch(self, runner):
...注册
对于MMCV的默认Hook，在执行runner.run()前会调用BaseRunner类中的register_training_hooks方法进行注册：
def register_training_hooks(self,
lr_config,
optimizer_config=None,
checkpoint_config=None,
log_config=None,
momentum_config=None,
timer_config=dict(type='IterTimerHook'),
custom_hooks_config=None):
"""Register default and custom hooks for training.
Default and custom hooks include:
+----------------------+-------------------------+
| Hooks | Priority |
+======================+=========================+
| LrUpdaterHook | VERY_HIGH (10) |
+----------------------+-------------------------+
| MomentumUpdaterHook | HIGH (30) |
+----------------------+-------------------------+
| OptimizerStepperHook | ABOVE_NORMAL (40) |
+----------------------+-------------------------+
| CheckpointSaverHook | NORMAL (50) |
+----------------------+-------------------------+
| IterTimerHook | LOW (70) |
+----------------------+-------------------------+
| LoggerHook(s) | VERY_LOW (90) |
+----------------------+-------------------------+
| CustomHook(s) | defaults to NORMAL (50) |
+----------------------+-------------------------+
If custom hooks have same priority with default hooks, custom hooks
will be triggered after default hooks.
"""
self.register_lr_hook(lr_config)
self.register_momentum_hook(momentum_config)
self.register_optimizer_hook(optimizer_config)
self.register_checkpoint_hook(checkpoint_config)
self.register_timer_hook(timer_config)
self.register_logger_hooks(log_config)
self.register_custom_hooks(custom_hooks_config)具体到单个注册函数，比如register_checkpoint_hook()，hook作为一个模块，还是使用build_from_cfg进行实例获取，然后调用BaseRunner类的register_hook()进行注册，这样所有Hook实例就都被纳入到runner中的一个list中。
def register_checkpoint_hook(self, checkpoint_config):
hook = mmcv.build_from_cfg(checkpoint_config, HOOKS)
self.register_hook(hook, priority='NORMAL')
def register_hook(self, hook, priority='NORMAL'):
priority = get_priority(priority)
hook.priority = priority
# 按照priority大小插入当前hook列表
inserted = False
for i in range(len(self._hooks) - 1, -1, -1):
if priority >= self._hooks[i].priority:
self._hooks.insert(i + 1, hook)
inserted = True
break
if not inserted:
self._hooks.insert(0, hook)调用
在runner执行过程中，会在特定的程序位点通过call_hook()函数调用相应的Hook。
def train(self, data_loader, **kwargs):
self.model.train()
self.mode = 'train'
self.data_loader = data_loader
self._max_iters = self._max_epochs * len(self.data_loader)
self.call_hook('before_train_epoch')
time.sleep(2) # Prevent possible deadlock during epoch transition
for i, data_batch in enumerate(self.data_loader):
self._inner_iter = i
self.call_hook('before_train_iter')
self.run_iter(data_batch, train_mode=True, **kwargs)
self.call_hook('after_train_iter')
self._iter += 1
self.call_hook('after_train_epoch')
self._epoch += 1前面调用register_hook()注册Hook的时候，会根据优先级将Hook加入到self._hooks这个列表中，在执行call_hook()时候，使用for循环就可以很简单的实现按照优先级依次调用指定的Hook了。
def call_hook(self, fn_name):
for hook in self._hooks:
getattr(hook, fn_name)(self)reference
(72条消息) MMDetection框架入门教程（完全版）_Maples丶丶的博客-CSDN博客_mmdetection
轻松掌握 MMDetection 整体构建流程(二) - 知乎 (zhihu.com)
(82条消息) MMDetection框架入门教程（三）：配置文件详细解析_Maples丶丶的博客-CSDN博客_mmdetection _base_
(87条消息) MMDetection框架入门教程（五）：Runner和Hook详细解析_mmdetection的runner_Maples丶丶的博客-CSDN博客