1. Faster RCNN
Girshick, Ross, et al. “Rich feature hierarchies for accurate object detection and semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
Girshick, Ross. “Fast r-cnn.” Proceedings of the IEEE international conference on computer vision. 2015.
Ren, Shaoqing, et al. “Faster r-cnn: Towards real-time object detection with region proposal networks.” Advances in neural information processing systems 28 (2015).
Faster RCNN是一种快速的目标检测模型，在RCNN基础上提出了RPN(Region Proposal Network)来取代SS(Selective Search)算法来生成候选锚框，从而大大加快了目标检测的速度。而在学习Faster RCNN之前必须要搞明白RCNN和Fast RCNN的相关概念和网络框架。
1.1 RCNN
如图所示，RCNN的实现步骤分为四步：
- 输入原始图片
- 在原始图片上使用SS算法选取2000个左右的锚框
- 将生成的锚框输入到CNN中进行特征提取
- 经过SVM分类器来区分每个锚框的类别
1.2 Fast RCNN
从RCNN的框架中，我们可以很容易地发现在步骤2，3中需要对2000个锚框进行特征提取，这是非常低效的，因此，Fast RCNN直接对原始图像进行特征提取，然后在生成的特征图上进行锚框生成，这样避免了大量的CNN的计算，大大提高了效率。
如图所示，可以分为5个步骤：
- 输入原始图像
- 通过CNN提取原始图像的特征图
- 在特征图上生成锚框
- 通过ROI Pooling层统一锚框的尺寸
- 用两个并行的全连接层分别进行分类任务和锚框定位任务
在这里我们还需要知道Fast RCNN是一个多任务的模型，即分类任务和边界框回归任务。假设真实的类别为 u u u，真实的边界框回归参数为 v v v,那么损失函数可以写为： L ( p , u , t u , v ) = L c l s ( p , u ) + λ [ u ≥ 1 ] L l o c ( t u , v ) L(p,u,t^u,v)=L_{cls}(p, u) + \lambda[u \ge 1]L_{loc}(t^u,v) L(p,u,tu,v)=Lcls(p,u)+λ[u≥1]Lloc(tu,v), 其中 L c l s ( p , u ) = − l o g ( p u ) L_{cls}(p, u)=-log(p_u) Lcls(p,u)=−log(pu)是多分类交叉熵损失。
而边界框回归参数 v = ( v x , v y , v w , v h ) v=(v_x, v_y,v_w,v_h) v=(vx,vy,vw,vh), 预测的边界框回归参数为 t u = ( t x u , t y u , t w u , t h u ) t^u=(t_x^u, t_y^u, t_w^u, t_h^u) tu=(txu,tyu,twu,thu)。 [ u ≥ 1 ] [u \ge 1] [u≥1] 代表的是如果锚框中框选中的是物体那么这一项就为1，否则就为0，因为锚框框选到的不一定是物体，而可能是背景，背景的类别设置为0。
在这里我们还需要弄明白边界框回归参数到底是个什么，模型最终输出的参数并不是预测锚框的位置，而是一组预测的偏移量，通过偏移量我们可以调整预测锚框的位置。
假设此时预测锚框的坐标为 ( P x , P y , P w , P h ) (P_x, P_y, P_w, P_h) (Px,Py,Pw,Ph), 通过预测偏移量 ( t x u , t y u , t w u , t h u ) (t_x^u, t_y^u, t_w^u, t_h^u) (txu,tyu,twu,thu)可以对其位置进行调整：
{ G x = P w ∗ t x u + P x G y = P h ∗ t y u + P y G w = P w ∗ e t w u G h = P h ∗ e t h u \left\{ \begin{aligned} G_x &= P_w * t_x^u + P_x\\ G_y &= P_h * t_y^u + P_y \\ G_w &= P_w * e^{t_w^u}\\ G_h &= P_h * e^{t_h^u}\\ \end{aligned} \right. ⎩⎪⎪⎪⎪⎨⎪⎪⎪⎪⎧GxGyGwGh=Pw∗txu+Px=Ph∗tyu+Py=Pw∗etwu=Ph∗ethu
对于边界框回归，这里采用的是 S m o o t h − L 1 Smooth-L1 Smooth−L1损失函数：
s m o o t h L 1 ( x ) = { 0.5 x 2 , i f ∣ x ∣ < 1 ∣ x ∣ − 0.5 , o t h e r w i s e smooth_{L_1}(x)=\left\{ \begin{aligned}& 0.5x^2, \quad \quad if |x| < 1\\ &|x|-0.5 , \quad \quad otherwise\\ \end{aligned} \right. smoothL1(x)={
0.5x2,if∣x∣<1∣x∣−0.5,otherwise
1.3 Faster RCNN
前两个工作的anchor是基于SS算法生成的，但是Faster RCNN提出了Region Proposal Network(RPN)来帮助网络决定anchors的生成。
主要可以分为三个步骤：
- 将图像输入到BackBone中生成相应的特征图，这里的特征图可以使用FPN来生成多个尺度的特征图。
- 使用RPN来生成候选框，将候选框投影至特征图上，生成相应的特征矩阵
- 每个特征矩阵通过ROI pooling层缩放到统一大小(7x7)的特征图，通过展平输入到全连接层进行相关预测
2. 代码实现
2.1 数据准备
以Pascal Voc数据集为例，目录如下所示：
其中Annotations中都是XML格式的文件，是一张图片的标注信息，包括ground truth anchors的坐标和类别信息，图片的高宽等。需要用lxml来对它进行解析，保存为字典格式。
import numpy as np
from torch.utils.data import Dataset
import os
import torch
import json
from PIL import Image
from lxml import etree
class VOCDataSet(Dataset):
"""读取解析PASCAL VOC2007/2012数据集"""
def __init__(self, voc_root, year="2012", transforms=None, txt_name: str = "train.txt"):
assert year in ["2007", "2012"], "year must be in ['2007', '2012']"
# 增加容错能力
if "VOCdevkit" in voc_root:
self.root = os.path.join(voc_root, f"VOC{
year}") # 获取数据集的根目录
else:
self.root = os.path.join(voc_root, "VOCdevkit", f"VOC{
year}")
self.img_root = os.path.join(self.root, "JPEGImages") # 图片文件存储目录
self.annotations_root = os.path.join(self.root, "Annotations") # 标注文件存储目录
# read train.txt or val.txt file
txt_path = os.path.join(self.root, "ImageSets", "Main", txt_name) # 训练数据相关信息保存在一个txt文件中
assert os.path.exists(txt_path), "not found {} file.".format(txt_name)
with open(txt_path) as read:
xml_list = [os.path.join(self.annotations_root, line.strip() + ".xml")
for line in read.readlines() if len(line.strip()) > 0]
self.xml_list = [] # 保存了所有标注文件名，以xml为后缀
# check file
for xml_path in xml_list:
if os.path.exists(xml_path) is False:
print(f"Warning: not found '{
xml_path}', skip this annotation file.")
continue
# check for targets
with open(xml_path) as fid:
xml_str = fid.read()
xml = etree.fromstring(xml_str)
data = self.parse_xml_to_dict(xml)["annotation"] # 将xml文件中的信息以字典形式存储
if "object" not in data:
# 如果数据中不存在物体对象，那么跳过此文件
print(f"INFO: no objects in {
xml_path}, skip this annotation file.")
continue
self.xml_list.append(xml_path) # 将有效文件路径存储起来
assert len(self.xml_list) > 0, "in '{}' file does not find any information.".format(txt_path)
# read class_indict
json_file = './pascal_voc_classes.json'
assert os.path.exists(json_file), "{} file not exist.".format(json_file)
with open(json_file, 'r') as f:
self.class_dict = json.load(f) # json文件转为字典
self.transforms = transforms
def __len__(self):
return len(self.xml_list)
def __getitem__(self, idx):
# read xml
xml_path = self.xml_list[idx] # 传入索引，选取一个有效的数据集文件
with open(xml_path) as fid:
xml_str = fid.read()
xml = etree.fromstring(xml_str)
data = self.parse_xml_to_dict(xml)["annotation"] # 读取标注信息
img_path = os.path.join(self.img_root, data["filename"]) # 读取标注数据中对应的图片
image = Image.open(img_path)
if image.format != "JPEG":
raise ValueError("Image '{}' format not JPEG".format(img_path))
boxes = [] # 边界框坐标
labels = [] # 标签
iscrowd = [] # 是否重叠
assert "object" in data, "{} lack of object information.".format(xml_path)
for obj in data["object"]:
xmin = float(obj["bndbox"]["xmin"])
xmax = float(obj["bndbox"]["xmax"])
ymin = float(obj["bndbox"]["ymin"])
ymax = float(obj["bndbox"]["ymax"])
# 进一步检查数据，有的标注信息中可能有w或h为0的情况，这样的数据会导致计算回归loss为nan
if xmax <= xmin or ymax <= ymin:
print("Warning: in '{}' xml, there are some bbox w/h <=0".format(xml_path))
continue
boxes.append([xmin, ymin, xmax, ymax])
labels.append(self.class_dict[obj["name"]])
if "difficult" in obj:
iscrowd.append(int(obj["difficult"]))
else:
iscrowd.append(0)
# convert everything into a torch.Tensor
boxes = torch.as_tensor(boxes, dtype=torch.float32)
labels = torch.as_tensor(labels, dtype=torch.int64)
iscrowd = torch.as_tensor(iscrowd, dtype=torch.int64)
image_id = torch.tensor([idx])
area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
target = {
}
target["boxes"] = boxes
target["labels"] = labels
target["image_id"] = image_id
target["area"] = area
target["iscrowd"] = iscrowd
if self.transforms is not None:
image, target = self.transforms(image, target)
return image, target
def get_height_and_width(self, idx):
# read xml
# 获取图片的尺寸
xml_path = self.xml_list[idx]
with open(xml_path) as fid:
xml_str = fid.read()
xml = etree.fromstring(xml_str)
data = self.parse_xml_to_dict(xml)["annotation"]
data_height = int(data["size"]["height"])
data_width = int(data["size"]["width"])
return data_height, data_width
def parse_xml_to_dict(self, xml):
"""
将xml文件解析成字典形式，参考tensorflow的recursive_parse_xml_to_dict
Args:
xml: xml tree obtained by parsing XML file contents using lxml.etree
Returns:
Python dictionary holding XML contents.
"""
if len(xml) == 0: # 遍历到底层，直接返回tag对应的信息
return {
xml.tag: xml.text}
result = {
}
for child in xml:
child_result = self.parse_xml_to_dict(child) # 递归遍历标签信息
if child.tag != 'object':
result[child.tag] = child_result[child.tag]
else:
if child.tag not in result: # 因为object可能有多个，所以需要放入列表里
result[child.tag] = []
result[child.tag].append(child_result[child.tag])
return {
xml.tag: result}
def coco_index(self, idx):
"""
该方法是专门为pycocotools统计标签信息准备，不对图像和标签作任何处理
由于不用去读取图片，可大幅缩减统计时间
Args:
idx: 输入需要获取图像的索引
"""
# read xml
xml_path = self.xml_list[idx]
with open(xml_path) as fid:
xml_str = fid.read()
xml = etree.fromstring(xml_str)
data = self.parse_xml_to_dict(xml)["annotation"]
data_height = int(data["size"]["height"])
data_width = int(data["size"]["width"])
# img_path = os.path.join(self.img_root, data["filename"])
# image = Image.open(img_path)
# if image.format != "JPEG":
# raise ValueError("Image format not JPEG")
boxes = []
labels = []
iscrowd = []
for obj in data["object"]:
xmin = float(obj["bndbox"]["xmin"])
xmax = float(obj["bndbox"]["xmax"])
ymin = float(obj["bndbox"]["ymin"])
ymax = float(obj["bndbox"]["ymax"])
boxes.append([xmin, ymin, xmax, ymax])
labels.append(self.class_dict[obj["name"]])
iscrowd.append(int(obj["difficult"]))
# convert everything into a torch.Tensor
boxes = torch.as_tensor(boxes, dtype=torch.float32)
labels = torch.as_tensor(labels, dtype=torch.int64)
iscrowd = torch.as_tensor(iscrowd, dtype=torch.int64)
image_id = torch.tensor([idx])
area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
target = {
}
target["boxes"] = boxes
target["labels"] = labels
target["image_id"] = image_id
target["area"] = area
target["iscrowd"] = iscrowd
return (data_height, data_width), target
@staticmethod
def collate_fn(batch):
return tuple(zip(*batch))
2.2 RPN
这里略去backbone的部分，因为backbone可以自由更换，如果采用了FPN，那么预测特征层存在多个。
2.2.1 RPN Head
对预测特征层列表进行遍历，针对预测特征层的每一个像素进行预测类别以及anchor的偏移量。因为每个像素会生成 n u m _ a n c h o r s num\_anchors num_anchors个anchor类别以及对应的偏移量。RPN中分类主要为了区分背景和物体类，因此是一个二分类问题。
class RPNHead(nn.Module):
"""
add a RPN head with classification and regression
通过滑动窗口计算预测目标概率与bbox regression参数
Arguments:
in_channels: number of channels of the input feature
num_anchors: number of anchors to be predicted
"""
def __init__(self, in_channels, num_anchors):
super(RPNHead, self).__init__()
# 3x3 滑动窗口
self.conv = nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=1, padding=1)
# 计算预测的目标分数（这里的目标只是指前景或者背景）
# 以特征图每个像素点来进行分类，因为边界框是基于像素点生产的，因此采用1x1卷积
self.cls_logits = nn.Conv2d(in_channels, num_anchors, kernel_size=1, stride=1)
# 计算预测的目标bbox regression参数
self.bbox_pred = nn.Conv2d(in_channels, num_anchors * 4, kernel_size=1, stride=1)
for layer in self.children():
if isinstance(layer, nn.Conv2d):
torch.nn.init.normal_(layer.weight, std=0.01)
torch.nn.init.constant_(layer.bias, 0)
def forward(self, x):
# type: (List[Tensor]) -> Tuple[List[Tensor], List[Tensor]]
# FPN的话会有多个feature map,所以需要对每个特征图进行遍历卷积
logits = []
bbox_reg = []
# 使用FPN时有多个预测特征层
for i, feature in enumerate(x):
# 遍历预测特征层列表
t = F.relu(self.conv(feature))
logits.append(self.cls_logits(t))
bbox_reg.append(self.bbox_pred(t))
# logits [num_feature_maps, Batch_size, num_anchors, H, W]
# bbox_reg [num_features_maps, Batch_size, num_anchors * 4, H, W]
return logits, bbox_reg
2.2.2 anchor生成
首先要基于(0,0）为每个预测特征层生成一系列anchors模板。
class AnchorsGenerator(nn.Module):
__annotations__ = {
"cell_anchors": Optional[List[torch.Tensor]],
"_cache": Dict[str, List[torch.Tensor]]
}
"""
anchors生成器
Module that generates anchors for a set of feature maps and
image sizes.
The module support computing anchors at multiple sizes and aspect ratios
per feature map.
sizes and aspect_ratios should have the same number of elements, and it should
correspond to the number of feature maps.
sizes[i] and aspect_ratios[i] can have an arbitrary number of elements,
and AnchorGenerator will output a set of sizes[i] * aspect_ratios[i] anchors
per spatial location for feature map i.
Arguments:
sizes (Tuple[Tuple[int]]):
aspect_ratios (Tuple[Tuple[float]]):
"""
def __init__(self, sizes=(128, 256, 512), aspect_ratios=(0.5, 1.0, 2.0)):
super(AnchorsGenerator, self).__init__()
if not isinstance(sizes[0], (list, tuple)):
# TODO change this
sizes = tuple((s,) for s in sizes)
if not isinstance(aspect_ratios[0], (list, tuple)):
aspect_ratios = (aspect_ratios,) * len(sizes)
assert len(sizes) == len(aspect_ratios)
self.sizes = sizes
self.aspect_ratios = aspect_ratios
self.cell_anchors = None
self._cache = {
}
def generate_anchors(self, scales, aspect_ratios, dtype=torch.float32, device=torch.device("cpu")):
# type: (List[int], List[float], torch.dtype, torch.device) -> Tensor
# 生成以(0,0)的anchors模板
"""
compute anchor sizes
Arguments:
scales: sqrt(anchor_area)
aspect_ratios: h/w ratios
dtype: float32
device: cpu/gpu
"""
scales = torch.as_tensor(scales, dtype=dtype, device=device)
aspect_ratios = torch.as_tensor(aspect_ratios, dtype=dtype, device=device)
h_ratios = torch.sqrt(aspect_ratios)
w_ratios = 1.0 / h_ratios
# [r1, r2, r3]' * [s1, s2, s3]
# number of elements is len(ratios)*len(scales)
ws = (w_ratios[:, None] * scales[None, :]).view(-1)
hs = (h_ratios[:, None] * scales[None, :]).view(-1)
# left-top, right-bottom coordinate relative to anchor center(0, 0)
# 生成的anchors模板都是以（0, 0）为中心的, shape [len(ratios)*len(scales), 4]
base_anchors = torch.stack([-ws, -hs, ws, hs], dim=1) / 2
return base_anchors.round() # round 四舍五入
def set_cell_anchors(self, dtype, device):
# type: (torch.dtype, torch.device) -> None
if self.cell_anchors is not None:
cell_anchors = self.cell_anchors
assert cell_anchors is not None
# suppose that all anchors have the same device
# which is a valid assumption in the current state of the codebase
if cell_anchors[0].device == device:
return
# 根据提供的sizes和aspect_ratios生成anchors模板
# anchors模板都是以(0, 0)为中心的anchor
# 为每个特征层生成模板
cell_anchors = [
self.generate_anchors(sizes, aspect_ratios, dtype, device)
for sizes, aspect_ratios in zip(self.sizes, self.aspect_ratios)
]
self.cell_anchors = cell_anchors
def num_anchors_per_location(self):
# 计算每个预测特征层上每个滑动窗口的预测目标数
return [len(s) * len(a) for s, a in zip(self.sizes, self.aspect_ratios)]
# For every combination of (a, (g, s), i) in (self.cell_anchors, zip(grid_sizes, strides), 0:2),
# output g[i] anchors that are s[i] distance apart in direction i, with the same dimensions as a.
def grid_anchors(self, grid_sizes, strides):
# type: (List[List[int]], List[List[Tensor]]) -> List[Tensor]
"""
anchors position in grid coordinate axis map into origin image
计算预测特征图对应原始图像上的所有anchors的坐标
Args:
grid_sizes: 预测特征矩阵的height和width
strides: 预测特征矩阵上一步对应原始图像上的步距
"""
anchors = []
cell_anchors = self.cell_anchors # 边界框模板
assert cell_anchors is not None
# 遍历每个预测特征层的grid_size，strides和cell_anchors
for size, stride, base_anchors in zip(grid_sizes, strides, cell_anchors):
grid_height, grid_width = size
stride_height, stride_width = stride
device = base_anchors.device
# For output anchor, compute [x_center, y_center, x_center, y_center]
# shape: [grid_width] 对应原图上的x坐标(列)
shifts_x = torch.arange(0, grid_width, dtype=torch.float32, device=device) * stride_width
# shape: [grid_height] 对应原图上的y坐标(行)
shifts_y = torch.arange(0, grid_height, dtype=torch.float32, device=device) * stride_height
# 计算预测特征矩阵上每个点对应原图上的坐标(anchors模板的坐标偏移量)
# torch.meshgrid函数分别传入行坐标和列坐标，生成网格行坐标矩阵和网格列坐标矩阵
# shape: [grid_height, grid_width]
shift_y, shift_x = torch.meshgrid(shifts_y, shifts_x)
shift_x = shift_x.reshape(-1)
shift_y = shift_y.reshape(-1)
# 计算anchors坐标(xmin, ymin, xmax, ymax)在原图上的坐标偏移量
# shape: [grid_width*grid_height, 4]
shifts = torch.stack([shift_x, shift_y, shift_x, shift_y], dim=1)
# For every (base anchor, output anchor) pair,
# offset each zero-centered base anchor by the center of the output anchor.
# 将anchors模板与原图上的坐标偏移量相加得到原图上所有anchors的坐标信息(shape不同时会使用广播机制)
# shifts_anchor [grid_width*grid_height, num_anchors, 4]
shifts_anchor = shifts.view(-1, 1, 4) + base_anchors.view(1, -1, 4)
anchors.append(shifts_anchor.reshape(-1, 4))
return anchors # List[Tensor(all_num_anchors, 4)] 每一个预测特征层的anchors的坐标
def cached_grid_anchors(self, grid_sizes, strides):
# type: (List[List[int]], List[List[Tensor]]) -> List[Tensor]
"""将计算得到的所有anchors信息进行缓存"""
key = str(grid_sizes) + str(strides)
# self._cache是字典类型
if key in self._cache:
return self._cache[key]
anchors = self.grid_anchors(grid_sizes, strides)
self._cache[key] = anchors
return anchors
def forward(self, image_list, feature_maps):
# type: (ImageList, List[Tensor]) -> List[Tensor]
# 获取每个预测特征层的尺寸(height, width)
# 因为FPN有多个预测特征层，因此需要进行遍历，获取每一个特征层的形状
grid_sizes = list([feature_map.shape[-2:] for feature_map in feature_maps])
# 获取输入图像的height和width
# 这里输入的图像高和宽是统一过的
image_size = image_list.tensors.shape[-2:]
# 获取变量类型和设备类型
dtype, device = feature_maps[0].dtype, feature_maps[0].device
# one step in feature map equate n pixel stride in origin image
# 计算特征层上的一步等于原始图像上的步长
strides = [[torch.tensor(image_size[0] // g[0], dtype=torch.int64, device=device),
torch.tensor(image_size[1] // g[1], dtype=torch.int64, device=device)] for g in grid_sizes]
# 根据提供的sizes和aspect_ratios生成anchors模板
self.set_cell_anchors(dtype, device)
# 计算/读取所有anchors的坐标信息（这里的anchors信息是映射到原图上的所有anchors信息，不是anchors模板）
# 得到的是一个list列表，对应每张预测特征图映射回原图的anchors坐标信息
anchors_over_all_feature_maps = self.cached_grid_anchors(grid_sizes, strides)
anchors = torch.jit.annotate(List[List[torch.Tensor]], [])
# 遍历一个batch中的每张图像
for i, (image_height, image_width) in enumerate(image_list.image_sizes):
anchors_in_image = []
# 遍历每张预测特征图映射回原图的anchors坐标信息
for anchors_per_feature_map in anchors_over_all_feature_maps:
anchors_in_image.append(anchors_per_feature_map)
anchors.append(anchors_in_image)
# 将每一张图像的所有预测特征层的anchors坐标信息拼接在一起
# anchors是个list，每个元素为一张图像的所有anchors信息
anchors = [torch.cat(anchors_per_image) for anchors_per_image in anchors]
# Clear the cache in case that memory leaks.
self._cache.clear()
return anchors # [Batch_size, total_anchors, 4]