DETR Series Mask Frozen-DETR: High Quality Instance Segmentation with One GPU Paper Reading Notes

write in front

  The highlight of this article is obviously that it can run with just one GPU! The way this topic is conceived is worth learning.

• Paper address: Mask Frozen-DETR: High Quality Instance Segmentation with One GPU
• Code address: arxiv version not provided
• Expected to submit at: a top conference
• Ps: Reading notes for a blog post every week in 2023. More useful information on the homepage . Welcome to follow me. I look forward to your participation among the 5,000 fans~

1. Abstract

  This paper aims to study the establishment of an instance segmenter, Mask Frozen-
DETR, that requires minimal training time and GPU, and can convert any DETR-based object detection model into an instance segmentation model. The proposed method only needs to train an additional lightweight mask network to predict instance masks within the Bounding box through a frozen DETR-based object detector. It works very well on the COCO data set and only requires a single V100 16G card for training.

2. Introduction

  First, we point out the difficulty of instance segmentation, some recent methods: Cascade Mask R-CNN, Mask DINO. These methods usually use ResNet-50 and Swin-L as backbone and require a lot of GPU time to train.
  This paper shows that existing DETR-based object detection models can be transformed into instance segmentation models. Take $Taking\; H$ -DETR and DINO-DETR as examples, two innovations are proposed: designing a lightweight instance segmentation network to effectively utilize the output of the frozen DETR target detector; and demonstrating the effectiveness of the method proposed in this article under different scale models.

3. Related work

Target Detection

  Object detection is a basic research field, involving a lot of work: Faster R-CNN, Cascade R-CNN, YOLO, DETR, Deformable DETR, DINO-DETR, H \mathcal{H $H$ -DETR. The method in this article is built on these DETR-based methods, achieving SOTA performance and 10 times the training speed.

Instance splitting

  The initial instance segmentation method is based on the R-CNN series, with the idea of ​​detecting first and then segmenting. For example, Mask R-CNN, Casacde Mask R-CNN, HTC. Some methods are relatively simple in design: SOLO, QueryInst. There are some methods based on DETR: MaskFormer, Mask2Former, Mask-DINO. There are also methods based on 3D instance segmentation, such as SPFormer.

discuss

  Most instance segmentation models do not utilize the weights of offline object detection models and therefore require a large amount of training time. The method in this article adopts a frozen DETR-based model and introduces a lightweight instance segmentation head. The figure below is a comparison of some methods:

4. Methods of this article

4.1 Baseline settings

  Use $H$ -DETR+ResNet50 as Baseline and reports$Comparative\; performance\; of\; H$ -DETR+Swin-L and DINO-DETR+FocalNet-L. $The\; H$ -DETR+ResNet50 model is pre-trained on the Object365 dataset and then fine-tuned on the COCO dataset.
  First, based on the query scores output by the last Transformer decoder layer of the target detection model, the top~100 target queries are selected for mask prediction. Then these queries{ qi ∣ qi ∈ R d } i = 1 N \{ { \mathbf{q}_i|\mathbf{q}_i\in\mathbb{R}^\mathsf{d}\}}_{ i=1}^N{ qi​∣qi​∈Rd}i=1N​with 1 / 4 1/4Image feature F ⁡ ∈ RHW 16 × d {\operatorname*{F}}\in\mathbb{R}^{\frac{\mathsf{H}\mathsf{W}}{16} at 1/4$original$ resolution$F\in R^{\frac{\mathsf{HW}}{16}\times \mathsf{d}}$ 相乘，得到实例分割 masks：
$$\begin{array}{rl}\text{F}& ={\mathrm{C}}_1+\text{interpolate}(\mathbf{E})\\ {\mathrm{M}}_i& =\text{interpolate}(\text{reshape}(\text{Sigmoid}({\mathbf{q}}_i{\mathbf{F}}^{\top })))\end{array}$$
其中 ${\mathrm{C}}_1\in {\mathbb{R}}^{\frac{\mathsf{HW}}{16}\times d}$ represents the feature map output by Backbone in the first stage,$\mathbf{E}\in R^{\frac{HW}{64}\times d}$ represents 1 / 8 1/8of the Transformer encoder output$Feature\; map\; at\; 1/8$ original resolution. $\mathsf{H}$，$\mathsf{W}$，$d$ represents the height, width and feature hidden dimensions of the input image respectively. ${\mathrm{M}}_i\in {\mathbb{R}}^{HW}$ represents the last predicted mask probability map. The picture below shows the overall roadmap:

  Next, the confidence score is calculated to reflect the quality of the mask:
$${\mathrm{s}}_i=c_i\times \frac{\mathrm{sum}({\mathbf{M}}_i[{\mathbf{M}}_i>0.5])}{\mathrm{sum}([{\mathbf{M}}_i>0.5])}$$Among them $c_i$Represents the $The\; classification\; score\; of\; i$ target query.
  Next, RoIAlign is used within the predicted Bounding boxes to aggregate the regional feature maps used for positioning:
$${\mathbf{R}}_i=\mathrm{reshape}(\mathrm{RoIAlign}(\mathrm{reshape}(\mathbf{F}),{\mathbf{b}}_i))$$其中 $R_i\in R^{\mathsf{HW}\times \mathsf{d}}$，$b_i$Indicates that according to ${\mathbf{q}}_i$Predicted bounding box. Default setting $\mathsf{h}=32$，$\mathsf{w}=32$。之后计算实例分割 mask：
$${\mathbf{M}}_i^r=\text{paste}(\text{interpolate}(\text{reshape}(\text{Sigmoid}({\mathbf{q}}_i{\mathbf{R}}_i^{\top }))))$$ First deform and scale the predicted area masks to the original image size and paste them onto an empty mask. Also based on M ir \mathbf{M}_i^rin a similar way$M_i^r$to calculate the confidence score.

result

  According to the Baseline experiments in the table above, it is shown that the performance of the model can be improved from three aspects: the design of the image feature encoder, the design of the box area feature encoder, and the design of the query feature encoder.
  Next, in the ablation experiment, the entire target detector network was frozen, and only the additional introduced parameters were fine-tuned, training for about 6 epochs.

Experimental setup

   AdamW optimizer, initial learning rate $1.5\times 10^{-4}$，$b_1=0.9$，$b_2=0.999$ , weight decay$5\times 10^{-5}$ , train the model 88500 times, that is, 6 epochs, and the learning rate decreases by 10% at 0.9 and 0.95 iterations. The steps of data preprocessing are the same as those of Deformable DETR, Batch_size 8, all experiments are performed on a single V100 16G card, and the evaluation indicators are COCO: AP, AP 50, AP75_,AP_S,_AM_,A_L.

4.2 Image feature encoder

  First, a trainable image feature encoder is integrated to convert the image feature map into a more suitable feature space for instance segmentation masks. The following figure shows the overall roadmap:

  As shown in the figure above, apply the image feature encoder to the feature map $\mathbf{E}$ 中：
$$\mathrm{F}={\mathrm{C}}_1+\text{interpolate}({\mathcal{F}}_e(E))$$ Among them,$F_e$Represents an image feature encoder that aims to refine the image feature maps of all target queries simultaneously. Its structure consists of three types of convolution or Transformer blocks.

Deformable encoder block

  The design of multi-scale deformable encoder is used to stack multi-scale deformable encoder blocks to enhance the multi-scale feature map:
$$\begin{array}{rl}\text{E}& =[{\mathbf{E}}_1,{\mathbf{E}}_2,E_3,E_4]\\ F_e(\mathbf{E})& =\text{MultiScaleDeformable}\mathrm{Enc}(|{\mathbf{E}}_1,E_2,E_3,E_4|)\end{array}$$E 1 \mathbf{E} _1$E_1$, $E_2$, $E_3$, $E_4$Represent the feature maps of different scales of the Transformer encoder in the object detector respectively. Each multi-scale deformable encoder follows the design of MSDeformAttn → LayerNorm → FFN → LayerNorm, and FFN adopts the design of Linear → GELU → Linear.

Swin Transformer encoder block

  Following the design of Swin Transformer, apply stacked multiple Swin Transformer blocks to the highest resolution feature map $E_1$Ex:
$${\mathcal{F}}_e({\mathbf{E}}_1)=\text{SwinTransformer}\mathrm{Enc}({\mathbf{E}}_1)$$ Each Swin Transformer block follows the route of LayerNorm → W-MSA → LayerNorm → FFN. W-MSA represents sliding window multi-head self-attention operation.

ConvNext encoder block

  Apply ConvNext encoder block on the feature map of Transformer encoder:
$$F_e\left(E_1\right)=\text{ConvNextBlock}({\mathbf{E}}_1)$$ Each ConvNext block follows the route of DWC → LayerNorm → FFN. DWC represents a large-scale convolution kernel$7\times$Depth-wise convolution of$7\; .$

result

4.3 Box feature encoder

  The Box feature encoder roadmap is shown below:

As shown in the figure above, in ${\mathbf{R}}_i$Simply add an extra transformer layer in:
$$R_i={\mathcal{F}}_b({\mathbf{R}}_i)$$ Similar to the image feature encoder, study the impact of different box feature encoder structures on model performance.

channel mapper

  Use the channel mapper as a simple linear layer to reduce $F\in R^{\frac{\mathsf{HW}}{16}\times d}$ feature channel dimensions, and apply the box region feature encoder on the updated feature map.

result

4.4 Query feature encoder

Object-to-Object attention

  The proximity of objects to each other may cause multiple instances to be in the same Bounding box, so Object-to-Object attention is added to assist Object Query in distinguishing them from each other. Specifically, the multi-head attention mechanism is used to process Queries:
$$[{\mathbf{q}}_1,{\mathbf{q}}_2,\cdots ,q_N]=\text{SelfAttention}([{\mathbf{q}}_1,q_2,\cdots ,q_N])$$

Box-to-object attention

  In frozen DETR-based object detectors, object queries are used to perform object detection and process the entire image features instead of individual regional features. So Box-to-object attention is introduced to transform queries into queries adapted to the segmentation task:
$$q_i=\text{CrossAttention}({\mathbf{q}}_i,{\mathbf{R}}_i)$$ where the target queries are updated by referring to the box area features.

FFN

  FFN may also have an auxiliary role in adjusting Object query representation.

result

  Conclusion: Just use the design of Box-to-object attention.

4.5 Other improvements

Mask Loss on sampled pixels

  In the calculation process of the final loss, the Mask loss of the sampling point is introduced. Specifically, consider $N$ points, oversampling rate$k(k>1)$ , sampling rate$b(b\in [0,1])$ . Randomly sample k N kNfrom the output mask$kN$ points, and randomly select$\beta N$ points. Then randomly sample other$(1-\beta )N$ points, calculate this$A\; loss\; of\; N$ points.

Mask rating

  Since the classification score output by frozen DETR cannot reflect the quality of the segmentation mask, Mask scoring is introduced to adjust the classification score. Specifically, the mask scoring head takes the mask and box area features as input to predict the IoU scores of the output masks and GT:
$${\mathrm{iou}}_i=\mathrm{MLP}(\mathrm{Flatten}(\mathrm{Conv}(\mathrm{Cat}({\mathbf{M}}_i,{\mathbf{R}}_i))))$$ Among them, Cat, Conv, and MLP represent splicing operations, convolutional layers, and multi-layer perceptrons respectively. The iou score predicted by the mask scoring header is used to adjust the classification score:
$${\mathrm{s}}_i={\mathrm{c}}_i\mathrm{io}{\mathrm{u}}_i$$inside $s_i$is the confidence score of the output mask.

Neck structure for Backbone features

  Feature ${\mathbf{C}}_1\in {\mathbb{R}}^{\frac{\mathsf{HW}}{16}\times d}$ may not contain sufficient semantic information. So a simple neck block is introduced to encode semantic information into high-resolution feature map$C_1$:
$$C_1=\mathrm{GN}(\mathrm{PWConv}({\mathbf{C}}_1))$$ where$\mathrm{GN}$ 和 $PWConv$ refers to group regularization and point-wise convolution respectively.

result

5. Comparison with SOTA

6. Ablation experiment and analysis

Output size of RoIAlign

Training Epoch

large scale jitter

Design of instance mask header

Bach Size

Layer index of encoder feature map

The impact of image feature encoder depth and box feature encoder

Fine-tuning the effect of DETR

Qualitative results

7. Conclusion

  This article shows in detail how to convert an offline DETR-based object detection model into an instance segmentation model with minimal training time and cost. Experiments show that it is very effective.

write on the back

  After procrastinating for so long, I finally finished writing this blog post. The idea of ​​​​the article is to draw on the idea of ​​Parameter-Efficiency and introduce new structures to fine-tune the original model, which is something. Especially the experiments are very abundant. There are also disadvantages. Fig1 is not cited in the paper; there are many ablation experiments and it is not necessary to separate them all, and several can be merged.