PolyFormer: Referring Image Segmentation as Sequential Polygon Generation Paper Reading Notes

write in front

  I am almost busy with the things I should be busy with. I will continue to read papers and write notes, and make up for what I missed in the previous two months! (There are still 7 blog posts left), welcome to pay attention and continue to be productive.

• Paper address: PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
• Code address: Not provided by the author yet
• It is expected to be submitted to CVPR 2023. The list was released yesterday or the day before yesterday. I don’t know what the situation is.
• 23.5.6 First update:
  included in: CVPR 2023
  project/code address: https://polyformer.github.io/

1. Abstract

  Current reference image segmentation generally does not directly predict the target mask, but rather as a polygon sequence generation task. This article proposes a regression-based decoder that can directly predict precise geometric position coordinates, abandoning the previous practice of needing to quantify coordinates onto a fixed grid. Not only does it perform well on conventional data sets, but it also generalizes well to reference video image segmentation.

2. Introduction

  As usual, let’s talk about the definition of the reference image segmentation task and click on the routine of traditional instance segmentation, which is now based on contour points.
  As shown in Figure 1, this paper proposes a sequence-sequence model Polygon transFormer (PolyFormer) to solve this task. The prediction of each vertex is based on the previously predicted contour vertex, so the overall predicted contour points are no longer independent of each other. The input and output formats of PolyFormer are fixed, a set of variable-length sequences, which avoids the previous multi-modal fusion operation. For the output sequence, multiple polygon outlines are separated by specific separators. The framework proposed in this article can also be directly used to predict bounding boxes, because it is equivalent to predicting two coordinates. So the network built in this article unifies the two tasks of reference image segmentation and understanding (what a gimmick!).

  Recent Seq2Seq models quantize precise coordinates into a set of grids, thereby transforming the coordinate prediction task into a classification task. This article points out that this approach is not very good, because no matter how quantified, it is not accurate, because the coordinates are in a continuous space, and quantization cannot avoid errors. Therefore, this article treats it as a regression task, because coordinate regression is a floating point number, and the quantization error is negligible. Inspired by Mask RCNN, this paper uses bilinear interpolation method to interpolate floating point coordinates based on two adjacent index embeddings.

  Contributions are as follows:

• This paper proposes PolyFormer to solve the tasks of Reference Image Segmentation (RIS) and Reference Expression Understanding (REC), treating them as sequence-to-sequence prediction problems.
• A regression-based decoder is proposed for accurately and directly predicting 2D coordinates, removing quantization errors. This paper is the first work that treats the coordinate positioning problem as a regression problem on the Seq2Seq framework.
• This paper works well on several conventional data sets and can be generalized to video and synthetic data.

3. Related work

3.1 Referring Image Segmentation (RIS)

  Let’s talk about the definition of RIS and the previous workflow. The closest thing to this article is SeqTR, but it can only generate a rough mask of 18 vertices, and cannot outline complex shapes and occlusion masks.

3.2 Referring Expression Comprehension (REC)

  Let’s talk about the definition of REC, the previous two-stage and one-stage methods, and the recent unified framework for multi-task learning (the effect is not good). The performance of the method proposed in this article on multi-tasks improves the performance of RIS and REC.

3.3 Sequence-to-Sequence (seq2seq) Modeling

  Let’s talk about the origin of the seq2seq framework and point out the shortcomings: there are errors in quantification. The PolyFormer proposed in this article treats the coordinate positioning problem as a regression problem and can predict continuous coordinates.

3.4 Contour-based instance segmentation

  Let’s talk about the contour-based method in instance segmentation and point out the shortcomings: it cannot solve the problem of dividing the same target into multiple blocks.

4. PolyFormer

4.1 Structure overview

4.2 Target sequence construction

4.2.1 Polygon representation

  A mask can be a combination of one or more polygons to outline a reference target. Therefore, each polygon parameter is converted into a sequence set of two-dimensional coordinates: { ( xi , yi ) } i = 1 K , ( xi , yi ) ∈ R 2 \left\{\left(x_{i}, y_{ i}\right)\right\}_{i=1}^{K},\left(x_{i}, y_{i}\right) \in \mathbb{R}^{2}{ (xi​,yi​)}i=1K​,(xi​,yi​)∈R2 , where the vertex closest to the upper left corner of the image is used as the starting point of the sequence. As shown below:

4.2.2 Vertices and special tokens

  Insert <SEP> between two polygon sequences to represent multiple polygons. Multiple polygons from the same target are sorted according to the distance between their vertices and the image origin. Finally, <BOS> and <EOS> are used to indicate the beginning and end of the sequence.

4.2.3 Unified sequence with bounding boxes

  A bounding box can be represented by two corner points, such as the upper left corner point $\left(x_1^b,y_1^b\right)$ and the lower right corner point$\left(x_2^b,y_2^b\right)$ . Then multiple polygons and bounding boxes are spliced ​​together to form a long sequence:
$$\begin{array}{l}[<\mathrm{BOS}>,\left(x_1^b,y_1^b\right),\left(x_2^b,y_2^b\right),\left(x_1^1,y_1^1\right)\left(x_2^1,y_2^1\right),\dots ,<\mathrm{SEP}>,\left(x_1^n,y_1^n\right),\dots ,<\mathrm{EOS}>]\end{array}$$其中 $\left(x_1^n,y_1^n\right)$ is thennthStarting vertices of$n\; polygons.$Usually the corner points of the bounding box and the vertices of the polygon are regarded as coordinate tokens, that is, <COO>.

4.3 Image and text feature extraction

4.3.1 Image encoder

  Use Swin Transformer to extract the image $I\in {\mathbb{R}}^{}$The fourth stage characteristic of${}^H$${}^{\times }$${}^W$${}^{\times }$${}^3$$F_v\in R^{\frac{H}{32}\times \frac{W}{32}\times C_{in}}$, as a visual representation.

4.3.2 Text encoder

  Use BERT to extract the input with $Linguistic\; description\; of\; L$ words$T\in R^L$ , get the word feature$F_l\in R^{L\times C_l}$。

4.3.3 Multi-modal Transformer encoder

  Level $F_v$Obtain a set of visual sequence features $F_v^{\prime }\in {\mathbb{R}}^{\left(\frac{H}{32}\cdot \frac{W}{32}\right)\times C_v}$, and use the fully connected layer to convert $F_v^{\prime }$and $F_l$Projected to the same embedding space:
$$F_v^{\prime }=F_v^{\prime }{W}_v+b_v,F_l^{\prime }=F_l{W}_l+b_l$$
Among them $W_v$，$W_l$，$b_v$， $b_l$are learnable weights and biases. Then splice $F_v^{\prime }$and $F_l^{\prime }$ 为：$F_M=[F_v^{\prime },F_l^{\prime }]$。

  Multi-modal encoder by $Composed\; of\; N$ Transformer layers, output multi-modal features$F_M^N$. In addition, 1D and 2D relative position encoding are also added to image and text features respectively.

4.4 Regression-based Transformer decoder

4.4.1 2D coordinate embedding

  Create a set of 2D codebooks $\mathcal{D}\in {\mathbb{R}}^{B_H\times B_W\times C_e}$, where $B_H$Japanese $B_W$are the number of cells along the height and width directions respectively. Then using bilinear interpolation, you can get any floating point coordinates $(x,y)\in {\mathbb{R}}^2$ : First use$\left(x,y\right)$Produces four exact grids: $(\underline{x},\underline{y})$，$\left(\bar{x},\underline{y}\right)$，$\left(\bar{x},\underline{y}\right)$，$\left(\bar{x},\bar{y}\right)$$\in {\mathbb{N}}^2$ , the corresponding index from$\mathcal{D}$ 里面取出： $e_{\left(\underline{x},\underline{y}\right)}=\mathcal{D}\left(\underline{x},\underline{y}\right)$, and then obtain the precise coordinate embedding $e_{(x,y)}$：
$$\begin{array}{rl}{e}_{(x,y)}=& (\bar{x}-x)(\bar{y}-y)\cdot e_{(\underline{x},\underline{y})}+(x-\underline{x})(\bar{y}-y)\cdot e_{(\bar{x},\underline{y})}+\\ & (\bar{x}-x)(y-\underline{y})\cdot e_{(\underline{x},\bar{y})}+(x-\underline{x})(y-\underline{y})\cdot e_{(\bar{x},\bar{y})}\end{array}$$

4.4.2 Transformer decoder layer

  Same as $N$ Transformer decoder layers.

4.4.3 Predictive header

The prediction head consists of a classification head and a coordinate head, and the input is the output QNQ^{N}   of the last layer of the decoder.$Q^{N.}$ _ The classification head is a single-layer linear layer, output token type (<SEP>,<COO>,<EOS>):$\hat{p}=W_c{Q}^N+b_c$, where $W_c$and $b_c$are the parameters of the linear layer.
  The coordinate head consists of 3 layers of FFN, with a RELU activation function between each layer. Predict the bounding box and polygon vertices of the 2D reference target: $(\hat{x},\hat{y})=\mathrm{Sigmoid}\left(FFN\left(Q^N\right)\right)$。

4.5 Training

4.5.1 Polygon enhancement

  A polygon is usually a sparse representation consisting of a dense set of target corner points. Given a dense set of corner points, the resulting sparse polygon is usually not unique. As shown below:

  Dense corner points are first interpolated from the original polygon outline, and then sparse polygons are produced by uniform downsampling from a set of iterable sample points within a fixed range. This operation thus creates different levels of polygon distribution, from coarse to fine-grained, and training with this prevents the model from overfitting to a fixed polygon representation.

4.5.2 Objectives

  模型来训练并预测下一个 token 及其类型：$$L_t=l_t{L}_{coo}((x_t,y_t),({\hat{x}}_t,{\hat{y}}_t)|I,T,(x_t,y_t{)}_{t=1:t-1})\cdot \mathbb{I}[p_t=-<\widehat{CO}>]+l_{cls}{L}_{cls}(p_t,{\hat{p}}_t|I,T,{\hat{p}}_{1:t-1})$$ except$l_t=\{\begin{array}{ll}{l}_{box},& t\le 2,\\ l_{poly},& otherwise,\end{array}$，$L_{coo}$is $L_1$Regression loss, $L_{cls}$is $L_1$Label smoothing cross-entropy loss, $I[\cdot ]$ is the index function. Regression loss$L_1$Only used to calculate coordinate tokens, $l_{box}$和$l_{poly}$is the corresponding tokens weight. $L_t$is the sum of losses for all tokens in a sequence.

4.5.3 Reasoning

  During the inference phase, other tokens are generated by inputting the <BOS> token. First, the token type will be obtained from the classification header. If it is a coordinate token, the coordinate head will obtain the next 2D coordinate prediction based on the previously predicted tokens; if it is a separation token, it represents the end of a polygon prediction, so the separation token will be added to the output. sequence.

  Once <EOS> starts to be output, sequence prediction stops. In the generated sequence, the first and second tokens are the coordinates of the bounding box and the coordinates of the polygon vertices. The final segmentation mask is obtained from these polygons.

5. Experimental results

5.1 Datasets and Metrics

5.1.1 Dataset

  RIS and REC: RefCOCO, RefCOCO+, and RefCOCOg.

5.1.2 Evaluation indicators

  mean IoU、overall IoU、[email protected]

5.2 Implementation details

5.2.1 Model settings

  PolyFormer-B：Swin-B + BERT-base
  PolyFormer-L：Swin-L + BERT-base

5.2.2 Training details

  Perform REC tasks, that is, pre-training, on data composed of Visual Genome, RefCOCO, RefCOCO+, RefCOCOg, and Flickr30k-entities. In the multi-task fine-tuning stage, RefCOCO, RefCOCO+, RefCOCOg training sets are fine-tuned. AdanW, lr=0.00005, 20 epoch pre-training 160batch + 100 epoch fine-tuning 128batch. $l_{box}=0.1$ ,$l_{poly}=1$，$l_{cls}=0.0005$ . Input image size$512\times 512$ , 50% polygon enhancement probability, 2D coordinate embedding codebook set to$64\times 64$ grids.

5.3 Main results

5.3.1 Reference image segmentation

5.3.2 Reference expression understanding

5.3.3 Zero-shot migration to reference video object segmentation

5.4 Ablation experiment

  Ablation experiments were performed on the RefCOCO, RefCOCO+, and RefCOCOg validation sets, using the model PolyFormer-B.

5.4.1 Coordinate classification vs. regression

5.4.2 Component analysis of target sequence components

5.5 Visualizing results

5.5.1 Cross attention map

5.5.2 Prediction visualization

6. Conclusion

  This paper proposes PolyFormer, a simple and unified sequence framework for RIS and REC. In addition, a novel decoder is designed to generate continuous 2D coordinates without quantization error, and the experimental results are very good.

Limitations and wider implications

  Training PolyFormer requires a large amount of accurately labeled bounding box and polygon vertex labeling data, and using weakly supervised data to achieve regional-level understanding requires further research.

appendix

A. Additional dataset details

  An introduction to the Ref-DAVIS17 dataset.

B. Additional implementation details

  PolyFormer-B 中$C_v$The feature dimension is 1024, and $C_v$The feature dimension is 1536. Linguistic features $C_l$and coordinate embedding $C_e$The dimensions are all 768; the attention heads are 12; GELU activation is used in each layer of Transformer; for $F_{cls}$, the label smoothing factor is set to 0.1.

C. Additional experimental results

D. More visual results

D.1 Cross attention map

D.2 Visualization of prediction results

write on the back

  The novelty of this article is quite good, and it is good at breaking the rules (maybe because the funds are in place, haha). The polygon data enhancement techniques and token splicing techniques are worth learning.

  PS: Continue tomorrow! Welcome to follow~