RES Series GRES: Generalized Referring Expression Segmentation Paper Reading Notes

write in front

  This week is almost over and I haven’t done much. After I came to school, I reinstalled the Ubuntu system, and the previous one crashed. Now back up using the command line.
  This is an article about a new data set of RES. Let’s see if there is any opportunity to publish some papers.

• Paper address: GRES: Generalized Referring Expression Segmentation
• Code address: https://github.com/henghuiding/ReLA
• Included in: CVPR 2023
• Ps: Reading notes for a blog post every week in 2023. More useful information on the homepage . Welcome to follow me. I look forward to your participation among the 5,000 fans~
• September 5, 2023, Update 1️⃣: This group of people released a GREC technical report on Arxiv. The content is estimated to be similar. Those who are interested can pick it up: GREC: Generalized Referring Expression Comprehension

1. Abstract

  First of all, we point out the definition of Referring Expression Segmentation (RES). The current classic RES data sets and methods generally only support single-target expressions, that is, one expression corresponds to one target, and do not consider multi-target and targetless ones. expression. Therefore, this article proposes Generalized Referring Expression Segmentation (GRES), which enables expressions to point to any number of target categories. At the same time, the first large-scale GRES data set gRefCOCO was constructed, including expressions of multiple targets, no targets, and single targets. Experiments show that the main problem of GRES lies in complex relationship modeling. Based on this, a region-based ReLA model is proposed, which can adaptively divide the image into several regions containing instances, and explicitly model region-region and region-language dependencies. The proposed ReLA performs well on GRES and classic RES datasets.

2. Introduction

  
  Definition, application, and data set of Referring Expression segmentation (RES).

• Limitations of Classic RES
  First, classic RES does not consider expressions without targets, which means that existing RES methods cannot define those image-text pairs whose targets are not on the input image. Second, most existing RES datasets such as RefCOCO do not contain expressions for multiple targets. As shown below:

  Experiments show that traditional RES methods fail to identify these multiple-target or target-less scenes well.

• New benchmark and dataset
  This article proposes a new benchmark: Generalized Referring Expression Segmentation (GRES), which enables expressions to point to any number of targets. GRES also takes an image and a referential expression as input, but further supports multiple targets contained in a single expression and no targets contained in an image. In contrast, existing referential expression datasets do not contain these, but only expressions of single targets. The new data set established for GRES is called gRefCOCO, which complements RefCOCO with two types of samples: multi-objective expressions and non-objective expressions.
• New baseline method
  Region-region interaction is very important in RES. However, the classic RES method only needs to detect one target, so that most methods without region-region interaction modeling can achieve good performance. In GRES, however, there will be greater reliance on long-range region-region dependence modeling. So a network was designed to divide the image into several regions, and then explicitly interact with each other. In addition, the network proposed in this paper can softly collect the features of each region to achieve greater flexibility. Extensive experiments show that explicitly modeling elastic interactions of regional features can contribute to the performance of GRES.

  The contributions of this article are summarized as follows:

• A bechmark is proposed: Generalized Referring Expression Segmentation (GRES), making RES more flexible and easy to deploy in real scenarios.
• The first one proposed a large-scale GRES dataset gRefCOCO, which can support the expression of any number of targets.
• A baseline method: ReLA is proposed to model the complex relationships between targets, and implements SOTA on the classic RES and GRES tasks.
• A large number of experiments analyzed the possible reasons and new challenges for the performance gap between RES and GRES.

3. Related work

Related referential tasks and data sets

  Referring Expression Comprehension (REC) output bounding box, early data sets for RES and REC have ReferIt, but each expression only points to a single instance. Then the RefCOCO and RefCOCOg data sets were proposed. Again these are one expression corresponding to one instance.
  Recently, some new data sets have been proposed, but they do not focus on or are not suitable for GRES. PhraseCut, for example, contains multiple target expressions, but only as candidates, that is, only when a target cannot be used as an independent referent. In addition, its expressions are template compositions rather than natural language expressions. Image caption Image caption is also close to RES, but does not ensure obfuscation of expressions and is therefore not suitable for referencing related tasks. Some refer to data sets using other modalities or learning plans, such as Scanrefer focusing on 3D objects and Clevrtex focusing on unsupervised learning. Furthermore, none of these datasets contain expressions without targets.

Referent segmentation method

  There are roughly two types: single-stage (top-down) and two-stage (bottom-up). Single-stage methods usually have an end-to-end network similar to FCN, and achieve prediction by classifying multi-modal features pixel by pixel. The two-stage method first finds a set of instance proposals and then selects the target instance from it. Most RES are single-stage methods, while REC is a two-stage method. Recent Transformer methods significantly surpass CNN-based methods. The zero-shot segmentation method uses category names as text information, focuses on identifying new categories, and uses natural language expressions to identify the user's focus.

4. Task settings and data sets

4.1 GRES settings

RES Review

  The current RES does not consider targetless expressions, so existing models may output wrong instance results when dealing with multiple referents or no target referents on the input image. As shown below:

Generalized RES

  In order to solve the limitations of classic RES, this article proposes a new benchmark: Generalized Referring Expression Segmentation (GRES), which can allow expressions to point to any number of target objects. A GRES data sample contains four elements: an image $I$ , a language expression$T$ , one by$T$ points to the GT Mask MGT M_{GT}containing all pixels$M_{GT}$, a binary target label $E_{GT}$, indicating that $Whether\; T$ is an untargeted expression. TTThere is no limit to the number of instances in$T.$The GRES model is based on$I$ and$T$ is the input, predict a mask$M.$ _ For targetless expressions,$M$ should be negative.
  Multiple target expressions can achieve user-defined open vocabulary segmentation, and target-free expressions can identify whether the image contains targets in language expressions, so it is more practical.

Evaluate

  In gRefCOCO, models are not forced to provide different instance descriptions. In addition to the general RES performance indicators: cumulative IoU (cIoU) and Precision@X, a new indicator is further proposed: generalized IoU (gIoU), which extends the average IoU to all samples (including untargeted samples) . In addition, the performance of target-free samples is evaluated separately by calculating No-target-accuracy (N-acc.) and Target-accuracy (T-acc.).

4.2 gRefCOCO: a large-scale GRES dataset

  gRefCOCO contains 278232 expressions: 80022 multi-target and 32202 no-target, pointing to 60287 salient instances in 19994 images. The bounding box and mask of all target instances are given. Some single-objective expressions are derived from RefCOCO. An online annotation tool was developed to find images, select instances, write expressions and verify results.
  The basic annotation process originates from ReferIt, ensuring annotation quality. Keep the dataset proportions the same as the UNC part of RefCOCO.

Multiple target samples

  Usually users focus on selecting multiple targets based on logical relationships or similarities, rather than randomly assembling multiple targets. The annotator then writes an unobfuscated referential expression to select these instances, with the following four properties and challenges.

Use of counting expressions

  The original expression in Figure 3(a) contains an ordinal word, so the model must distinguish important quantities from the ordinal word quantities. Explicit or implicit target counting capabilities can be used to solve such expressions.

Compound word structure without set relationship

  As shown in Figure 3, "A and B", "A except B", "A with B or C", this puts higher requirements on the model to understand the long-distance dependence of images and sentences.

attribute domain

  When multiple targets are present in an expression, the different targets may share the same or different properties. This requires the model to understand all attributes and be able to map these attributes to the corresponding targets.

more complex relationship

  As shown in Figure 3(b), two similar expressions are applied to one image but point to two different target sets. Therefore, in GRES, relationships are not only used to describe goals but also imply the number of goals. This requires the model to understand all interactions between instances.

No target sample

  When untargeted expressions do not set any conditional pairs, annotators tend to write a large number of simple or general expressions that are very different from those that are valid. So two rules are set: the expression cannot be irrelevant to the overall image; the annotator can choose a misleading expression from other RefCOCO images with the same data distribution.

5. Proposed method

5.1 Structure preview

  The model structure is shown in the figure above. The input image is extracted by the Swin-Transformer encoder to extract the visual features $F_i\in {\mathbb{R}}^{H\times W\times C}$ , where$H$，$W$ is the spatial size of the image,$C$ is the channel dimension. The input language expression is processed by BERT to obtain the language feature$F_t\in R^{N_t\times C}$ , where$N_t$is the number of words in the expression. Next, $F_i$Send it to the pixel decoder to get the mask feature $F_m$. At the same time $F_i$and $F_t$It is sent to the proposed ReLAtionship building module to further decompose the feature map into $P\times P=P^2$ areas and then model the relationship between them. However, the shape and size of the feature map and its spatial area are not predetermined, but dynamically divided by ReLA. ReLA produces two types of features: regional featuresF r = { frn } n = 1 P 2 F_r=\{f_r^n\}^{P^2}_{n=1}Fr​={ frn​}n=1P2​and area filter $F_f=\{f_f^n{\}}_{n=1}^{P^2}$. For the $n$ regions, their regional characteristics$f_r^n$Used to find a scale $x_r^n$, indicating the probability that it contains the target, and its area filter $f_f^n$With mask feature $F_m$Perform dot multiplication to obtain the region segmentation mask $M_r^n\in {\mathbb{R}}^{h\times W}$ , indicating the area of ​​this region. Next we aggregate these masks by weighting them on the predicted masks:
$$M=\sum _n(x_r^n{M}_r^n)$$

output and loss

  Predicted mask $M$ through GT mask$M_{GT}$supervision. $P\times P$ probability map$x_r$By $M_{GT}$Downsampling to the smallest graph for supervision. At the same time, through all regional features $F_r$global average features to predict an untargeted label $E.$ _ During reasoning, if$E$ predicts a positive value, then the output mask$M$ will be set to empty. $M$、$x_r$、$E$ guides training through cross-entropy loss.

5.2 ReLAtionship modeling

  The proposed ReLAtionship modeling contains two modules: Region-Image Cross Attention (RIA), Region-Language Cross Attention (RLA). RIA elastically collects regional image features, and RLA captures region-region and region-image dependencies.

Region-Image Cross Attention (RIA)

  RIA uses visual features $F_i$and $P^2$ learnable region-based queries$Q_r$As input, it is supervised by the minimal graph in Figure 4. Each query corresponds to a spatial area in the image and is responsible for decoupling the features of this area. Its structure is shown in Figure 5(a). First, by performing image features $F_i$and $P^2$ 个 query embedding $Q_r\in {\mathbb{R}}^{P^2\times C}$attention to generate$P^2$ attention maps:
$$A_{\mathbf{r}i}=\mathrm{softmax}(Q_{r}s(F_i{W}_i{)}^T)$$where$W_{rk}$is $C\times C$ learnable parameters,$\sigma$ is GeLU. The output result$A_{\mathbf{r}i}\in {\mathbb{R}}^{P^2\times HW}$H × WH\times Wto each query$H\times The\; attention\; map\; of\; W$ indicates its corresponding area in the image. Next use these attention maps to get regional features from the corresponding regions:
$$\begin{array}{rlr}{F}_r^{\prime }& =& A_{ri}s(F_i{W}_{iv}{)}^T\end{array}$$Among them $W_{iv}$is $C\times C$ learnable parameters. In this approach, the features of each region are dynamically collected through their relative locations. Mask-based prediction$F_r^{\prime }$Get a region filter $F_f$, contains regional clues. $F_r^{\prime }$It is further fed into RLA for region-region and region-word interaction modeling.

Region-Language Cross Attention (RLA)

  Regional image feature $F_r^{\prime }$Originating from the image features that do not contain regional and language information relationships, the RLA module is proposed to model region-region and region-language interactions. As shown in Figure 5(b), RLA consists of a self-attention mechanism and multi-modal cross-attention. The self-attention module models region-region dependencies. Output the relationship-aware region feature F r 1 F_{r1} by calculating the attention matrix of a single region with all other regions$F_r$. At the same time, cross attention adopts language features $F_t$is the input of Value and key, regional image feature $F_t$As query input. So the relationship between each word and each region is modeled:
$$A_l=\mathrm{softmax}\left(\sigma \right(F_r^{\prime }{W}_{lq}\left)p\right(F_t{W}_{lk}{)}^T)$$其中 $A_l\in {\mathbb{R}}^{P^2\times N_t}$. The word-region attention is then shaped into regional features of language perception: $F_r=A_l{F}_t$. Finally, an MLP is used to aggregate the interactive sensing area features $F_r$and regional features of language perception $F_r$and regional image features $F_r^{\prime }$：$F_r=\text{MLP}(F_r^{\prime }+F_r+F_r)$。

6. Experiment

6.1 Evaluation indicators

  In addition to the widely used RES indicators: cumulative IoU (cIoU) and Precision@X (Pr@X), No-target accuracy (N-acc.), Target accuracy (T-acc.), and generalized IoU (gIoU) are further introduced. for GRES.

cIoU and Pr@X.

  cIoU calculates the ratio of the overall intersection pixel to the full set of pixels. Pr@X counts when IoU is greater than a certain threshold XXThe percentage of sample size at$X.$Samples without targets are not calculated in Pr@X. Since multi-target samples have a larger foreground area, it is easier to obtain higher cIoU scores. So the threshold$X$ is used for Pr@X starting from 0.7.

N-acc. and T-acc.

  These two indicators evaluate the performance of the model on untargeted samples: for untargeted samples, when there are no foreground pixels in the prediction, it is a positive sample (TP), otherwise it is a negative sample (FN). Then N-acc. is used to measure the performance of the model on untargeted samples: $\text{N-acc.}=\frac{TP}{TP+FN}$。

  T-acc. reflects the degree to which non-target samples affect target samples, that is, how many target samples are misclassified as non-target samples: T-acc. = $\text{T-acc.}=\frac{TN}{TN+FP}$。

gIoU.

  Since larger objects in cIoU have higher scores, it happens that targets in GRES have greater prospects. Therefore, generalized IoU (gIoU) is introduced to treat all samples equally. gIoU calculates the average over all samples for each image. For those untargeted samples, the IoU value of positive untargeted samples is regarded as 1, while the IoU value of false negative samples with targets is regarded as 0.

6.2 Ablation studies

The need for data sets

RIA design options

RLA design options

Regional PPnumber of$P$

6.3 Results on GRES

Comparison with RES sota method

Quantitative results

Failure cases & discussions

  The main reason is that the expression is too deceptive. The model needs to further find the fine-grained details of all objects and understand the details of the image context.

6.4 Results on classic RES

7. Conclusion

  This paper analyzes and solves the limitations of classical RES tasks, namely the inability to solve multimodal and goalless expressions. Based on this, a new benchmark is proposed: Generalized Referring Expression Segmentation (GRES), which allows any number of targets to be included in the expression. Correspondingly, a large-scale data set gRefCOCO was constructed, and a baseline method was proposed: ReLA, which explicitly models the relationship between different image areas and words, and implements a new SOTA on the classic RES and GRES tasks. The proposed GRES opens up new application scenarios, such as image retrieval.

write on the back

  This is an article with a lot of work. It proposes data sets, methods, and indicators. It can be said that it has dug a big hole. Interested students are advised to pay attention.