REC Series Visual Grounding with Transformers Paper Reading Notes

write in front

  Hello, another week has passed and school is about to start. I wonder if your friends have adjusted themselves? Come on~
  This is also an article about REC. The article is earlier, but it is also a must-read article for beginners.

• Paper address: VISUAL GROUNDING WITH TRANSFORMERS
• Code address: https://github.com/usr922/vgtr
• In: ICME 2022
• Ps: Reading notes for a blog post every week in 2023. More useful information on the homepage . Welcome to follow me. I look forward to your participation among the 5,000 fans~

1. Abstract

  This paper proposes a Transformer-based method for visual positioning. Unlike existing methods that first extract proposals and then sort them, which rely heavily on pre-trained target detectors, or proposal-less framework methods, a set of offline single-stage detectors are updated by fusing textual embeddings. The method proposed in this article, Visual Grounding with TRansformers VGTR, is built on the Transformer framework, independent of pre-trained detectors and word embedding, and is used to learn semantically discriminative visual features. The experiment achieved SOTA performance.

2. Introduction

  First, point out the definition, application and difficulties of visual positioning. Early approaches viewed visual localization as a special case of text-based image retrieval and framed it as the task of retrieving a referent from a set of candidate regions in a given image. These methods rely heavily on pre-trained object detectors and often ignore the visual context of the object. Furthermore, most methods require additional computational cost to generate and process candidate proposals.
  Some recent work removes the generation process of Proposals and directly locates the target, but the visual and textual features are still independent of each other. To alleviate this problem, this paper proposes an end-to-end Transformer-based network VGTR, which can capture global visual and language context without generating target proposals. Compared with those methods based on offline detectors or grid features, VGTR treats visual localization as a coordinate regression problem based on the target bounding box of the query sentence.

As shown in the figure above, VGTR consists of four modules: a basic encoder, used to calculate basic image-text pair tokens; a dual-stream positioning encoder, used to perform joint reasoning and cross-modal interaction of visual language; a positioning decoding The processor treats text tokens as positioning queries and proposes target-related features from the encoded visual tokens; a prediction head is used to perform coordinate regression of the bounding box. In addition, a new self-attention mechanism is designed to replace the original one and applied to visual tokens, establishing the association between the two modalities and learning text-guided visual features without reducing the positioning ability.

  The contributions are summarized below:

• An end-to-end framework VGTR is proposed for visual localization without the need for pre-trained detectors and pre-trained language models;
• A text-guided attention module is proposed to learn visual features guided by language descriptions;
• The method reaches SOTA.

3. Related work

3.1 Visual positioning

  This article divides visual positioning into two categories: proposal-rank and proposal-free. The former first generates a set of candidate target proposals from the input image through an offline detector or proposal generator, then associates language descriptions and scores these candidate proposals, and selects the one with the highest score as the positioning target. These methods rely heavily on the performance of pre-trained detectors or proposal generators.
  Proposal-free methods focus on directly locating the referent target, and have great potential in terms of accuracy and reasoning speed.

3.2 Visual Transformer

  Transformer has been very popular in target detection and image segmentation recently. The DETR series transforms visual feature maps into a set of tokens to achieve SOTA.

4. Method

4.1 Basic visual and text encoders

  Given an image and a reference expression pair $(I,E)$ , the visual localization task aims to use the Bounding box to locate the target instance described by the target expression.
  First resize the image to$w\times h$ , and then sent to the ResNet backbone to extract the image feature map$F\in {\mathbb{R}}^{\frac{w}{s}\times \frac{h}{s}\times d}$ , where$s$ is the step size of backbone output,$d$ is the number of channels. Then the visual feature map$F$ is transformed into visual tokensX v = { vi } i = 1 T v X_v=\{ { { v}_i}\}_{i=1}^{T_v}Xv​={ vi​}i=1Tv​​，其中 $T_v=\frac{w}{s}\times \frac{h}{s}$is the number of tokens, $v_i$The dimensions are $d$ .
  Use RNN-based soft-parser to extract text tokens: for the given expressionE = { et } t = 1 TE=\{e_t\}^{T}_{t=1}E={ et​}t=1T​, where $T$ represents the number of words. First, use the learnable embedding layer, that is,$u_t=\text{Embedding}(e_t)$ will each word$e_t$Convert to vector $u_t$. Bidirectional LSTM is then applied to encode the context of each word. Then calculate the $k$ text tokens at$t$ 个单词上的注意力：
$$\begin{array}{rl}{h}_t& =Bi-LSTM(u_t,h_{t-1})\\ a_{k,t}& =\frac{\exp (f_k^T{h}_t)}{{\sum }_{i=1}^T\exp (f_k^T{h}_i)}\end{array}$$After $k$ text tokens are defined as the sum of weights of word embedding:
$${\mathbf{l}}_k=\sum _{t=1}^T{a}_{k,t}{\mathbf{u}}_t$$The final text tokens are represented as X l = { lk } k = 1 T lXl​={ lk​}k=1Tl​​, where $T_l$is the number of tokens, $l_k$The dimensions are $d$。

4.2 Positioning encoder

  The localization encoder consists of stacked $Composed\; of\; N$ independent layers, each layer contains two independent visual + language branches for processing visual and text tokens. Like the Transformer layer, each branch contains three sub-layers: Norm Layer, multi-head self-attention layer, and fully connected forward layer FFN.

Self-attention text branch

  Given queries $q_l$, from ii .Text tokens of layer$i$$X_l^i$Get keys $k_l$ 和 values $v_l$, the output of the text self-attention layer is:
$$T-Attn({\mathbf{q}}_l,k_l,v_l)=\mathrm{softmax}\left(\frac{q_l{k}_l^T}{\sqrt{d}}\right)\cdot v_l$$Then apply FFN, defined as ${\text{FFN}}_l$Get text features $X_l^{i+1}$：
$$X_l^{i+1}={\text{FFN}}_l\text{ (T-Attn}(q_l,k_l,v_l))$$

Text-guided visual branch of self-attention

  The structure of the visual branch is similar to the text branch, but there is an additional component called text-guided self-attention, which aims to extract salient visual features guided by text descriptions. Specifically, given queries $q_v$, from section iiVisual tokensX vi X_v^$i\; of\; layer\; i$$X_v^i$Get keys $k_v$ 和 values $v_v$. Next, the text feature $X_l^{i+1}$Supplement visual queries as additional guidance information. For smooth implementation, use specific tokens to pair text tokens $X_l^{i+1}$Sum the weights and add visual queries $q_v$. where the weights pass $q_v$ 和 $X_l^{i+1}$ 的点乘得到：
$$\begin{gathered} \mathrm{V}-\mathrm{Attn}({\hat{q}}_v,{\mathbf{k}}_v,{\mathbf{v}}_v)=\mathrm{softmax}\left(\frac{{\hat{q}}_v{k}_v^T}{\sqrt{d}}\right)\cdot v_v \\ \begin{array}{rl}{\hat{q}}_v& =q_v+softmax\left(\frac{q_v(X_l^{i+1}{)}^T}{\sqrt{d}}\right)\cdot X_l^{i+1}\end{array} \end{gathered}$$Similarly, apply FFN, expressed as ${\text{FFN}}_v$Get visual tokens $X_v^{i+1}$：
$$X_v^{i+1}={FFN}_v(\mathrm{V}-\mathrm{Attn}({\hat{q}}_v,k_v,v_v))$$
The following figure is a schematic diagram of cross-modal attention and the text-guided attention mechanism proposed in this article:

  Queries of a typical multi-modal self-attention mechanism originate from one modality, while Keys and Values ​​originate from another modality to perform standard self-attention operations, similar to the self-attention operations in the Transformer decoder. However, integrating text information into image features in this way may damage positioning capabilities, so this paper proposes to guide visual features through text tokens to achieve higher performance.

4.3 Positioning decoder

  The decoder is also composed of $It\; is\; composed\; of\; N$ independent layers stacked. Each layer has 4 sub-layers: Norm Layer, positioning query self-attention layer, grounding query self-attention layer, encoder-decoder self-attention layer, and fully connected feed-forward (FFN) layer.
  The input of the positioning decoder is the modified text tokens$X_l^N$, followed by positioning queries $G$ services, in addition to visual tokens$X_v^N$Participation. Under the guidance of positioning queries, positioning query self-attention, and encoder-decoder self-attention mechanisms, text-guided visual features are decoded.

Positioning query self-attention

  Given queries $q_g$, from ii .Positioning queries for layer$i$$G^{}$Obtain keyskg k_g${}^{from\; i}$$k_g$ 和 values $v_g$. Then apply the standard self-attention mechanism to perform query enhancement:
$$\text{G-Attn}(q_g,k_g,v_g)=\text{softmax}\left(\frac{q_g{k}_g^T}{\sqrt{d}}\right)\cdot v_g$$The modified positioning queries are then obtained through layer normalization (LN):
$$G^{i+1}=LN(G-Attn(q_g,k_g,v_g))$$

Encoder-decoder self-attention

  The encoder-decoder self-attention will locate queries $G^{i+1}$ is treated as queries$f_g^q$, from the encoded visual tokens $X_v^N$Get keys $f_v^k$ 和 values $f_v^v$As input, output the extracted text-related features:
$$ED-Attn(q_g,k_v,v_v)=\mathrm{softmax}\left(\frac{q_g{k}_v^T}{\sqrt{d}}\right)\cdot v_v$$Finally, FFN is applied, denoted as ${\text{FFN}}_{ed}$Get the final embeddings $Z$：
$$Z={\text{FFN}}_{ed}\left(\text{ED-Attn}\right(q_g,k_v,v_v))$$

4.4 Prediction head and training target

  This paper treats the referential target positioning task as a bounding box coordinate regression problem. Get the transformed embedding from the positioning decoder Z = { zi } i = 1 K ∈ RK × d Z=\{z_{\boldsymbol{i}}\}_{i=1}^{K}\in\ mathbb{R}^{K\times d}Z={ zi​}i=1K​∈RK × d , then concatenate all transformed vectors, and then use the prediction head to return the coordinates of the center point and the width and height of the target. The prediction head consists of two fully connected layers followed by a ReLU activation layer.
  The training objectives are L1 loss and general IoU (GIoU) ​​loss${\mathcal{L}}_{iou}(\cdot )$ L oss = λ L 1 ∣ ∣ b − b ^ ∣ ∣ 1 + λ L iou L iou ( b , b ^ ) Loss=\lambda_{L_1}||b-\hat{b} ;
$$Loss=l_{L_1}||b-\hat{b}|{|}_1+l_{L_{iou}}{L}_{iou}(b,\hat{b})$$where$\hat{b}$ represents the bounding box of the predicted target,$b$ is GT,$l_{L_1}$、$l_{L_{iou}}\in R$ is a hyperparameter that balances the two losses.

5. Experiment

5.1 Dataset

  Flickr30k Entities、RefCOCO/RefCOCO+/RefCOCOg。

5.2 Implementation details

Hyperparameter settings

  Input image size $512\times 512$ , the maximum sentence length is 20, and the output step size of Backbone is$s=32$ . For all datasets, 4 text tokens are extracted. The number of heads in multi-head attention is 8, and the hidden layer size$d=256$ , the number of layers of VGTR$N=2$，$l_{L_1}=5$ ,$l_{L_{iou}}=2$.

Training and evaluation details

  AdamW optimizer, initial learning rate $1e-4$ , weight decay$1e-5.$ CNN backbone: ResNet50/101. The initialization weights adopt the weights pre-trained on MSCOCO data. A total of 120 epochs are trained. At the 70th and 90th epochs, the learning rate decreases by$10\%$ . Using [email protected] as the evaluation index,

5.3 Comparison with SOTA method

  

5.4 Ablation experiment

Contribution of each part

  

Effectiveness of text-guided self-attention

  

number of layers

  Same table 3.

5.5 Qualitative analysis

  

6. Conclusion

  This paper proposes a single-stage Transformer-based framework VGTR for visual positioning tasks, and experiments show that the method is effective.

write on the back

  This paper is relatively short, but it can be regarded as a good application and improvement of Transformer in 2021. Now it looks like the effect is not that explosive. Moreover, the author's writing skills in the paper also need to be improved. Many concepts are just clicked without going into depth. In addition, a part of the chapter is wasted on the Transformer structure, which is not appropriate.