[Paper Sharing] VOLO: Vision Outlooker for Visual Recognition

VOLO: Vision Outlooker for Visual Recognition

Overview

Visual transformers (ViTs) have been extensively explored in the field of visual recognition. Due to the lower efficiency of encoding fine features, ViT still performs worse than state-of-the-art CNNs when trained from scratch on medium-sized datasets like ImageNet.

Through experimental analysis, the author found that:
1) Simple tokenization of input images fails to model important local structures such as edges and lines, resulting in low efficiency of training samples; 2
) ViT's redundant attention backbone design leads to fixed calculations Budget and limited training samples have limited feature richness.

To overcome these limitations, the authors propose a new simple general architecture called Vision Outlooker (VOLO), which implements a novel Outlook attention operation to dynamically perform local features on the input image in a sliding window manner. aggregation mechanism. Unlike self-attention, which focuses on modeling the global dependencies of local features at a coarse level, the foreground attention proposed by the authors aims to encode finer features, which are crucial for recognition but ignored by self-attention. Outlook Attention breaks the bottleneck of self-attention, and its computational cost scales quadratically with the input space dimension, making it more memory efficient.

In other words: the author proposes a new Vision Outlooker module that only uses pixel space adjacent information to generate attention weights.

method

Outlooker uses an outlook attention layer to encode spatial information, and then uses an MLP to exchange information between different channels.

Outlook attention:
1) The features at each spatial location are sufficiently representative to generate attention weights for locally aggregating their neighboring features; 2
) Dense local spatial aggregation can effectively encode fine-level information.

Let ${\mathbf{V}}_{D_{i,j}}\in {\mathbb{R}}^{C\times K^2}$ represents all values ​​within the local window centered on (i,j), that is,
V Δ i , j = { V i + p − ⌊ K 2 ⌋ , j + q − ⌊ K 2 ⌋ } , 0 ≤ p , q < K . \begin{equation*} \mathbf {V}_{\Delta _{i,j}}=\lbrace \mathbf {V}_{i+p-\lfloor \frac{K}{2} \rfloor,j+q-\lfloor \frac{K}{2} \rfloor }\rbrace, \quad 0 \leq p,q < K. \tag{1} \end{equation*}VDi,j​​={ Vi+p−⌊2K​⌋,j+q−⌊2K​⌋​},0≤p,q<K.​(1)​

The foreground weight at position (i,j) is directly used as the attention weight for value aggregation, reshaping it into ${\hat{\mathbf{A}}}_{i,j}\in {\mathbb{R}}^{K^2\times K^2}$ , followed by the Softmax function.

$$\begin{array}{c}{\mathbf{Y}}_{D_{i,j}}=\mathrm{MatMul}(\mathrm{Softmax}({\hat{A}}_{i,j}),{\mathbf{V}}_{D_{i,j}}).\end{array}\text{\hspace{1em}(2)}$$

Outlook attention-intensively aggregates predictive value representations. Adding different weight values ​​at the same position from different local windows gives the output
$$\begin{array}{c}{\tilde{\mathbf{Y}}}_{i,j}=\sum _{0\le m,n<K}{Y}_{D_{i+m-\lfloor \frac{K}{2}\rfloor ,j+n-\lfloor \frac{K}{2}\rfloor }}^{i,j}.\end{array}\text{\hspace{1em}(3)}$$

Our foreground attention inherits the advantages of convolution and self-attention. It has the following advantages.

• First, outlook attention encodes spatial information by measuring the similarity between pairs of token representations, which is more parameter efficient than convolution for feature learning, as studied in previous work [57], [66] .
• Secondly, foreground attention adopts a sliding window mechanism to fine-grained local encoding of marker representations and preserves key location information for visual tasks to a certain extent [42], [71].
• Third, the generation of attention weights is simple and efficient. Unlike self-attention, which relies on query key matrix multiplication, our foreground weights can be directly generated by a simple reshape operation, thus saving computation. To see this, we compare the

$$\begin{array}{rl}\text{M-Adds}(\mathbf{SA})& \approx 4HW{C}^2+2(HW{)}^{2C}\_\\ \text{M-Adds}(\mathbf{LSA})& \approx 4HW{C}^2+2HW{K}^{2C}\_\\ \text{M-Adds}(\mathbf{OA})& \approx HWC(\text{2}C+N{K}^4)+HWK\_{\_}^{2C}\end{array}\text{\hspace{1em}(4)(5)(6)}$$

Network Architecture

The overall network architecture of VOLO architecture. The image is first sent to the convolutional stem for patch embedding. The main body of our VOLO consists of two phases, consisting of the Outlooker block proposed in Phase I and the Transformer block in Phase II. Outlooker is responsible for the fine-level feature encoding. More detailed architectural information can be found in Table 2.

experiment

Training settings

ablation experiment

Method comparison

Model performance evaluation

reference

@article{Yuan2022Sep,
author = {Yuan, Li and Hou, Qibin and Jiang, Zihang and Feng, Jiashi and Yan, Shuicheng},
title = { {VOLO: Vision Outlooker for Visual Recognition}},
journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
volume = {45},
number = {5},
pages = {6575–6586},
year = {2022},
month = sep,
urldate = {2023-08-24},
issn = {1939-3539},
publisher = {IEEE},
language = {english},
doi = {10.1109/TPAMI.2022.3206108}
}