Interpretation of Rotation Equivariant Networks for Tracking paper

1. Introduction

The task of visual object tracking with Siamese networks, referred as Siamese tracking, transforms the problem of tracking into similarity estimation between a template frame and sampled region from a candidate frame.

The twin network is a problem of describing the tracking task as a similarity response between the template and the search region

Although Siamese trackers are generally shown to work well, they are prone to failure under challenges such as partial occlusion、scale change or when one of the two inputs is rotated

Although Siamese performs well, it is prone to tracking failures during occlusions, scale changes, and rotations

The CNN archietectures used in Siamese trackers are not inherently equivariant to in-plane rotations of the target. The implication is that the model may perform well on object orientations that are represented in the training set, but may fail on other previously unseen orientations

The CNN framework in Siamese does not have in-plane rotation equivariance in nature, which means that the model will perform well in the target directions represented in the training set, but in other directions that have not been generated, the model performance will fail.

A straightforward approach to enforce learning of rotated variants is to use training dataset where in-plane rotations occur naturally or through data augmentation

A straightforward way to force the model to learn to rotate variables is to use datasets with natural rotation information or via data augmentation

Limitations of Data-Augmentation

1. Such procedures would require learning separate representations for different rotated variants of the data

This will allow the model to learn the expression of different rotation variables of the data

2. The more variations are considered, the more flexible tracker model needs to be to capture them all

The more variables to consider, the more flexible the model needs to be to capture more variables

3. Futher, such an approach would make the model invariant to rotations, thus making the predictions unreliable when the target is surrounded by similar objects, e.g.,tracking a fish in a school of fishes.

Also, this approach makes the model rotation invariant, thus making predictions unreliable, such as finding a fish in a school of fish

rotation inequalities

Example demonstrating rotation non-equivariance in regular CNN models used in object tracking:

The example describes the conventional CNN model in object tracking, which does not have rotation equivariance
$\psi_\theta(f(/cdot)) \neq f(\psi_\theta(\cdot))$

isotropic

Equivariant and other denaturation:

Operators and functions can be interchanged, and there is commutability
$t r an s f or m [F (x)] = F (t r an s f or m [x])$
Invariant invariance:

the input x is transformed, but the output after F is unchanged
$F (x) = F (t r an s f or m [x])$
Covariant covariance:

the input x is transformed into transform, and the output after F is also transformed, but it is not transform, but another transformation can be used to make the result the same
$transform^*F(x) = F(transform[x])$

2. Related Work

Equivariant CNNs

SiamRPN++ proposed a training strategy which removes the spatial bias introduced in non fully-convolutional backbone

SiamRPN++ proposes a training strategy that removes the spatial bias in the backbone

Deeper and wider siamese networks for real-time visual tracking showed that existing tracking models induce positional bias, which breaks strict translation equivariance

Deeper and wider siamese networks for real-time visual tracking pointed out that existing tracking models cause position bias and break the equivariant transformation

Scale Equivariance Improves Siamese Tracking(SE-SiamNet) introduced scale-equivariant Siamese trackers which is crucial when the camera zooms its lens or when the target moves into depth

Scale Equivariance Improves Siamese Tracking (SE-SiamNet) introduces a scale equivariance twin network, which has a huge impact when the camera zooms in or the target moves in the depth of field

3. Rotation Equivariant CNNs

Rotational background

Rotation Equivariance

SFC-NNs

Learning steerable filters for rotation equivariant cnns indicated that one of the more robust ways of enforcing rotation equivariance in CNNs is through the use of steerable filter（SFC-NNs）

Learning steerable filters for rotation equivariant cnns pointed out that a relatively Lubang way to make CNNs have rotation equivariant is to use steerable filters (SFC-NNs)

For rotation equivariance with steerable filters, the network must perform convolutions with different rotated versions of each filter

Using the rotation equivariance of the controllable filter requires that each convolution filter of the network corresponds to a different rotation

Steerable filters not only facilitate efficiently computing responses for an arbitrary number of discrete filter rotations, but they also exhibit strong expressive power as well

Steerable filters not only make it more efficient to compute the rotational response of any number of discrete filters, they are also powerful

Knowledge Expansion: Spherical Harmonics

How to use the spherical harmonic function in the paper

Spherical coordinates without $\theta$ are the circular harmonic function system

$\\ \psi_{jk}(r,\varphi) = \tau_j(r)^{jk\varphi}\\ \qquad \\$

The following two parameters control the bias function (radial function $\tau_j$ ) offset range

$\qquad$
$\varphi \in (-\pi,\pi]$

Current times $j=1,2,\dots,J$

$\qquad$

Control pole sits $x_1,x_2)$ mark rotation angle

$\qquad$
$(r,\phi)$

$\qquad$

Angular function $(e^{jk\varphi})$ , also becomes the order

$\qquad$
$\in Z Its value is related to the current function degree j of the function system Z \in [-j,j]$

$\qquad$

Using Euler's rotation theorem to express the rotation of the target

$\qquad \\ \rho_{\theta}\psi_{jk}(x) = e^{-ik\theta}\psi_{jk}( x) \\$
$e^{-ik\theta} means clockwise rotation \theta, e^{+ik\theta} means counterclockwise rotation \theta$

$\qquad$
$\psi_{jk}(x) here refers to \psi_{jk}(\cdot) , x refers generally, not specifically$

$\qquad$

$w_{jk} \in \mathbb{C} is constructed as a linear connection between basic filters linear connection$

$\qquad \\ \Psi(x) = \sum_{j=1}^{J}\sum_{k=0}^{K}w_{jk}\psi_{jk}(x) \\ \qquad \\$

$\theta angle, the control of the synthesis filter can be realized by the phase control of the basic filter$

$\quad \\ \rho_{\theta}\Psi(x) = \sum_{j=1}^ {J}\sum_{k=0}^{K}w_{jk}e^{-ik\theta}\psi_{jk}(x) \\\quad\\$
$Re\Psi ( x ) A rotation direction of the filter can be obtained through the real part of \Psi, called Re\Psi(x)$

$\qquad$

$\qquad$

4. Rotation Equivariant Siamese Trackers

$\qquad$

4.1 Formulation Based on Siam-FC

Author started from and modified the basic SiamFC model due to its simple design.

The author chose to modify based on SiamFC because it is simple

$\qquad \\ h(z,x)=f(z)*f(x) \\ \qquad \\$

$network\qquad f(\cdot) refers to the feature extraction network$

$cross-correlation\qquad * refers to the convolution operation of cross-correlation$

For rotational Siamese tracker, author introduced rotation equivariant modules and a group max pooling module that selects the cross-correlation encoding for the most approximate orientations among the multiple heatmaps generated in setup

The author introduces the rotation equivariant module and group max pooling, and group max pooling is used to select the cross-correlation encoding of the most approximate direction from the generated heat maps

network model

The Candidate Head of the network (processing the Search region) uses a search image (unchanged)

$\qquad$
The Template Head of the network is modified so that multiple template images (as shown in the figure, the rotated template) can be input as input, and a series of rotation variables $\Lambda$ definition is Z collection, in which $Z=\{z_{1}, z_{2},\dots, z_{\Lambda}\}$ , that is, all possible rotation angles

$\qquad$
of the initial traget $f (z)$ , then rotate $f (z)$ , since it is a rotational equivariant network, it is theoretically possible to do this

$\qquad$
Rotate the Target in the Template:
$\qquad$

$\qquad \\ y_{\tilde{c}}^{(1)}(x,\theta) = Re \sum_{c=1}^{C}\sum_{j=1}^{J}\sum_{k=0}^{K}w_{\hat{c}cjk}e^{-ik\theta}(I_c * \psi_{jk})(x) \\ \qquad \\$
in

$I_c is a picture with channel c, c \in \{ 1, 2, \dots, C\}$

$filter\rho_{\theta}\Psi_{\hat{c}c}^{(1)}rotational filter$

$\hat{c} \in \{1, 2,\dots, \hat{C} \}$

$\theta can be obtained by the set \Theta=\{0, \Lambda, \dots, 2\pi \ frac{\Lambda-1}{\Lambda}\}$

$\beta_{\hat{c}}^{(1)} is used in the layer (First layer): \zeta_{\hat{c}}^{(1)}Get the feature map$

$\sigma_{\hat{c}}^{(1)} is used in the layer (First layer): \zeta_{\hat{c}}^{(1)}Get the feature map$

rotationally equivariant convolution

$\qquad \\ y_{\hat{c}}^{(l)} = Re\sum_{c=1}^{C}\sum_{\phi \in \Theta}\sum_{j,k}w_{\hat {c}cjk,\theta - \phi}\hspace{1mm}e^{-ik\theta}(\zeta_c^{l-1}(\dot, \phi)*\psi_{jk})(x) \\ \qquad \\$
Among them,

$\theta-\phi in the weight item w refers to the group convolution operation performed in the angle dimension$ $\qquad$ $in the angle dimension$

Rotationally equivariant pooling

$\qquad$
The output of the last grouped volume layer is further processed in the rotated dimension. Unlike the traditional classification network, this pooling is not performed on the dimension of W\times H (spatial dimension), but on the angle grouping ( $\ {0, \frac{2\pi}{8}, \frac{4\pi}{8}, \dots, \frac{14\pi}{8} \}$ ) dimension to preserve the features of rotation equivariance

$\qquad$

Cross-correlation of rotational equivariance

$\qquad$

$Re-SiamNet\{\phi( z) and \phi(x)\}$

$\qquad$

$Λ\phi(z) is the feature-map set of rotation angle\Lambda$

$\qquad$

$layer\{ \hat{h}(z,x)\}, calculate the heat map of the Template feature map of different rotation angles\Lambda, h_i(z, x)=\phi(z)*\phi(x)$

$\qquad$

$\{\hat {h}(z, x)\} undergoes global maximum pooling, and outputs a heat map h(Z,x), that is, picks out the largest \hat in \{\hat{h}(z,x)\} {h}$

$\qquad$

$\qquad$

4.2 Constructing RE-SiamNet Framework

Identify the precision of the tracker in terms of discriminating between orientations of the rotational degree of freedom. Author considered here $\Lambda$ rotation groups, based on which RE-SiamNets would be perfectly equivariant to angles defined by the set $\Theta=\{\frac{(i-1)}{\Lambda}*2\pi\}_{i=1}^{\Lambda} \Rightarrow \{(i-1)\frac{2\pi}{8}\}_{i=1}^{\Lambda=8}$

Differentiate the accuracy of the tracker in terms of the difference between different rotation angles. The author here uses a set of arithmetic difference angle sets, as shown in the formula

Define the non-parametric encoding $\phi(\cdot)$ based on existing Siamese trackers. Based on the choice of $\phi(\cdot)$ ，discriminative power of trackers varies.

Define a parameterless encoder based on the existing Siamese tracker. The discriminative power of the tracker will change based on the selection of these encoders

Replace all the convolutional layers of $\phi(\cdot)$ with the rotation-equivariant modules.

Use the rotation equivariant module to cancel the convolution module in SiamFC

Here e2CNN is used to achieve rotation

Instead of a single convolution to generate $h=(z,x)，\Lambda$ convolutions are performed to generate $\Lambda$ different heatmap

8 convolutions generate 8 different heatmaps, canceling the single heatmap generated by a single convolution

Perform Global max-pooling over the feature maps to generate $h (Z, x)$ , which is then processed to localize the target.

The global maximum pooling performed on the generated 8 sets of feature maps will be sent to the head processing for target positioning.

$\qquad\$

5. Unsupervised Relative Rotation Estimation

$\qquad$

5.1 Unsupervised 2D pose estimation

$\qquad$

The inherent design of RE-SiamNet allows to obtain an estimation of the relative changes of 2D pose of the target in a fully unsupervised manner. This information can be obtained from the result of the group maxpooling step

The design of RE-SiamNet inherently has the ability to estimate the 2D pose-related changes of objects in an unsupervised manner. This information can be obtained by group max pooling

Let $\in \{1,2,\dots, \Gamma\}$ denote one of $\Lambda$ orientations of the template. Then, $i$ is the number of rotation groups by which the pose of the template differs from that of its appearance in the candidate image if :
$h(Z,x)=\hat{h}(z_i, x)=group-maxpool(\{z, x\})$

Let i be one of the 8 template rotations, which refers to the number of rotations, a single 45 degrees

$\qquad$