R2D2 (NIPS 2019) Feature Point Detection Paper Notes

1 Introduction

Thesis address
code address

  The article is basically optimized on D2-Net . The author puts forward the two attributes of Repeatable (repeatability) and Reliable (reliability) corresponding to R2. My personal understanding of repeatability is that the feature points remain unchanged under various shooting angles, light, and seasons. It can be detected, and the reliability is that the descriptor can be matched correctly. The authors argue that previous approaches focused on repeatability but neglected reliability.

Figure 1: Toy examples to illustrate the key difference between repeatability (2nd column) and reliability (3rd column) for a given image. Repeatable regions in the first image are only located near the black triangle, however, all patches containing it are equally reliable. In contrast, all squares in the checkerboard pattern are salient hence repeatable, but are not discriminative due to self-similarity.

  A small experiment in the paper, the key point detection in the middle of the two sets of images corresponds to the repeatability, such as the first set around the triangle, the second set of squares or corners of the chessboard; the reliability of the descriptor on the right corresponds to the reliability , the first group of patches containing triangles is also reliable (the receptive field corresponding to the feature contains this triangle, so the features can be matched correctly), and the second group has no difference due to similarity (the pattern of the receptive field corresponding to the feature is very similar, so The features are all the same and cannot be matched correctly).
  The author believes that in natural images, common textures, such as leaves, windows of buildings, waves, etc., have prominent feature points, but are difficult to match because of repetition and instability. Repeatable regions are not necessarily discriminative (repeatable regions are not necessarily discriminative), so suboptimal key points can be selected.

  In fact, I feel that the strength of this experiment is not enough. It is best to show that the best point matching is not as good as the second advantage on natural images.

2. Method

2.1 Network structure

  The network has three outputs:
(1) $\mathbf{X}\in {\mathbb{R}}^{H\times W\times D}$ corresponds to the descriptor
(2)$\mathbf{S}\in [0,1{]}^{H\times W}$ corresponds to the position of the feature point (repeatability, the latter is named the detection score)
(3)$\mathbf{R}\in [0,1{]}^{H\times W}$ corresponds to the reliability of the descriptor (the latter is called the feature score)

  The backbone network is L2-Net, and two modifications have been made:
(1) The downsampling is replaced by expanded convolution, and the feature maps of each stage maintain the resolution of the original image
(2) The last $8\times The\; convolution\; of\; 8$ uses 3$2\times 2$ Convolutional Alternatives

  The output 128-dimensional feature map
(1) of the backbone network is normalized by L2 to obtain the descriptor $X$
(2) undergoes a square operation,$1\times 1$ convolution, softmax to get$S$
(3) After the same operation as (2), get$R$

2.2 Repeatability (assay score)

  The authors show that some supervised training (Lift, SuperPoint ) cannot solve the repeatability problem, because they all imitate existing detectors, rather than discovering potentially better keypoints.
  pair image $I$ do a homography transformation to get$I^{\prime }$ , so that there is an accurate correspondence of each pixel$U\in {\mathbb{R}}^{H\times W\times 2}$，$U_{ij}=\left(i^{\prime },j^{\prime }\right)$RepresentativeIIPixel in$I$$(i,j)$ given$I^{}$Pixels in${}^{\prime }$$\left(i^{\prime },j^{\prime }\right)$correspond;$I,I^{\prime }$ detection score is$S,S^{\prime }$，$S^{\text{'}}$ A$U$ is transformed to get$S_U^{\prime }$.
  The goal is to make $S$ 和 $S_U^{\prime }$The local maximum value pair of , such as directly maximizing the cosine similarity between the two, but in practice, it is unrealistic to use this method because of various occlusions, artifacts, etc. Therefore, choose to calculate the mean value of the cosine similarity of a local area, and the size of the detection score map is $N\times$Overlapping patches of$N$ as a set P = { p } \mathcal{P}=\{p\}P={ p}
$${\mathcal{L}}_{cosi}\left(I,I^{\prime },U\right)=1-\frac{1}{|\mathcal{P}|}\sum _{p\in \mathcal{P}}cosim\left(\mathbf{S}[p],{\mathbf{S}}_U^{\prime }[p]\right)$$

  为了不让得分变得全都一样，用了一个损失来最大化局部峰值：
$${\mathcal{L}}_{\text{peaky }}(I)=1-\frac{1}{|\mathcal{P}|}\sum _{p\in \mathcal{P}}\left(\underset{(i,j)\in p}{\max }{\mathbf{S}}_{ij}-{\mathrm{mean}}_{(i,j)\in p}{\mathbf{S}}_{ij}\right)$$

  最终的检测得分损失：
$${\mathcal{L}}_{\text{rep }}\left(I,I^{\prime },U\right)={\mathcal{L}}_{\text{cosim }}\left(I,I^{\prime },U\right)+\frac{1}{2}\left({\mathcal{L}}_{\text{peaky }}(I)+{\mathcal{L}}_{\text{peaky }}\left(I^{\prime }\right)\right)$$

  $N$ 不同的取值效果如下：

2.3 可靠性（特征得分）

  这里论文说到了，特征得分 $R$ 的作用是提高描述符之间的区别，或者避免在天空或地面这种平坦区域检测出特征点。
  $I$ 中每一个像素 $(i,j)$ 都是一个 $M\times M$ 的 patch $p_{ij}$ 的中心，其描述符为 ${\mathbf{X}}_{ij}$，可以和 $I^{\prime }$ 所有 patches 的描述符 { X u v ′ } \{\boldsymbol{X}^\prime_{uv}\} { Xuv′​} 作比较。已知 $U$，所以可以用 AP（平均精度）来估计 $p_{ij}$ 的可靠性。按一篇论文对 AP 进行优化得到一个可求导的近似，记作 $\widetilde{\mathrm{A}\mathrm{P}}$。$B$ 代表一个 batch 中 patch 的数量。
$${\mathcal{L}}_{AP}=\frac{1}{B}\sum _{ij}1-\widetilde{\mathrm{A}\mathrm{P}}\left(p_{ij}\right)$$

  虽然每个像素都提取了描述符，但并不是每个都适合作为特征点，通常平滑区域不太好，但事实上纹理丰富的地方也不一定好，所以修改了下 loss，让网络关注次优的区域。
$${\mathcal{L}}_{AP,\mathbf{R}}=\frac{1}{B}\sum _{ij}1-\widetilde{\mathrm{A}\mathrm{P}}\left(p_{ij}\right){\mathbf{R}}_{ij}+\kappa \left(1-{\mathbf{R}}_{ij}\right)$$

  $\kappa \in [0,1]$，代表是否可靠的 AP 阈值，取0.5效果较好。这里论文说了一堆分析也没太看明白，个人从这个 loss 公式分析，为了让 loss 降低，其实每个点都会想让 $\widetilde{\mathrm{A}\mathrm{P}}$ 和 $\mathbf{R}$ 升高，这里 $\widetilde{\mathrm{A}\mathrm{P}}$ 的计算方式不看代码也不清楚细节，就当做匹配的效果或者说是匹配的难易程度。当 $\widetilde{\mathrm{A}\mathrm{P}}$ 较高，意味着特征明显、容易匹配、匹配的很正确，那么网络就会更倾向于拉高这个点的 $\mathbf{R}$，可靠性越高，符合逻辑。这就可以实现有些点的检测得分高，纹理特征强，但是因为匹配难度高，降低它的特征得分。

  （2022.1.21）写论文梳理时看了一下代码发现这边可靠性得分的损失应该是写错了，return 1 - ap*rel - (1-rel)*self.base，可以看出这边的加号应该是减，也解决了之前的困惑。这样也才更符合逻辑，当 $\widetilde{\mathrm{A}\mathrm{P}}$ 小于0.5时降低可靠性得分；假如是加号，训练会很麻烦，只能训练到一半，依赖梯度下降速度得到相对可靠的得分，彻底训练的话得分应该都是1。

3. 个人总结

(1) The loss of D2-Net is mainly based on feature matching, and a related item of detection score is added as a weight; while R2D2 specifically designs a loss for detection score, that is, the local score and the corresponding local score should be similar. Open local maxima and mean difference as regularization.
(2) The idea of ​​feature scoring is to use the actual feature matching effect to converge; if a region is smooth and has no features, the network cannot integrate unique features, and the matching effect is naturally not good, and the matching degree is low at this time; If an area is rich in texture, the network can be a characteristic feature, but there are too many places similar to this feature (leaves at the beginning, windows of buildings, waves, etc.), so the matching is relatively difficult, so it is not used as a feature Point; in this way, the network can obtain feature points that are both characteristic (rich in geometric information) and characteristic (low repeatability in the image, very distinctive and easy to match).