BEVSimDet：Simulated Multi-modal Distillation in Bird’s-Eye View for Multi-view 3D Object Detection

Reference code: BEVSimDet

1 Overview

Introduction: During the actual deployment of the model, due to factors such as the lack of actual sensors and the limitation of computing resources, the actual deployment of the model is cut, and the natural performance will also decline. For such a situation, knowledge distillation is generally used to achieve performance compensation. For the common intra-modal, cross-modal, and multi-modal distillation methods, they need to have the same number of sensors input in the source and target, so as to achieve distillation. Specifically, lidar is not used in all models in autonomous driving scenarios, so how to improve the perception performance of cars without lidar? In this regard, the article proposes to add a simulated Lidar feature to the image feature to realize the migration of part of the Lidar information, so as to compensate for the performance impact caused by the lack of Lidar.

2. Method implementation

The method of this article is based on the BEVFusion method, so it involves the fusion of Lidar and Camera, but this is in the source, and Lidar does not exist in the target. In this regard, this article proposes to predict a feature similar to the Lidar feature from the image feature to replace the Lidar. The structure of the distillation is shown in the figure below:

For knowledge distillation, the difference between the source and the target is relatively large, so all knowledge transfer is in the BEV space. Completed, that is, on the corresponding CMD, IMD, MMD-F, and MMD-P above. First, perform knowledge transfer for the features that exist in both source and target, that is, IMD, MMD-F, and MMD-P.

IMD:
Image features get BEV features after view-trans, then add constraints to the corresponding regions in source and target:
$$L_{IMD}=MSE(F_{C_{bev}}^T,F_{C_{bev}}^S)$$

MMD-F:
This part is distilled after the fusion of Lidar and Camera features. The constraints used are:

$$L_{MMD-F}=MSE(F_{U_{bev}}^T,F_{U_{bev}}^S)$$

MMD-P:
This part is to distinguish the bbox prediction results in the prediction header ( $P_B$) and cls features ( $P_C$) for distillation, and for the cls feature to be a continuous value between [0,1], then Quality Focal Loss (QFL) is used here for adaptation to balance the data distribution, and the constraint is: LMMD − P
$$L_{MMD-P}=SmoothL1(P_B^T,P_B^S)\cdot s+QFL(P_C^T,P_C^S)\cdot s$$
where$s$ is the loss weighting coefficient determined by the IoU of bbox and GT.

CMD:
Since there is no Lidar sensor in the target, it is obtained by adding branches to the image features for prediction, and the generation of some features uses the Deformable Attention Layer (it has also been verified by experiments that it is better than DeformConv) to build Geometry Compensation Module (GCM ), used to generate a feature expression with a stronger ability to express geometric information, the generated Lidar feature description is:
$$F_{L_{bev}}^S=GC{M}_{bev}\left(\rho \right(Y(GC{M}_{uv}(F_{C_{uv}}^S))\times \Theta (GC{M}_{uv}(F_{C_{uv}}^S)))$$

Here record $r,Y,\Theta$ represents bev_pool operation, context feature generation, and depth feature expression, respectively. After obtaining the virtual Lidar data, it needs to be aligned with the Lidar features, refer to the pair$({\bar{x}}_i,{\bar{y}}_i)$ as the center to draw a 2D Gaussian distribution:
$$H(x,y)=\sum _i^{H}exp(-\frac{(x-{\bar{x}}_i{)}^2+(y-{\bar{y}}_i{)}^2}{2p^2})$$
among them，$p=max(f(h,w),t),t=2$ , also refer to the settings in CenterPoint. For the bbox to project in the BEV space to obtain the binary mask$B$：
$$M_o=B\odot H$$
then uses this mask to distill Lidar features (focus on the foreground area):
$$L_{CMD}=M_o\odot MSE(F_{L_{bev}}^T,F_{L_{bev}}^S)$$

The actual performance impact of these modules:

3. Experimental results

Effectiveness of the distillation method: