Multi-target tracking - [Transformer] MOTR: End-to-End Multiple-Object Tracking with TRansformer

Paper link: https://arxiv.org/abs/2105.03247

Article focus

1. Inspired by DERT , the Object Object in target detection$Object$ $Query$ migrated to multi-target tracking and constructed$Track$ $Query$ . Followingthe detection network inDERT$Object$ $Query$ greatly enhances the feature extraction ability of the target.
2. Many current detection-based tracking methods first obtain detection results, track the detection results to extract appearance and motion, and then go to data association. This method is an end-to-end tracing model,
3. In order to ensure the effectiveness of timing modeling, MOTR proposes a tracklet-aware label assignment [ tracklet-aware label assignment ] training strategy + joint average loss [ collective average loss ] to enhance the timing modeling of the model.

将 $Object$ $Query$ becomes$Track$ $Query$ The problem to be solved

Generally speaking, although target detection and target tracking are both in the CV field, since the fundamental tasks at the lower end are different, there must be problems in direct application, so careful design is required.

1. Use a $Track$ $Query$ tracks the same target. Becausethe Object ObjectinDERT$Object$ $Query$ is based on each frame recognition, each target and$Object$ $Query$ does not have a corresponding relationship, as shown in Figure (a) below. However, multi-target tracking needs to generate tracking trajectories for each target in the sequence, which inevitably requires the consistency of target trajectories, and the problem of ID Switch cannot occur. This means that target detection + target track matching must use$Track$ $Query$ to achieve, this is the essence of end-to-end, cancel the post-processing. This paper introduces a tracklet-aware label assignment [tracklet-aware label assignment] training strategy, so that the bounding box with the same ID can be used to supervise the process of training detection + matching.
   
2. Handling of newly appearing and disappearing targets. Because a target suddenly disappears or appears suddenly in multi-target tracking, the fixed-length $Track$ $Query$ cannot meet actual needs. Therefore,this paper proposes two sets of variables—$Track$ $Query$ (variable length) and$Detect$ $Query$ (fixed length) to handle the appearance and disappearance of the target. As shown in Figure (b) above, T Track Trackmust be iteratively updated for each frame$Track$ $Query$ , change the disappearing target to its corresponding$Track$ $Query$ is deleted, and each frame uses$Detect$ $Query$ detects how many targets there are in the frame, and new targets pass$Detect$ $Query$ detects it and joins$Track$ Q u e r y Query In the collection of$Q$$u$$ery\; .$The specific process is shown in the figure below:
   

Overall network structure - timing fusion network

It can be seen that the structure analysis of the above figure is as follows:

1. Enc represents the feature extraction stage: Backbone network + Encoder of Deformable DERT ;
2. Dec represents the Decoder of Deformable DERT .
   • In the first frame, since the tracking target has not yet appeared, the input is a fixed-length $q_d$and qtr q_{tr} for the empty set$q_{tr}$, and the input of subsequent frames is $q_d$and the qtr q_{tr} passed in the previous frame$q_{tr}$。
   • The output is the intermediate state feature, which is used to generate the tracking prediction result and the input of the QIM .

QIM——Query Interactive Module

The role of this module is to handle things like the appearance and disappearance of objects. The scores in the figure represent the classification scores of Head's predicted tracked targets.

• Input: The intermediate state characteristics of the Decoder output, as shown in the leftmost input in the above figure. The yellow part means $q_d$, orange is $q_{tr}$。
• The first step: Input it and the classification score of the head prediction tracking target into the two branches of processing (a) target appearance and (b) target disappearance. Here, two thresholds are set as filters to filter valid queries.
• The second step: for (a) in the target occurrence branch, the detection target whose classification score is greater than the threshold is regarded as a new target.
• Step 3: For (b) target disappearing branch , when new $Track$ $Before\; Query$ , you have to go through[Timing Enhanced Network]TAN, which is essentially a self-attention mechanism. The input is the target query qtri q_{tr}^iof this frame$q_{tr}^i$, The intermediate state characteristics of the output of the first step (b) branch. This output is the tracking target for the next frame.
• Output: The output of the second step and the third step are concatenated into the tracking target $q_{tr}^{i+1}$。

training logic

Tracklet-Aware Label Assignment

【The purpose is for $Track$ $Query\; models\; the\; one-\; to-$ one relationship between trajectories and targets. 】
TALAhas two strategies, corresponding to$Detect$ $Query$ and$Track$ $Query$ 's training strategy

• For $Detect$ $Query$ : Follow the detection strategy in DERT to detect new targets that appear in each frame of the tracking sequence. The training strategy is for$Detect$ $Query$ performs two-way matching with the GroundTruth of the newly added target.
  

• For $Track$ $Query$ : This paper designs a training strategy with the same goal. T track Trackof this frame$Track$ $Query$ by T track Trackof the previous frame$Track$ $Query$+$Detect$ $Query$ . For the first frame,$Track$ $Query$ is an empty set.
  

Collective Average Loss

【The purpose is for $Track$ $Query$ models the frames before and after the transfer of timing information. ]
The usual training strategy is to calculate the loss of the frame, such a strategy ignores the motion information about the target that exists in the sequence. Therefore, this paper designs a joint average loss to predict the loss with video clip as the basic unit. Joint average loss = sum of (tracking loss of a single frame + detection loss of a single frame) / number of frames.

With strong classmates.