[Paper Reading Notes 16] BoT-SORT Paper Formula Derivation and Code Interpretation

---

Paper address: BoT-SORT: Robust Associations Multi-Pedestrian Tracking , like OCSORT (see OCSORT
for interpretation ), is an improvement on the Kalman filter. OCSORT is aimed at the problem that the Kalman prediction variance becomes larger when the observation (detector) is unreliable. The trajectory is smoothed. BoT-SORT is aimed at the problem of camera motion, and camera motion compensation is added. That is, in addition to using Kalman to predict the new position of the target, it also uses sparse optical flow (that is, to extract the picture outside the target The key points in to obtain the movement of the picture between two frames, and then compensate the result obtained by Kalman )

---

1. Kalman filtering

Kalman filtering follows the update step and prediction step as follows:

预测步:
$$\begin{gathered} {\hat{x}}_{t|t-1}=F_t{\hat{x}}_{t-1|t-1}+B_t{u}_t \\ {P}_{t|t-1}=F_t{P}_{t-1|t-1}{F}_t+Q \end{gathered}$$

更新步:
$$\begin{gathered} {\hat{x}}_{t|t}={\hat{x}}_{t|t-1}+K_t(z_t-H_t{\hat{x}}_{t|t-1}) \\ {K}_t=\frac{P_{t|t-1}{H}_t^T}{H_t^T{P}_{t|t-1}{H}_t+R_t} \\ {P}_{t|t}=P_{t|t-1}-K_t{H}_t{P}_{t|t-1} \end{gathered}$$

where $Q$, $R$ is the covariance matrix of input (observation) and state noise, respectively.

2. Selection of state variables

In SORT, the state variable is [center point, area, aspect ratio, speed, area change rate], that is, $x=[x_c,y_c,a,s,\dot{x_c},\dot{y_c},\dot{s}]$, since DeepSORT, the state variables basically use$x=[x_c,y_c,a,h,\dot{x_c},\dot{y_c},\dot{a},\dot{h}]$.

In BoT-SORT, the author simply changed the state variable to xywh and its derivatives, namely: $x=[x_c,y_c,w,h,\dot{x_c},\dot{y_c},\dot{w},\dot{h}]$, similarly, the input variable of the observer is$z=[z_{xc},z_{yc},z_w,z_h]$.

Paper (1) (2) formula

Paper (3) (4) formula, the definition of the noise covariance matrix is ​​not understood for the time being, and it will be supplemented after understanding

From the results of the ablation experiment, if the state variable MOTA is changed in this way, the IDF1 and HOTA indicators will be improved by about 0.1 to 0.2

3. Camera Motion Compensation

This is the core idea proposed by BoT-SORT.

Due to camera motion, the matching may be inaccurate. However, we lack prior information of camera motion (such as navigation, etc.), therefore, the image registration between two adjacent frames can approximate the camera motion in image two projection on the dimensional plane.

It can be understood here that the camera movement is equivalent to the transformation of the coordinate system, so we need to re-project the position of the target, the direction of movement, etc. into the new coordinate system, so we need to solve the transformation matrix of the coordinate system. As shown in the following figure :

Therefore, use the RANSAC algorithm to obtain the affine matrix of the plane coordinate change, and describe the 2-dimensional plane rotation change matrix as $M\in {\mathbb{R}}^{2\times 2}$ , the translation change vector is$T\in R^2$ , define the affine matrix
$$A=[MT]\in R^{2\times 3}$$

Essay (5)

Suppose the result after Kalman prediction is ${\hat{x}}_{t|t-1}=[x_c,y_c,w,h,\dot{x_c},\dot{y_c},\dot{w},\dot{h}{]}^T$ , we need to apply affine transformation to both the center point and the height and width. In order to realize it with a matrix multiplication, construct:

M k ∣ k − 1 ′ = d i a g { M , M , M , M } ∈ R 8 × 8 T k ∣ k − 1 ′ = [ T , 0 , 0 , 0 , 0 , 0 , 0 ] ∈ R 8 M'_{k|k-1}=diag\{M, M, M, M\}\in\mathbb{R}^{8\times 8}\\ T'_{k|k-1}=[T ,0, 0, 0, 0, 0, 0]\in\mathbb{R}^{8} Mk∣k−1′​=diag{ M,M,M,M}∈R8×8Tk∣k−1′​=[T,0,0,0,0,0,0]∈R8

therefore:

$${\hat{x}}_{t|t-1}^{\prime }=M_{k|k-1}^{\prime }{\hat{x}}_{t|t-1}+T_{k|k-1}^{\prime }$$

${\hat{x}}_{t|t-1}$The covariance matrix of becomes (the random variable is linearly transformed, and the variance is the coefficient square relationship):
$$P_{t|t-1}^{\prime }=M_{k|k-1}^{\prime }{P}_{t|t-1}{M}_{k|k-1}^{{}^{\prime }T}$$

Thesis (5)~(8)

Then use the corrected ${\hat{x}}_{t|t-1}^{\prime }$and $P_{t|t-1}^{\prime }$Then perform the update step.

Essay (9)

In the implementation, we deduct the target in the picture, and detect the key points in the rest to calculate the affine matrix: (tracker\gmc.py line120)

		# find the keypoints
        mask = np.zeros_like(frame)
        # mask[int(0.05 * height): int(0.95 * height), int(0.05 * width): int(0.95 * width)] = 255
        mask[int(0.02 * height): int(0.98 * height), int(0.02 * width): int(0.98 * width)] = 255
        if detections is not None:
            for det in detections:
                tlbr = (det[:4] / self.downscale).astype(np.int_)
                mask[tlbr[1]:tlbr[3], tlbr[0]:tlbr[2]] = 0

        keypoints = self.detector.detect(frame, mask)

        # compute the descriptors
        keypoints, descriptors = self.extractor.compute(frame, keypoints)

Then match the key points of two adjacent frames

		# Match descriptors.
        knnMatches = self.matcher.knnMatch(self.prevDescriptors, descriptors, 2)

If the number of key points is sufficient, calculate the affine matrix

		 # Find rigid matrix
        if (np.size(prevPoints, 0) > 4) and (np.size(prevPoints, 0) == np.size(prevPoints, 0)):
            H, inliesrs = cv2.estimateAffinePartial2D(prevPoints, currPoints, cv2.RANSAC)

The above part depends on the cv2 library. After getting the affine matrix, correct the result of Kalman (tracker\bot_sort.py line68):

def multi_gmc(stracks, H=np.eye(2, 3)):
        if len(stracks) > 0:
            multi_mean = np.asarray([st.mean.copy() for st in stracks])
            multi_covariance = np.asarray([st.covariance for st in stracks])

            R = H[:2, :2]
            R8x8 = np.kron(np.eye(4, dtype=float), R)
            t = H[:2, 2]

            for i, (mean, cov) in enumerate(zip(multi_mean, multi_covariance)):
                mean = R8x8.dot(mean)
                mean[:2] += t
                cov = R8x8.dot(cov).dot(R8x8.transpose())

                stracks[i].mean = mean
                stracks[i].covariance = cov

4. Matching strategy

In the matching process, the exponential moving average is used to balance the past and current appearance features, and then the motion features and appearance features are linearly combined as the cost matrix, as shown in the following two formulas:

For objects with similar appearance and IoU, give a smaller cost, otherwise set it to 1, and update the elements in C accordingly:

The final result is shown in the table below: