[Multiple target tracking study notes] Optical flow method, motion reconstruction and target tracking

---

The optical flow method is a traditional method for target tracking, refer to the video station B to learn and take notes.

---

1. Motion Field and Optical Flow

Optical flow, as the name implies, is the flow of light. For images, what is the flow of light? Imagine that the camera is moving (the same is true if the target is moving), and the movement of the target is actually the "movement" of "pixels" , so it is called optical flow, as shown in the following figure:

As shown in the figure above, if the camera is moving, the brightness pattern of the tree will change, and this trend of change actually represents the movement of the target.

Question 1: So, what is the relationship between what we think is the change of the so-called pixel brightness pattern and the real movement of the target?

We call the projection (Projection) of the real physical movement of the target on a certain observation as the motion field (Motion Field), for example, the following figure: The target is moving to the upper left, and its projection on the sensor (that is, our observation) also has a Corresponding direction of movement, that is, downward movement.

  \space  
Question 2: Are motion fields and optical flow equivalent?

The answer is no. Please look at the following situations:

1. With sports field without optical flow or with optical flow without sports field

On the left side of the figure below, it is assumed that the sphere is rotating, but the light source is not moving. It is a situation where there is a sports field and no optical flow. On the right side, the sphere is not moving, but the light source is moving.

  \space 

2. 
   The example that the optical flow and the sports field are not equal is more direct. For example, the colored lights at the entrance of the barber shop, we know that the real physical movement is horizontal to the right, however, if we focus on the change of pixel brightness, the direction of the optical flow is downward , the two are exactly perpendicular.

  \space 

2. Optical flow constraint equation

How can we better estimate motion through optical flow ?

First of all, we have to make some assumptions, so that the problem can go on. We consider the following two consecutive frames of pictures:

In this picture, the eagle is the moving target. If we focus on one of the pixels $(x,y)$ , assuming that the position of the next frame is at$(x+\delta x,y+\delta y)$ , we can get the speed of the pixel (that is, the brightness motion mode, the theoretical optical flow direction):

$$(u,v)=(\delta x/\delta t,\delta y/\delta t)$$

Assumption 1 The brightness of corresponding pixels between two frames remains unchanged

We think that $\delta t$ is small enough so that the pixels we care about will not produce brightness changes between two frames, assuming that the brightness is represented by$I(\cdot )$表示， 有:
$$I(x,y,t)=I(x+\delta x,y+y,t+\delta t)$$

We carry out the Taylor expansion of the multivariate function on the right side of the equal sign, and ignore the quadratic terms and higher-order terms, as follows:
$$I(x+\delta x,y+y,t+\delta t)=I(x,y,t)+\frac{\partial I}{\partial x}\delta x\_+\frac{\partial I}{\partial y}y+\frac{\partial I}{\partial t}\delta t$$

联立上两式, 有:
$$\frac{\partial I}{\partial x}\delta x\_+\frac{\partial I}{\partial y}y+\frac{\partial I}{\partial t}\delta t=0$$

令$\delta t\to 0$ . Divide both sides with$\delta t,$ there are:

$$\frac{\partial I}{\partial x}\frac{\delta }{\delta t}+\frac{\partial I}{\partial y}\frac{}{\delta t}+\frac{\partial I}{\partial t}=0$$

Form $\frac{\delta }{\delta t},\frac{}{\delta t}$Respectively, the optical flow true value (velocity) $u,v$ , so the above formula can be written as:

$$I_{x}u+I_{y}v+I_t=0$$

The above is the optical flow constraint equation .

I x , I y , I $I_x,I_y,I_t$The method is similar to the way of calculating the gradient in image processing , fixing one variable and calculating the remaining variables to get the difference, such as $I_x$Calculate as follows:

  \space  
Ok, now we know $I_x,I_y,I_t$, if discussing $u,v$ plane, formula$$I_{x}u+I_{y}v+I_t=0$$
specifies a straight line on the plane. But$u,v$ is a variable, we cannot get the exact value.

Specifically, as shown in the figure above, assume that the true value of the optical flow is $u=(u,v)$ , we decompose it into vertical straight line direction and parallel straight line direction
$\mathbf{u}=u_{\mathbf{n}}+u_{\mathbf{p}}$.

According to the knowledge of geometry, it is quickly obtained:

$$u_n=\frac{|I_t|}{\sqrt{I_x^2+I_y^2}}(I_x,I_y)$$

However, $u_p$We don't know, so more constraints are needed.

3. Lucas Kanade Algorithm

Continuing from the above, we need more constraints. The assumptions made by the Lucas Kanade algorithm are:

Assumption 2 assumes that for each pixel, the motion field is the same as the optical flow and the pixels in its neighborhood

Suppose a certain neighborhood of a pixel is $W$ , the size is$n\times n$ rectangle, then forWWEach pixel in$W$$(k,l)\in W$ ,光流都相同, 因此有:
$$I_x(k,l)and+I_y(k,l)v+I_t(k,l)=0,(k,l)\in W$$
above is$n^2$ equations, we write in matrix form:
$$\left[\begin{array}{cc}{I}_x(1,1)& I_y(1,1)\\ ...& ...\\ I_x(n,n)& I_y(n,n)\end{array}\right]\left[\begin{array}{c}u\\ v\end{array}\right]=\left[\begin{array}{c}-I_t(1,1)\\ ...\\ -I_t(n,n)\end{array}\right]$$

记为:
$$\mathbf{Au}=\mathbf{B},A\in {\mathbb{R}}^{n^2\times 2},u=[u,v{]}^T,B\in R^{n^2\times 1}$$

$n^{The\; number\; 2}$ is a bit large, for the convenience of calculation, we write the equation as:

$$({\mathbf{A}}^{T}A)and={\mathbf{A}}^{\mathbf{T}}\mathbf{B}$$
其中$A^{T}A\in {\mathbb{R}}^{2\times 2}$ . Station$A^{T}A$reversible, yes:
$$\mathbf{u}=({\mathbf{A}}^{T}A{)}^{-1}{\mathbf{A}}^{\mathbf{T}}\mathbf{B}$$

It can be seen that the optical flow estimate is essentially $n^2$ constraint equations$Au=$The least squares solution of$B$ , so the optical flow is solved. However, the solution method above the LK algorithm also has its limitations, andmust require:

1. $A^{TA}reversible$.
2. $A^{T}A$must be well-conditioned. The so-called well-conditioned refers to its two eigenvalues$l_1,l_2$Satisfied:
   neither of them is too close to 0, and
   $l_1\ge l_2$And
   not $l_1>>l_2$.

The meaning of weil-conditioned is that if the eigenvalues ​​are too small, $u$ is also small, the result is unreliable.
If the two eigenvalues ​​differ greatly, the specific direction of motion cannot be guaranteed.

4. Coarse to fine optical flow estimation (optical flow pyramid, Coarse to fine estimation)

The LK algorithm has a priori assumption, namely $\delta t$ is small enough, that is, the assumption of small motion. But if the range of motion is large, how to do it?

We can reduce the resolution of the picture, for example, by pooling, so that the original large motion will become small motion. As shown in the figure below.

After converting into small motion, we can use the previous algorithm to solve the optical flow.

Assume that we have four resolutions, from low to high are 1, 2, 3, 4. At 1 resolution, we use the estimation of small motion to calculate the optical flow, assuming the result is ( u , v ) 0 (u $(u,v{)}^0$.

In this way, we use the optical flow calculated on the 1 resolution on the 2 resolution to generate a new image (similar to frame interpolation). This image will be closer to the motion of the original resolution, but not accurate. This step is called " Wrap".

Therefore, similarly, we recalculate the optical flow for the 2-resolution after frame interpolation, and obtain the optical flow change $D(u,v{)}^{1.}$ So at this point we can get a more accurate optical flow estimate$(u,v{)}^1=(u,v{)}^0+D(u,v{)}^{1.}$ Thisstep is called "Plus".

Then apply this optical flow interpolation frame to 3 resolutions, and continue like this. As shown in the figure:

Therefore, we estimate the large motion of fine resolution step by step from the small motion of coarse resolution, in short, it is an iterative process of Warp+Plus.

5. Scene reconstruction under camera motion

The premise of discussing this issue is that the camera is assumed to be an Orthographic Camera, that is, the size of the object in the final rendered picture remains the same regardless of whether the object is far or near from the camera. In other words, the difference in the scene imaged by the camera is much greater than the depth of field, so the size of the object is no longer important. At this time, we hope to restore the 3D information of the key points in the 2D image when the camera is moving. As shown in the figure below.

5.0 Problem Modeling

In the world coordinate (in the real three-dimensional space), suppose the $The\; coordinates\; of\; the\; t$ -frame camera are$C_f\in {\mathbb{R}}^3$ , the coordinates of a key point in the scene are$P_f\in R^3$ , but the camera obtains a 2D image, assuming that the coordinates on the imaging plane are$[u,v{]}^T\in R^2$ , as shown in the figure below (where O is the origin of the world coordinate,$i,j\in R^3$ is the three-dimensional vector of the unit coordinate system of 2D imaging (the vector in the world coordinate), which represents the orientation (position) of the camera):

According to the geometric relationship, there are:
$$\begin{gathered} u=i^T{x}_c \\ v=j^T{x}_c \\ {x}_c=P-C \end{gathered}$$
sou
$$u=i^T(P-C),v=j^T(P-C)$$
$u,$We know$v$$i,j$ ) and real coordinates$P,$We don't know$C.$ So how to solve$What\; about\; P$ ?

Make an assumption here, assuming that the origin of the world coordinate is at the center of all key points in the imaging:
$$\sum _p{P}_{f,p}=0,\forall f$$

where $p$ means the pth point.

We consider the center coordinates in the 2D picture under this definition:
$$\begin{gathered} {\overline{u}}_f=\frac{1}{|P|}\sum _{p\in P}{u}_{f,p} \\ =\frac{1}{|P|}\sum _{p\in P}{i}_f^T(P_p-C_f) \\ =-\frac{1}{|P|}\sum _{p\in P}{i}_f^T{C}_f=-i_f^T{C}_f \end{gathered}$$

We move the origin of the 2D coordinate system to the above center coordinates, then the new coordinates are
$$\begin{gathered} {\hat{u}}_{f,p}=u_{f,p}-{\overline{u}}_f \\ =i_f^T(P_p-C_f)-(-i_f^T{C}_f)=i_f^T{P}_p \end{gathered}$$

Therefore, under the above assumptions, the new coordinates have nothing to do with the real position of the camera.

for $v$ coordinates, similarly there is${\hat{v}}_{f,p}=j_f^T{P}_p$.

Now we have $F$ frames, assuming each frame maintains$N$ key points, each point has the following relationship:
$$\left[\begin{array}{c}{\hat{u}}_{f,p}\\ {\hat{v}}_{f,p}\end{array}\right]=\left[\begin{array}{c}{i}_f^T\\ j_f^T\end{array}\right]P_p$$
So we can list the following equations:

concisely written as:
$$W_{2F\times N}=M_{2F\times 3}{S}_{3\times N}$$

The two matrices on the right side of the equation are unknown. The Tomasi-Kanade method is introduced below to solve

5.1 Problem Solving

We consider $Perform\; singular\; value\; decomposition\; of\; W$ to see if it can be decomposed into the form of multiplication of two matrices and then determine$MandS$ . LetWWFor WSVD:$W$
$$W=U\Sigma V^T$$
is obviously$\text{rank}(W)\le 3$ , so$W$ also has only three non-zero singular values, taking its truncated SVD:

$$W=U_1{S}^{\prime }{V}_1^T$$
其中$U_1\in {\mathbb{R}}^{2F\times 3},S^{\prime }\in {\mathbb{R}}^{3\times 3},V_1^T\in R^{3\times N}$.

We will $W$ is rewritten as

$$W=U_1{S}^{\prime 1/2}Q{Q}^{-1}{S}^{\prime 1/2}{V}_1^T$$
where $Q\in R^{3\times 3}$ is a non-singular matrix, let
$$\begin{gathered} M=U_1{S}^{\prime 1/2}Q \\ S=Q^{-1}{S}^{\prime 1/2}{V}_1^T \end{gathered}$$
Now the problem is to solve $Q$ , can be calculated. We note thatwe have not used the orthogonality of 2D coordinates.

$$M=U_1{S}^{\prime 1/2}Q=\left[\begin{array}{c}{\hat{i}}_1^T\\ ...\\ {\hat{i}}_F^T\end{array}\right]Q=\left[\begin{array}{c}{\hat{i}}_1^{T}Q\\ ...\\ {\hat{i}}_F^{T}Q\end{array}\right]$$
therefore:

$$\begin{gathered} \forall f,{\hat{i}}_f^{T}Q{Q}^T{\hat{i}}_f=1, \\ \forall f_1\ne f_2,{\hat{i}}_{f_1}^{T}Q{Q}^T{\hat{i}}_{f_2}=0 \end{gathered}$$

The above is $3F$ equations, and$Q$ has only nine elements, so this is an overdetermined equation, and$Q$ . Solve for$After\; Q$ , bring it into the result formula of SVD, you can calculate$M$ and$S$.