[Stereo Vision (2)] Epipolar Geometry and Key Matrix

This is a personal study note. I have borrowed a lot of good pictures and articles (references are at the end of the article), mainly to sort out relevant knowledge in order to form my own system. The text expression is pieced together, and the symbolic formula is manually entered. Please point out if there are any mistakes.

1. Epipolar geometry

Now I want to find two frames of images $I_1、I_2$Movement between ( $R,t$ ), if a pair of points with the same name p 1 , p 2 p_1, p_2are detected in two frames of images$p_1、p_2$as the picture shows.
Forget about symbolic meaning in imaging principles for a moment.

• Baseline: $O_1{O}_2$connection
• Epipoles: $e_1、e_2$
• 极线（Epipolar line）：$l_1、l_2$
• Epipolar plane: $O_1、O_2,$ the plane formed by$P$

Let the space point P of the left camera coordinate system coordinates $P=[X,Y,Z{]}^T$ , thenthe two pixel positions known fromthe pinhole camera model

$$s_1{p}_1=KP，s_2{p}_2=K(RP+t)$$

Among them, $K$ is the internal reference matrix,$R,t$ is the motion relationship between two cameras (specifically,$R_{21}，t_{21}$ ）。$s_1，s_2$is PPThe zz$of\; point\; P$ in the two coordinate systems$z$ -axis coordinates, when using homogeneous coordinates, a vector will be equal to itself multiplied by any non-zero constant, which is usually used to express a projection relationship, called equal up to a scale. Write$sp\simeq p$ . So there are
$$p_1\simeq KP，p_2\simeq K(RP+t)$$

Next, start deriving:

1. KK $K$ is moved to the left side of the equation:
   $$K_1^1{p}_1\simeq P，K_2^1{p}_2\simeq RP+t$$
2. On the normalized plane, $x_1=K_1^1{p}_1，x_2=K_2^1{p}_2$，则$x_1=P，x_2=RP+t$ , then
   $$x_2=R{x}_1+t$$
3. Use $t$ cross product (vector and self cross product is 0) has$t\times x_2=t\times R{x}_1+t\times t$，即
   $$t\times x_2=t\times R{x}_1$$
4. Multiply both sides of the equation by $x_2^T$，得 $x_2^{T}t\times x_2=x_2^{T}t\times R{x}_1$, the left side of the equation is obviously 0, then
   $$x_2^{T}t\times R{x}_1=0$$
5. Since the cross product is equivalent to the point product of antisymmetric matrices, ttThe antisymmetric matrix of$t$$t^{\wedge }$ (equivalent to both sides simultaneously with$t$ as outer product), then
   $$x_2^T{t}^{\wedge }R{x}_1=0$$

This formula describes two pixel coordinates $x_1、x_2$The connection between the spatial point at the imaging point of the two cameras through $The\; t$ external parameter matrix establishes an equality relationship, or a constraint relationship, and this constraint is calledan epipolar constraint.
Note that above$x_1、x_2$are normalized coordinates, if $p_1、p_2$Substitute $x_1、x_2$, there is
$$p_2^T{K}_2^{-T}{t}^{\wedge }R{K}_1^1{p}_1=0$$
These two formulas are equivalent, both are called epipolar constraints, and the geometric meaning is$O_1、O_2、P、p_1、p_2$Coplanar.

---

2. Key Matrix

Key matrices: essential matrix, fundamental matrix, homography matrix.
The key matrix can establish the coordinate connection between the common points of two views, or complete the coordinate transformation between the common points. They are like a bridge between the dual views of stereo vision, so that they are closely connected to each other to form a whole, and then there is the concept of stereo vision system.

1) Essential Matrix and Fundamental Matrix

Remember $E=t^{\wedge }R$， $F=K_2^{-T}{t}^{\wedge }R{K}_1^1$, are called Essential Matrix and Fundamental Matrix respectively , then the epipolar constraint can be expressed as:
$$\begin{gathered} x_2^{T}E{x}_1=0 \\ {p}_2^{T}F{p}_1=0 \end{gathered}$$

The epipolar constraint concisely gives the spatial position relationship of the two matching points, so the camera pose estimation problem becomes the following two steps:

1. Calculate $E$ or$F$ 。
2. According to $E$ or$F$ Calculate$R、t$

It can be seen that the essential matrix $E$ and Fundamental Matrix$The\; difference\; of\; F$ : the essential matrix directly and the normalized coordinate$x$ establishes a relationship, but$x$ needs to be determined by the internal parameter matrix$K$ and pixel coordinates$p$ is calculated, so the premise of using the essential matrix is ​​that the internal reference matrix$K$ ; while the fundamental matrix is ​​directly related to the pixel coordinates$p$ establishes a relationship without knowing the internal reference matrix. Or:

• Essence matrix: reflects the relationship between the representations of [the image point of a point P in space] in the [camera coordinate system] under [cameras with different viewing angles].
• Fundamental matrix: reflects the relationship between the representations of [image points of a point P in space] in the [image coordinate system] under [cameras with different viewing angles].

Due to $E$ and$F$ only differs from the internal parameters of the camera (the solution method of the two is the same), and the internal parameters are usually known in problems such as SLAM, so in practice, the simpler form of EE is often$E$。

2) Solving the essential matrix

Since translation and rotation each have 3 degrees of freedom, $E=t^{\wedge }R$has 6 degrees of freedom in total. Since the epipolar constraint is a constraint that the equality is zero, after multiplying E by any non-zero constant, the antipolar constraint is still satisfied, that is,$E$ is equivalent at different scales, so$E$ actually has 5 degrees of freedom. This shows that we can use at least 5 pairs of points to solve$E.$ _ However,$The\; inherent\; nature\; of\; E$ is a nonlinear property, which will cause trouble when estimating. Therefore, it is also possible to only consider its scale equivalence and use 8 pairs of points to estimate E—this is the classic eight-point method (Eight -point-algorithm). The eight-point method only utilizes$The\; linear\; nature\; of\; E$ , so it can be solved under the framework of linear algebra.
Consider a pair of matching points whose normalized coordinates are$x_1=[u_1,v_1,1{]}^T$ ， $x_2=[u_2,v_2,1{]}^T$ ,wavelength range,wave
$$\left[\begin{array}{ccc}{u}_2& v_2& 1\end{array}\right]\left[\begin{array}{ccc}{e}_1& e_2& e_3\\ e_4& e_5& e_6\\ e_7& e_8& e_9\end{array}\right]\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=0$$
will matrix$E$ expansion, mapping form:
$$e=[e_1,e_2,e_3,e_4,e_5,e_6,e_7,e_8,e_9{]}^{TThen}$$
, the epipolar constraint can be written as the same as$e$ Correlated linear form:
$$[u_2{u}_1,u_2{v}_1,u_2,v_2{u}_1,v_2{v}_1{v}_2,u_1,v_1,1]\cdot e=In\; the\; same\; way\; as\; 0$$
, it has the same representation for other point pairs. Then a system of linear equations ($u^i,v^i$ display$i$ feature points, and so on):
$$\left[\begin{array}{c}{u}_2^1{u}_1^1,u_2^1{v}_1^1,u_2^1,v_2^1{u}_1^1,v_2^1{v}_1^1{v}_2^1,u_1^1,v_1^1,1\\ u_2^2{u}_1^2,u_2^2{v}_1^2,u_2^2,v_2^2{u}_1^2,v_2^2{v}_1^2{v}_2^2,u_1^2,v_1^2,1\\ u_2^3{u}_1^3,u_2^3{v}_1^3,u_2^3,v_2^3{u}_1^3,v_2^3{v}_1^3{v}_2^3,u_1^3,v_1^3,1\\ ...\\ u_2^8{u}_1^8,u_2^8{v}_1^8,u_2^8,v_2^8{u}_1^8,v_2^8{v}_1^8{v}_2^8,u_1^8,v_1^8,1\end{array}\right]\left[\begin{array}{c}{e}_1\\ e_2\\ e_3\\ e_4\\ e_5\\ e_6\\ e_7\\ e_8\\ e_9\end{array}\right]=0$$
Solving the linear equation$Ae=0$ to get the vector$e$ , the essential matrix$E.$ _ arrow$e$ is the coefficient matrix$The\; null\; space\; of\; A$ , from the knowledge of linear algebra, we can know that the number of special solutions of the equation is$9-rank(A)$，当 $rank(A)=8$ , there is a unique special solution, and all the solutions of the equation are constant times of the special solution, corresponding toEEScale invariance of$E.$

3) Decomposition of the essential matrix

The essential matrix $E=t^{\wedge }R$is a$3\times 3$ matrix, its singular value must be$[p,s,0{]}^{The\; form\; of\; T}$ is called the intrinsic property of the essential matrix (inherited from the antisymmetric matrix$t^{\wedge }$ properties). RRcan be obtained by SVD decomposition$R$ and$t$ , if$E$ -like SVD decomposition is$E=U\Sigma V^T$ , then$R$ and$t$ 可能的结果有：
$$\begin{gathered} t_1^{\wedge }=UZ{U}^T,t_2^{\wedge }=U{Z}^T{U}^T, \\ {R}_1=UW{U}^T,R_2=U{W}^T{U}^{Then} \end{gathered}$$
,$W=\left[\begin{array}{ccc}0& -1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right]$(Specific), $Z=\left[\begin{array}{ccc}0& 1& 0\\ -1& 0& 0\\ 0& 0& 1\end{array}\right]$(antisymmetric matrix, if the vector z= $W={\left(\begin{array}{ccc}0& -1& 0\end{array}\right)}^T$，则$z^{\wedge }=Z$）。

solve $When\; t$ , it is not necessary to solve for$t^{\wedge }$ ，以 $t_1^{\wedge }=UZ{U}^{Take\; T}$ as an example, multiply both sides by$t_1$，有$t_1^{\wedge }{t}_1=UZ{U}^T{t}_1$，即 $UZ{U}^T{t}_1=0$ , multiply$U^T$，有$to{u}^T{t}_1=0$，因为$z^{\wedge }=Z$，且 $z^{\wedge }z=0$ , so$U^T{t}_1=z$，得$t_1=Uz\_=U(0,0,1{)}^T$ , that is,$t_1$is the matrix $The\; third\; column\; of\; U$ , ie
$$t_1=U.col(3)，t_2=-t_1$$
There are four possible sets of $R$ and$Combinations\; of\; t$ :

However, only in the first combination above $P$ has positive depth in both cameras. Therefore, as long as any observation point is brought into the verification, the correct set of solutions can be obtained.

EE solved from the linear equation$E$ , may not satisfy$Intrinsic\; properties\; of\; E$ : the singular value is$(p,s,0)$ . The usual practice is that the EEobtained by the eight-point methodAfter the SVD decomposition of$E$$S=diag(\sigma 1,p2,\sigma 3)$，set$p1⩾p2⩾\sigma 3$，取：
$$E=Udiag(\frac{p_1+p_1}{2},\frac{p_1+p_1}{2},0)$$
This is equivalent to projecting the obtained matrix toEEOn the manifold where$E\; is\; located.$A simpler approach is to take the singular value matrix as$diag(1,1,0)$ , because$E$ has scale equivalence, so it is also reasonable to do so.

The premise of applying the essential matrix is ​​that the internal parameter matrix $K.$ _ The basic matrix does not require internal parameters, but there are many unknowns, and it is difficult to decompose to obtain accurate internal and external parameters. We need to make some assumptions, such as having an initial focal length value, such as the principal point at the center of the image, etc. Regardless of estimating the essential matrix or estimating the fundamental matrix, the camera parameters obtained by decomposition can only be used as initial values. To obtain accurate parameters, it is necessary to solve the exact values ​​through nonlinear iterative optimization.

3) Homography matrix and its solution

The essential matrix and fundamental matrix are not the mutual conversion relationship between two pixels, but an inherent constraint between coordinates. So is there an expression that directly converts the pixel coordinates of the left view to the pixel coordinates of the corresponding point in the right view? Yes, that is, the homography matrix (Homography matrix) H , when the scene in space is the same plane, their projection points in the left and right views can be converted one-to-one through the reversible homography matrix.
Homography is a concept in projective geometry, also known as projective transformation. It maps points (three-dimensional homogeneous vectors) on one projective plane to another projective plane, and maps lines to lines. In other words, a homography is a linear transformation of a three-dimensional homogeneous vector.

Consider a plane in space that satisfies the equation: $n^{T}P+d=0$ , combined with the above, we have

$$\begin{gathered} p_2\simeq K(RP+t) \\ {p}_2\simeq (RP+t(-\frac{n^{T}P}{d})) \\ {p}_2\simeq K(R-\frac{n^{T}P}{d})P \\ {p}_2\simeq K(R-\frac{n^{T}P}{d})K^{-1}{p}_1 \end{gathered}$$
令 $H=K(R-\frac{n^{T}P}{d})K^{-1}$ , there is
$$\begin{gathered} p_2\simeq H{p}_1 \\ {p}_1\simeq H^{-1}{p}_2 \end{gathered}$$

The homography matrix describes the mapping relationship between two planes. If the feature points in the scene all fall on the same plane (such as walls, ground, etc.), motion estimation can be performed through homography. Its definition is related to the parameters of rotation, translation and plane, which is a $3\times 3$ matrix. Similarly,$H$ , decomposed to calculate$R、t$。

$$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]\simeq \left[\begin{array}{ccc}{h}_1& h_2& h_3\\ h_4& h_5& h_6\\ h_7& h_8& h_9\end{array}\right]\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]$$

Let $h_9=1$ (when it takes a non-zero value), according to the normal processing method of the homogeneous formula, we will divide the first and second lines by the third line to remove the scale factor, which can be obtained (note the uu inthe
formula$u$ give$v$ difference)

$$\begin{gathered} u_2=\frac{h_1{u}_1+h_2{v}_1+h_3}{h_7{u}_1+h_8{v}_1+h_9} \\ {v}_1=\frac{h_4{u}_1+h_5{v}_1+h_6}{h_7{u}_1+h_8{v}_1+h_9} \end{gathered}$$

整理得
$$\begin{gathered} h_7{u}_1+h_8{v}_1+h_9-h_7{u}_1{u}_2-h_8{v}_1{u}_2=u_2 \\ {h}_4{u}_1+h_5{v}_1+h_6-h_7{u}_1{v}_2-h_8{v}_1{v}_2=v_2 \end{gathered}$$

Such a set of matching point pairs can construct two constraints (in fact, there are three constraints, but because of linear correlation, only the first two are taken), so the homography matrix with 8 degrees of freedom can be calculated by 4 pairs of matching feature points ( Note that these feature points cannot have three points collinear), that is, solve the following linear equations (when h9 = 0, the right side is zero).
$$\left[\begin{array}{cccccccc}{u}_1^1& v_1^1& 1& 0& 0& 0& -u_1^1{u}_2^1& -v_1^1{u}_2^1\\ 0& 0& 0& u_1^1& v_1^1& 1& -u_1^1{v}_2^1& -v_1^1{v}_2^1\\ u_1^2& v_1^2& 1& 0& 0& 0& -u_1^2{u}_2^2& -v_1^2{u}_2^2\\ 0& 0& 0& u_1^2& v_1^2& 1& -u_1^2{v}_2^2& -v_1^2{v}_2^2\\ u_1^3& v_1^3& 1& 0& 0& 0& -u_3^2{u}_2^3& -v_1^3{u}_2^3\\ 0& 0& 0& u_1^3& v_1^3& 1& -u_1^3{v}_2^3& -v_1^3{v}_2^3\\ u_1^4& v_1^4& 1& 0& 0& 0& -u_3^4{u}_2^4& -v_1^4{u}_2^4\\ 0& 0& 0& u_1^4& v_1^4& 1& -u_1^4{v}_2^4& -v_1^4{v}_2^4\end{array}\right]\left[\begin{array}{c}{h}_1\\ h_2\\ h_3\\ h_4\\ h_5\\ h_6\\ h_7\\ h_8\end{array}\right]=\left[\begin{array}{c}{u}_2^1\\ v_2^1\\ u_2^2\\ v_2^2\\ u_2^3\\ v_2^3\\ u_2^4\\ v_2^4\end{array}\right]$$
This approach puts $The\; H$ matrix is ​​regarded as a vector, and HHis recovered by solving the linear equation of the vector$H$ , also known as **Direct Linear Transform (DLT)*. Similar to the essential matrix, after obtaining the homography matrix, it needs to be decomposed to obtain the corresponding rotation matrix$R$ and the translation vector$t$ . Decomposition methods include numerical methods and analytical methods. The decomposition of the homography matrix will also return 4 sets of rotation matrices and translation vectors, and at the same time, the normal vectors of the planes corresponding to the scene points can be calculated. If the depths of the map points known to be imaged are all positive (i.e. in front of the camera), then two sets of solutions can be ruled out. In the end, there are only two sets of solutions left, and more prior information is needed for judgment.

Homography is of great significance in SLAM. When the feature points are coplanar or the camera undergoes pure rotation, the degrees of freedom of the fundamental matrix decrease, which is called degenerate. The data in reality always contains some noise. At this time, if you continue to use the eight-point method to solve the fundamental matrix, the extra degrees of freedom of the fundamental matrix will be mainly determined by the noise. In order to avoid the impact of degradation phenomena, we usually estimate the fundamental matrix $F$ and the homography matrix$H$ , choose the one with smaller reprojection error as the final motion estimation matrix.

In addition to DLT, RANSAC can also be used.

---

references:

[1] Gao Xiang, Zhang Tao, Visual SLAM Lecture Fourteen: From Theory to Practice (2nd Edition)
[2] Stereo Vision Introductory Guide (2): Key Matrix (Essential Matrix, Fundamental Matrix, Homography Matrix)
[3] Yes Polar Geometry and Triangulation
[4] Computer Vision 6 - Epipolar Geometry and Fundamental Matrix
[5] Homography Matrix and Epipolar Geometry