Solve the correspondence between picture pixels between different cameras (homography matrix solution)

1. Scene

The relative positions of camera 1 and camera 2 remain unchanged, and the pictures taken by the cameras overlap. Find the one-to-one correspondence between their overlapping parts. The mathematical language description is that the pixel (u1, v1) of point P in the picture of camera 1 is known, and the pixel value of point P in camera 1 in the picture of camera 2 is (u2, v2). There is some kind of transformation between them, so find the transformation matrix.

Because the scene involved is relatively simple and currently does not involve depth, and the target collected at the same time is approximately plane, the scene can be simplified and solved using a homography matrix. Therefore, the transformation matrix involved above is assumed to be the homography matrix H (3*3 matrix), and they satisfy the following relationship.

In this case, it is much simpler (if the scene is complex, involves depth or the acquisition object is not a plane, you can use the essential matrix/basic matrix method to obtain the transformation matrix), and you only need to use the standard calibration plate to calculate H.

2. Homography Matrix

Definition: There is a homography between images of [the same plane object] taken from different positions with an [ideal imaging] camera , which can be expressed by [perspective transformation] . There is the following formula:

The next step is to solve the H matrix. The above formula is expanded as follows:

Based on the corresponding relationship between plane coordinates and homogeneous coordinates, the above formula can be expressed as:

further,

Write it in the form of AX=0, as follows. There are many ways to solve this form, which are also covered in the previous blog . However, it should be pointed out that although the H matrix has 9 unknowns, it only has 8 degrees of freedom (plane relationship), among which h33=1. Therefore, only 4 non-collinear points are needed to solve the equation.

3. Actual effects

3.1 All code

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cv2
import numpy as np


def OnMouseAction(event, x, y, flags, param):
    """
    鼠标的回调函数，处理鼠标事件
    :param event:
    :param x:
    :param y:
    :param flags:
    :param param:
    :return:
    """
    if event == cv2.EVENT_LBUTTONDOWN:
        global gimg_x, gimg_y, gis_ok
        gimg_x = x
        gimg_y = y
        gis_ok = True
    elif event == cv2.EVENT_RBUTTONDOWN:
        print("右键点击")
    elif flags == cv2.EVENT_FLAG_LBUTTON:
        print("左鍵拖曳")
    elif event == cv2.EVENT_MBUTTONDOWN:
        print("中键点击")


def verification(img1, img2, H):
    def nothing(x):
        pass

    cv2.namedWindow('image1')
    cv2.setMouseCallback('image1', OnMouseAction)
    # create trackbars for color change
    cv2.createTrackbar('thr', 'image1', 121, 255, nothing)
    cv2.createTrackbar('Shading', 'image1', 255, 255, nothing)
    count = 0
    while True:
        cv2.imshow("image1", img1)
        cv2.imshow("image2", img2)

        k = cv2.waitKey(1) & 0xFF
        # 通过关闭窗口的右上角关闭
        if cv2.getWindowProperty('image1', cv2.WND_PROP_AUTOSIZE) < 1:
            break
        # 通过按键盘的ESC退出
        if k == 27:
            break
        global gimg_x, gimg_y, gis_ok
        if gis_ok:
            count += 1
            cv2.circle(img1, (gimg_x, gimg_y), 3, (0, 0, 255), -1)
            cv2.putText(img1, str(count), (gimg_x, gimg_y), 2, 1, (0, 0, 255))
            gis_ok = False

            (x, y, z) = np.matmul(H, np.array([gimg_x, gimg_y, 1]).T)

            cv2.circle(img2, (int(x / z), int(y / z)), 2, (0, 0, 255), -1)
            cv2.putText(img2, str(count), (int(x / z), int(y / z)), 2, 1, (0, 0, 255))


def getHomography(img1, img2):
    gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
    gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

    ret, corners1_1 = cv2.findChessboardCorners(gray1, (gcols, grows), None)
    if not ret:
        return ret, None
    # sub-pixel corner detection
    corners1_2 = cv2.cornerSubPix(gray1, corners1_1, (11, 11), (-1, -1), criteria)

    ret, corners2_1 = cv2.findChessboardCorners(gray2, (gcols, grows), None)
    if not ret:
        return ret, None
    # sub-pixel corner detection
    corners2_2 = cv2.cornerSubPix(gray2, corners2_1, (11, 11), (-1, -1), criteria)

    H, mask = cv2.findHomography(corners1_2, corners2_2, cv2.RANSAC)

    return True, H


gimg1Root = "./image_homography/1080p_1.png"
gimg2Root = "./image_homography/1080p_2.png"
gimg3Root = "./image_homography/720p_3.png"
(grows, gcols) = (8, 11)
def main():
    img1 = cv2.imread(gimg1Root)
    img2 = cv2.imread(gimg3Root)
    is_ok, H = getHomography(img1, img2)
    print(H)

    verification(img1, img2, H)


global gimg_x, gimg_y, gis_ok
gimg_x = 0
gimg_y = 0
gis_ok = False
if __name__ == "__main__":
    main()