Internal parameters include camera focal length, pixel size, principal point position, etc.;

External parameters include camera position, orientation, etc.

Through camera calibration, pixel coordinates can be associated with actual physical coordinates to realize machine vision applications, such as 3D reconstruction, pose estimation, target tracking, etc. The purpose of camera calibration is to obtain accurate camera parameters, so as to ensure the accuracy and robustness of image processing algorithms.

b. Purpose of Camera Calibration

The main purpose of camera calibration is to obtain the internal and external parameters of the camera, and establish the correspondence between pixel coordinates and actual physical coordinates to realize machine vision applications. Specifically, the purpose of camera calibration includes the following aspects:

Accurate measurement: Through camera calibration, information such as the size, shape, position, and attitude of an object can be accurately measured.

3D reconstruction: Through camera calibration, the coordinate information of objects in the 3D scene can be obtained, thereby realizing 3D reconstruction.

Target tracking: through camera calibration, the pixel coordinates can be associated with the actual physical coordinates to realize the tracking of targets between different frames.

Robot navigation: Through camera calibration, the position and attitude information of the robot relative to the environment can be obtained, so as to realize the navigation and control of the robot.

Automated production: Through camera calibration, functions such as object detection, measurement and sorting in automated production lines can be realized.

In short, the purpose of camera calibration is to accurately obtain the internal and external parameters of the camera, so as to realize machine vision applications and improve production efficiency and product quality. To put it bluntly, the purpose of camera calibration is to obtain the camera matrix and distortion coefficients.

The process of camera calibration

a. Calibration board selection

In camera calibration, it is very important to choose a suitable calibration board.

In terms of size, it should be selected according to the camera’s field of view and the minimum pixel size. Usually, the calibration plate fills at least 80% of the camera’s field of view; in terms of shape, there are rectangles, prototypes, checkerboards, etc. Here we use a checkerboard grid, it can provide enough calibration points, and the calibration position is relatively easy to determine.

b. Calibration board placement and shooting

The placement of the calibration board and the shooting process have a great impact on the accuracy and reliability of camera calibration.

In terms of placement, it should be placed within the field of view of the camera, and the flatness and stability of the calibration board must be ensured to avoid distortion and bending of the calibration board. At the same time, the calibration plate should be placed perpendicular to the optical axis of the camera. In terms of shooting angle, the calibration board should be taken from different angles, including looking down, looking up, left and right side views, etc. At the same time, shadows and reflections should be avoided when shooting the calibration plate to ensure that the surface of the calibration plate is smooth and even. In terms of the number of shots, usually, at least 10 photos of the calibration board at different angles and positions need to be taken for camera calibration. In terms of shooting conditions, the stability of the shooting environment should be ensured when the calibration board is shooting, and factors such as light changes and object movement should be avoided from affecting the calibration results. At the same time, make sure that the camera's exposure, focus and other settings are consistent when shooting.

c. Extraction of the corner points of the calibration board

Here we use the OpenCV method in Python, using the cv2.findChessboardCorners() function to automatically extract the corner points of the calibration board.

Its technical route is as follows:

Image preprocessing: Before corner extraction, it is usually necessary to preprocess the image of the calibration plate, including grayscale, smoothing, binarization, etc. These processes can improve the accuracy and stability of corner extraction.
Calibration board parameter setting: When using the cv2.findChessboardCorners() function, you need to set the parameters of the calibration board, including the size of the calibration board, the number of corner points in each row and column, etc. These parameters need to be set according to the actual situation to ensure the accuracy and stability of corner point extraction.
Corner extraction algorithm selection: OpenCV provides a variety of corner extraction algorithms, including Harris, Shi-Tomasi, FAST, etc. When performing camera calibration, a corner point extraction algorithm suitable for the calibration board is usually selected to improve the accuracy and stability of corner point extraction.
Corner coordinate optimization: The cv2.findChessboardCorners() function can automatically extract the corner coordinates of the calibration board, but there may be some errors in these coordinates. In order to improve the calibration accuracy, the corner coordinates can be optimized, for example, use the cv2.cornerSubPix() function to optimize the corner points at the sub-pixel level.

Therefore, it is necessary to pay attention to preprocessing, parameter setting, algorithm selection and coordinate optimization when extracting the corner points of the calibration board to ensure the accuracy and stability of the corner point extraction.

Zhang Zhengyou Calibration Method

Zhang Zhengyou's calibration method is a classic method of camera calibration, here we reason it from a mathematical point of view.

a. Reverse camera matrix

Without considering the distortion, the corner detection of the calibration board in the image is carried out to obtain the pixel coordinate value of the corner point, which is $[u \quad v \quad 1]^{T}$ converted to the coordinates of the corner point of the calibration board in the world coordinate system according to the size of the checkerboard $[^{W}X\quad ^{W}Y\quad ^{W}Z\quad 1]^{T}$ .

In this way we can get such a formula:

$\left [ \begin {array}{ccc}\hat{u} \\ \hat{v}\\ 1 \end{array}\right ]^{T}=K[^{C}_{W}R \quad ^{C}P_{w_{0}}]\left [ \begin {array}{ccc}^{W}X \\ ^{W}Y\\ ^{W}Z\\1 \end{array}\right ]^{T}$

But for the convenience of calculation, we often $^{W}Z=0$ put the calibration plate in the X—Y plane.

This formula simplifies to:

$\left [ \begin {array}{ccc}\hat{u} \\ \hat{v}\\ 1 \end{array}\right ]^{T}=K[r_{1}\quad r_{2} \quad p ]\left [ \begin {array}{ccc}^{W}X \\ ^{W}Y\\1 \end{array}\right ]^{T}$

$r_{1}$ The sum here $r_{2}$ is the first two columns of the rotation matrix $^{C}_{W}R$ , and the homography matrix from the calibration plate to the imaging plane is set as

$H=\left [ \begin {array}{ccc} h_{11} &h_{12}&h_{13}\\ h_{21}&h_{22}&h_{23}\\ h_{31}&h_{32}&h_{33}\end{} \right ]=[h_{1} \quad h_{2}\quad h_{3} ]$

$H$ is a nonsingular 3×3 matrix with scaling factors.

Based on the known $H$ inverse solution internal parameter matrix , we can know $K$ by substituting $h_{i}=Kr_{i}(i=1,2)$

$\small \left\{\begin{matrix} h_{1}^{T}Qh_{2}=0\\ h_{1}^{T}Qh_{1}=h_{2}^{T}Qh_{2}=1 \end{matrix}\right.$

Among them $\small Q=K^{-T}K^{-1}$ , the symmetric matrix $K$ contains 5 elements and requires the only closed solution $H$ that can be solved by 3 sets of equations $K$ . Therefore, more than three sets of pictures need to be taken during calibration. $K$ The corresponding extrinsic parameter matrix can be calculated .

$h_{i}^TQh_{j}=[h_{i1} \quad h_{i2} \quad h_{i3} ]\left [ \begin {array}{ccc} q_{1} &h_{2}&h_{3}\\ h_{2}&h_{4}&h_{5}\\ h_{3}&h_{5}&h_{6}\end{} \right ] \left [ \begin {array}{ccc}h_{j1} \\ h_{j2}\\ h_{j3} \end{array}\right ]=v_{ij}^{T}Q$

can be rewritten as

$\left [ \begin {array}{ccc} v_{12} ^{T}\\ v_{11}^{T}-v{22}^T\end{} \right ] q=0$

Among them, the matrix $V$ is completely derived from the known $H$ , $q$ which is $Q$ the vector form of . Since one can only contribute two linear equations at most, at least three calibration plates are required to determine the internal parameter matrix (generally 10 to 20 calibration plates are suitable $H$ for engineering). $V_{q}=0$ Solve the calculation, and the solution is the camera internal parameter matrix.

In the above calculation, the influence of camera distortion is ignored (of course, this is also the condition given at the beginning), and the least square method is used to estimate the distortion coefficient of the actual radial distortion (ignoring tangential distortion) for the internal and external parameters. Finally, through The maximum likelihood estimation method is used for optimization to obtain higher precision values.

b. Inverse solution distortion coefficient

(Zhang Zhengyou’s calibration method only considers radial distortion, not tangential distortion) Use the obtained internal parameter matrix to inversely solve the distortion parameters.

set up:

The undistorted pixel coordinates are $[\hat{u} \quad \hat{v}]^{T}$

The coordinates of the imaging plane are $[\hat{x} \quad \hat{y}]^{T}$

The distorted pixel coordinates are $[u \quad v]^{T}$

The coordinates of the imaging plane are $[x \quad y]^{T}$

internal reference $s=0$

Then there are:

$\begin{cases} u=f_{u}x+c_{u}\\ v=f_{v}y+c_{v} \end{cases}$ $\begin{cases} \hat{u}=f_{u}\hat{x}+c_{u}\\ \hat{v}=f_{v}\hat{y}+c_{v} \end{cases}$

Subtract the two formulas, and substitute into $f_{u}x=u-c_{u}$ , $f_{v}y=v-c_{v}$ and the radial distortion formula

$\begin{cases} \hat{x}=x(1+k_{1}r^{2}+k_{2}r^{4})\\ \hat{y}=y(1+k_{1}r^{2}+k_{2}r^{4}) \end{cases}$

inferred

$\left [ \begin {array}{ccc} (u-c_{u})r ^{2}&(u-c_{u})r ^{4}\\ (v-c_{v})r^{2}&(v-c_{v})r^{4} \end{} \right ] [ \begin {array}{ccc} k_{1}\\k_{2} \end{}]=[ \begin {array}{ccc} \hat{u}-u\\ \hat{v}-v \end{}]$

but due to

$\left [ \begin {array}{ccc}\hat{u} \\ \hat{v}\\ 1 \end{array}\right ]^{T}=K[^{C}_{W}R \quad ^{C}P_{w_{0}}]\left [ \begin {array}{ccc}^{W}X \\ ^{W}Y\\ ^{W}Z\\1 \end{array}\right ]^{T}$

For $M$ the calibration board pictures, each picture takes $N$ a corner point, then the non-homogeneous linear equation can be constructed.

$D_{2MN\times 2}k=d_{2MN\times 1}$

where $k=[k_{1}\quad k_{2}]^{T}$ is the distortion coefficient. Finally, the regression is performed by the method of least squares

$k=(D^{T}D)^{-1}D^{T}d$

Camera Calibration Using Python

a. Install OpenCV

First you need to install the OpenCV library, you can use pip to install:

pip install opencv-python
pip install opencv-contrib-python

b. Prepare the calibration board picture

This part is for the situation where the calibration board is not purchased, you can use the code to generate, and then print and take pictures.

import cv2
import numpy as np

# 定义棋盘格的尺寸
size = 140
# 定义标定板尺寸
boardx = size * 10
boardy = size * 7

canvas = np.zeros((boardy, boardx, 1), np.uint8) # 创建画布
for i in range(0, boardx):
    for j in range(0, boardy):
        if (int(i/size) + int(j/size)) % 2 != 0: # 判定是否为奇数格
            canvas[j, i] = 255
cv2.imwrite("chessboard.png", canvas)

The image size generated here is 1400*980. If you want to modify the size, using ps to modify is the easiest way.

c. Corner detection using OpenCV

When performing camera calibration, it is necessary to detect the corner points on the calibration board and use them for subsequent calibration calculations. OpenCV provides a function 'findChessboardCorners' for corner detection.

Here is a sample code for corner detection using OpenCV:

import cv2
import numpy as np

pic_name='chessboard.png'
img = cv2.imread(pic_name)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# 设置棋盘格尺寸和数量
board_size = (6, 9)
# 找到棋盘格角点
_, corners = cv2.findChessboardCorners(gray, board_size, None)

if _:
    # 提取亚像素角点
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
    # 在图像上绘制角点
    img = cv2.drawChessboardCorners(img, board_size, corners, _)
    cv2.imwrite('corners.png', img)
else:
    print(f'Unable to find corners in the {pic_name} image.')

This is the effect of a separate detection, but in the actual checkerboard photos, the effect is not very good, and it is waiting for improvement.

d. Solve the homography matrix

The following code refers to the blog of Eating Lee . Here I will give a careful explanation and analysis, and try to ensure that I have added my own things to this blog.

#homography.py

import numpy as np
from scipy import optimize as opt

# 求输入数据的归一化矩阵
def normalizing_input_data(coor_data):
    x_avg = np.mean(coor_data[:, 0])
    y_avg = np.mean(coor_data[:, 1])
    sx = np.sqrt(2) / np.std(coor_data[:, 0])
    sy = np.sqrt(2) / np.std(coor_data[:, 1])

    norm_matrix = np.matrix([[sx, 0, -sx * x_avg],
                             [0, sy, -sy * y_avg],
                             [0, 0, 1]])
    return norm_matrix


# 求取初始估计的单应矩阵
def get_initial_H(pic_coor, real_coor):
    # 获得归一化矩阵
    pic_norm_mat = normalizing_input_data(pic_coor)
    real_norm_mat = normalizing_input_data(real_coor)

    M = []
    for i in range(len(pic_coor)):
        # 转换为齐次坐标
        single_pic_coor = np.array([pic_coor[i][0], pic_coor[i][1], 1])
        single_real_coor = np.array([real_coor[i][0], real_coor[i][1], 1])

        # 坐标归一化
        pic_norm = np.dot(pic_norm_mat, single_pic_coor)
        real_norm = np.dot(real_norm_mat, single_real_coor)

        # 构造M矩阵
        M.append(np.array([-real_norm.item(0), -real_norm.item(1), -1,
                           0, 0, 0,
                           pic_norm.item(0) * real_norm.item(0), pic_norm.item(0) * real_norm.item(1),
                           pic_norm.item(0)]))

        M.append(np.array([0, 0, 0,
                           -real_norm.item(0), -real_norm.item(1), -1,
                           pic_norm.item(1) * real_norm.item(0), pic_norm.item(1) * real_norm.item(1),
                           pic_norm.item(1)]))

    # 利用SVD求解M * h = 0中h的解
    U, S, VT = np.linalg.svd((np.array(M, dtype='float')).reshape((-1, 9)))
    # 最小的奇异值对应的奇异向量,S求出来按大小排列的，最后的最小
    H = VT[-1].reshape((3, 3))
    H = np.dot(np.dot(np.linalg.inv(pic_norm_mat), H), real_norm_mat)
    H /= H[-1, -1]

    return H


# 返回估计坐标与真实坐标偏差
def value(H, pic_coor, real_coor):
    Y = np.array([])
    for i in range(len(real_coor)):
        single_real_coor = np.array([real_coor[i, 0], real_coor[i, 1], 1])
        U = np.dot(H.reshape(3, 3), single_real_coor)
        U /= U[-1]
        Y = np.append(Y, U[:2])

    Y_NEW = (pic_coor.reshape(-1) - Y)

    return Y_NEW


# 返回对应jacobian矩阵
def jacobian(H, pic_coor, real_coor):
    J = []
    for i in range(len(real_coor)):
        sx = H[0] * real_coor[i][0] + H[1] * real_coor[i][1] + H[2]
        sy = H[3] * real_coor[i][0] + H[4] * real_coor[i][1] + H[5]
        w = H[6] * real_coor[i][0] + H[7] * real_coor[i][1] + H[8]
        w2 = w * w

        J.append(np.array([real_coor[i][0] / w, real_coor[i][1] / w, 1 / w,
                           0, 0, 0,
                           -sx * real_coor[i][0] / w2, -sx * real_coor[i][1] / w2, -sx / w2]))

        J.append(np.array([0, 0, 0,
                           real_coor[i][0] / w, real_coor[i][1] / w, 1 / w,
                           -sy * real_coor[i][0] / w2, -sy * real_coor[i][1] / w2, -sy / w2]))

    return np.array(J)


# 利用Levenberg Marquart算法微调H
def refine_H(pic_coor, real_coor, initial_H):
    initial_H = np.array(initial_H)
    final_H = opt.leastsq(value,
                          initial_H,
                          Dfun=jacobian,
                          args=(pic_coor, real_coor))[0]

    final_H /= np.array(final_H[-1])
    return final_H


# 返回微调后的H
def get_homography(pic_coor, real_coor):
    refined_homographies = []

    error = []
    for i in range(len(pic_coor)):
        initial_H = get_initial_H(pic_coor[i], real_coor[i])
        final_H = refine_H(pic_coor[i], real_coor[i], initial_H)
        refined_homographies.append(final_H)

    return np.array(refined_homographies)

What is realized is the solution of homography transformation in image registration.

Specifically, the implementation steps:

The input image and ground truth coordinate data are normalized using a normalization matrix.

Construct the M matrix and solve for h in M*h = 0 by SVD.

The estimated homography is fine-tuned to obtain the final homography.

For this part, it can be understood in combination with the formula of the above inverse camera matrix.

e. Calculation of internal and external matrix parameters

#InExparam.py

import numpy as np

# 返回每一幅图的外参矩阵[R|t]
def get_ex_param(H, intrinsics_param):
    ex_param = []

    inv_intrinsics_param = np.linalg.inv(intrinsics_param)
    for i in range(len(H)):
        h0 = (H[i].reshape(3, 3))[:, 0]
        h1 = (H[i].reshape(3, 3))[:, 1]
        h2 = (H[i].reshape(3, 3))[:, 2]

        scale_factor = 1 / np.linalg.norm(np.dot(inv_intrinsics_param, h0))

        r0 = scale_factor * np.dot(inv_intrinsics_param, h0)
        r1 = scale_factor * np.dot(inv_intrinsics_param, h1)
        t = scale_factor * np.dot(inv_intrinsics_param, h2)
        r2 = np.cross(r0, r1)

        R = np.array([r0, r1, r2, t]).transpose()
        ex_param.append(R)

    return ex_param

# 返回pq位置对应的v向量
def create_v(p, q, H):
    H = H.reshape(3, 3)
    return np.array([
        H[0, p] * H[0, q],
        H[0, p] * H[1, q] + H[1, p] * H[0, q],
        H[1, p] * H[1, q],
        H[2, p] * H[0, q] + H[0, p] * H[2, q],
        H[2, p] * H[1, q] + H[1, p] * H[2, q],
        H[2, p] * H[2, q]
    ])


# 返回相机内参矩阵A
def get_in_param(H):
    # 构建V矩阵
    V = np.array([])
    for i in range(len(H)):
        V = np.append(V, np.array([create_v(0, 1, H[i]), create_v(0, 0, H[i]) - create_v(1, 1, H[i])]))

    # 求解V*b = 0中的b
    U, S, VT = np.linalg.svd((np.array(V, dtype='float')).reshape((-1, 6)))
    # 最小的奇异值对应的奇异向量,S求出来按大小排列的，最后的最小
    b = VT[-1]

    # 求取相机内参
    w = b[0] * b[2] * b[5] - b[1] * b[1] * b[5] - b[0] * b[4] * b[4] + 2 * b[1] * b[3] * b[4] - b[2] * b[3] * b[3]
    d = b[0] * b[2] - b[1] * b[1]

    alpha = np.sqrt(w / (d * b[0]))
    beta = np.sqrt(w / d ** 2 * b[0])
    gamma = np.sqrt(w / (d ** 2 * b[0])) * b[1]
    uc = (b[1] * b[4] - b[2] * b[3]) / d
    vc = (b[1] * b[3] - b[0] * b[4]) / d

    return np.array([
        [alpha, gamma, uc],
        [0, beta, vc],
        [0, 0, 1]
    ])

get_ex_param realizes the calculation of the external parameter matrix [R|t] of each picture from the homography matrix H, where R represents the rotation matrix and t represents the translation vector.

The specific implementation process is as follows:

First, invert the camera intrinsic parameter matrix, which is recorded as inv_intrinsics_param.

For the homography matrix H of each picture, three column vectors h0, h1, h2 are extracted.

Calculate the scaling factor scale_factor, which is calculated as the reciprocal of the 2-norm that multiplies inv_intrinsics_param with h0.

Calculate the three columns r0, r1, and r2 of the rotation matrix R according to the scaling factor, where r0 and r1 are multiplied by inv_intrinsics_param and h0, h1 respectively and multiplied by the scaling factor, and r2 is calculated by vector cross product.

Computes the translation vector t, which is calculated by multiplying inv_intrinsics_param by h2 and multiplying by the scaling factor.

Concatenate R and t into an external parameter matrix [R|t], and add it to the list ex_param.

After completing the above calculations for all homography matrices H, return the ex_param list.

create_v Given a homography matrix H and the subscripts of two points p and q, calculate the eigenvector v corresponding to these two points on the image. The feature vector v here is determined by the positional relationship between two points on the image, which reflects the geometric information between the two points, such as rotation, translation, scale, etc.

Specifically, this code implements the following steps:

Reconstruct the H matrix into a 3x3 form;

Calculate the feature vector v according to the subscripts of p and q, where v is a 6-dimensional vector;

returns v.

get_in_param realizes estimating the internal parameter matrix A of the camera from the homography matrix of multiple images, where A is a 3x3 matrix, including the focal length of the camera, the principal point position and the distortion parameters of the camera, etc.

The main process is as follows:

According to the input homography matrix H, construct V matrix, the size of V is 2n rows and 6 columns, where n is the number of input homography matrix H.

Perform singular value decomposition on V to obtain singular value matrix S and right singular vector matrix VT .

Take the right singular vector corresponding to the last singular value of S, that is, the last row of VT, as the solution b of Vb=0.

Calculate the camera internal parameter matrix A according to b.

Return the camera intrinsic parameter matrix A .

f. Distortion correction coefficient solution

#distortion.py

import numpy as np

# 返回畸变矫正系数k0,k1
def get_distortion(intrinsic_param, extrinsic_param, pic_coor, real_coor):
    D = []
    d = []
    for i in range(len(pic_coor)):
        for j in range(len(pic_coor[i])):
            # 转换为齐次坐标
            single_coor = np.array([(real_coor[i])[j, 0], (real_coor[i])[j, 1], 0, 1])

            # 利用现有内参及外参求出估计图像坐标
            u = np.dot(np.dot(intrinsic_param, extrinsic_param[i]), single_coor)
            [u_estim, v_estim] = [u[0] / u[2], u[1] / u[2]]

            coor_norm = np.dot(extrinsic_param[i], single_coor)
            coor_norm /= coor_norm[-1]

            # r = np.linalg.norm((real_coor[i])[j])
            r = np.linalg.norm(coor_norm)

            D.append(np.array([(u_estim - intrinsic_param[0, 2]) * r ** 2, (u_estim - intrinsic_param[0, 2]) * r ** 4]))
            D.append(np.array([(v_estim - intrinsic_param[1, 2]) * r ** 2, (v_estim - intrinsic_param[1, 2]) * r ** 4]))

            # 求出估计坐标与真实坐标的残差
            d.append(pic_coor[i][j, 0] - u_estim)
            d.append(pic_coor[i][j, 1] - v_estim)
         

    D = np.array(D)
    temp = np.dot(np.linalg.inv(np.dot(D.T, D)), D.T)
    k = np.dot(temp, d)

    return k

The distortion correction of the camera is realized, and the distortion correction coefficients k0 and k1 are returned.

The specific implementation process is as follows:

Input the camera's intrinsic parameter matrix (intrinsic_param), extrinsic parameter matrix (extrinsic_param), real image coordinates (real_coor) and distorted image coordinates (pic_coor).

For each real coordinate (real_coor[i])[j], convert it to a homogeneous coordinate and use the internal parameter matrix and external parameter matrix to find the estimated image coordinates (u_estim, v_estim).

Calculate the radius r according to the normalized coordinates (coor_norm) of the real coordinates, and calculate the distortion coefficient D by using the estimated image coordinates and the internal parameter matrix, and store it in the D array.

Calculate the residual of the estimated coordinates and the true coordinates and store them in the d array.

Calculate the distortion correction coefficients k0 and k1 by using the least squares method on the D array and the d array, and return the result.

g. Realization of camera calibration program

#refine_all.py

import numpy as np
import math
from scipy import optimize as opt


# 微调所有参数
def refinall_all_param(A, k, W, real_coor, pic_coor):
    # 整合参数
    P_init = compose_paramter_vector(A, k, W)

    # 复制一份真实坐标
    X_double = np.zeros((2 * len(real_coor) * len(real_coor[0]), 3))
    Y = np.zeros((2 * len(real_coor) * len(real_coor[0])))

    M = len(real_coor)
    N = len(real_coor[0])
    for i in range(M):
        for j in range(N):
            X_double[(i * N + j) * 2] = (real_coor[i])[j]
            X_double[(i * N + j) * 2 + 1] = (real_coor[i])[j]
            Y[(i * N + j) * 2] = (pic_coor[i])[j, 0]
            Y[(i * N + j) * 2 + 1] = (pic_coor[i])[j, 1]

    # 微调所有参数
    P = opt.leastsq(value,
                    P_init,
                    args=(W, real_coor, pic_coor),
                    Dfun=jacobian)[0]

    # raial_error表示利用标定后的参数计算得到的图像坐标与真实图像坐标点的平均像素距离
    error = value(P, W, real_coor, pic_coor)
    raial_error = [np.sqrt(error[2 * i] ** 2 + error[2 * i + 1] ** 2) for i in range(len(error) // 2)]

    print("total max error:\t", np.max(raial_error))

    # 返回拆解后参数，分别为内参矩阵，畸变矫正系数，每幅图对应外参矩阵
    return decompose_paramter_vector(P)


# 把所有参数整合到一个数组内
def compose_paramter_vector(A, k, W):
    alpha = np.array([A[0, 0], A[1, 1], A[0, 1], A[0, 2], A[1, 2], k[0], k[1]])
    P = alpha
    for i in range(len(W)):
        R, t = (W[i])[:, :3], (W[i])[:, 3]

        # 旋转矩阵转换为一维向量形式
        zrou = to_rodrigues_vector(R)

        w = np.append(zrou, t)
        P = np.append(P, w)
    return P


# 分解参数集合，得到对应的内参，外参，畸变矫正系数
def decompose_paramter_vector(P):
    [alpha, beta, gamma, uc, vc, k0, k1] = P[0:7]
    A = np.array([[alpha, gamma, uc],
                  [0, beta, vc],
                  [0, 0, 1]])
    k = np.array([k0, k1])
    W = []
    M = (len(P) - 7) // 6

    for i in range(M):
        m = 7 + 6 * i
        zrou = P[m:m + 3]
        t = (P[m + 3:m + 6]).reshape(3, -1)

        # 将旋转矩阵一维向量形式还原为矩阵形式
        R = to_rotation_matrix(zrou)

        # 依次拼接每幅图的外参
        w = np.concatenate((R, t), axis=1)
        W.append(w)

    W = np.array(W)
    return A, k, W


# 返回从真实世界坐标映射的图像坐标
def get_single_project_coor(A, W, k, coor):
    single_coor = np.array([coor[0], coor[1], coor[2], 1])

    # '''
    coor_norm = np.dot(W, single_coor)
    coor_norm /= coor_norm[-1]

    # r = np.linalg.norm(coor)
    r = np.linalg.norm(coor_norm)

    uv = np.dot(np.dot(A, W), single_coor)
    uv /= uv[-1]

    # 畸变
    u0 = uv[0]
    v0 = uv[1]

    uc = A[0, 2]
    vc = A[1, 2]

    # u = (uc * r**2 * k[0] + uc * r**4 * k[1] - u0) / (r**2 * k[0] + r**4 * k[1] - 1)
    # v = (vc * r**2 * k[0] + vc * r**4 * k[1] - v0) / (r**2 * k[0] + r**4 * k[1] - 1)
    u = u0 + (u0 - uc) * r ** 2 * k[0] + (u0 - uc) * r ** 4 * k[1]
    v = v0 + (v0 - vc) * r ** 2 * k[0] + (v0 - vc) * r ** 4 * k[1]
    '''
    uv = np.dot(W, single_coor)
    uv /= uv[-1]
    # 透镜矫正
    x0 = uv[0]
    y0 = uv[1]
    r = np.linalg.norm(np.array([x0, y0]))
    k0 = 0
    k1 = 0
    x = x0 * (1 + r ** 2 * k0 + r ** 4 * k1)
    y = y0 * (1 + r ** 2 * k0 + r ** 4 * k1)
    #u = A[0, 0] * x + A[0, 2]
    #v = A[1, 1] * y + A[1, 2]
    [u, v, _] = np.dot(A, np.array([x, y, 1]))
    '''

    return np.array([u, v])


# 返回所有点的真实世界坐标映射到的图像坐标与真实图像坐标的残差
def value(P, org_W, X, Y_real):
    M = (len(P) - 7) // 6
    N = len(X[0])
    A = np.array([
        [P[0], P[2], P[3]],
        [0, P[1], P[4]],
        [0, 0, 1]
    ])
    Y = np.array([])

    for i in range(M):
        m = 7 + 6 * i

        # 取出当前图像对应的外参
        w = P[m:m + 6]

        # 不用旋转矩阵的变换是因为会有精度损失
        '''
        R = to_rotation_matrix(w[:3])
        t = w[3:].reshape(3, 1)
        W = np.concatenate((R, t), axis=1)
        '''
        W = org_W[i]
        # 计算每幅图的坐标残差
        for j in range(N):
            Y = np.append(Y, get_single_project_coor(A, W, np.array([P[5], P[6]]), (X[i])[j]))

    error_Y = np.array(Y_real).reshape(-1) - Y

    return error_Y


# 计算对应jacobian矩阵
def jacobian(P, WW, X, Y_real):
    M = (len(P) - 7) // 6
    N = len(X[0])
    K = len(P)
    A = np.array([
        [P[0], P[2], P[3]],
        [0, P[1], P[4]],
        [0, 0, 1]
    ])

    res = np.array([])

    for i in range(M):
        m = 7 + 6 * i

        w = P[m:m + 6]
        R = to_rotation_matrix(w[:3])
        t = w[3:].reshape(3, 1)
        W = np.concatenate((R, t), axis=1)

        for j in range(N):
            res = np.append(res, get_single_project_coor(A, W, np.array([P[5], P[6]]), (X[i])[j]))

    # 求得x, y方向对P[k]的偏导
    J = np.zeros((K, 2 * M * N))
    for k in range(K):
        J[k] = np.gradient(res, P[k])

    return J.T


# 将旋转矩阵分解为一个向量并返回，Rodrigues旋转向量与矩阵的变换,最后计算坐标时并未用到，因为会有精度损失
def to_rodrigues_vector(R):
    p = 0.5 * np.array([[R[2, 1] - R[1, 2]],
                        [R[0, 2] - R[2, 0]],
                        [R[1, 0] - R[0, 1]]])
    c = 0.5 * (np.trace(R) - 1)

    if np.linalg.norm(p) == 0:
        if c == 1:
            zrou = np.array([0, 0, 0])
        elif c == -1:
            R_plus = R + np.eye(3, dtype='float')

            norm_array = np.array([np.linalg.norm(R_plus[:, 0]),
                                   np.linalg.norm(R_plus[:, 1]),
                                   np.linalg.norm(R_plus[:, 2])])
            v = R_plus[:, np.where(norm_array == max(norm_array))]
            u = v / np.linalg.norm(v)
            if u[0] < 0 or (u[0] == 0 and u[1] < 0) or (u[0] == u[1] and u[0] == 0 and u[2] < 0):
                u = -u
            zrou = math.pi * u
        else:
            zrou = []
    else:
        u = p / np.linalg.norm(p)
        theata = math.atan2(np.linalg.norm(p), c)
        zrou = theata * u

    return zrou


# 把旋转矩阵的一维向量形式还原为旋转矩阵并返回
def to_rotation_matrix(zrou):
    theta = np.linalg.norm(zrou)
    zrou_prime = zrou / theta

    W = np.array([[0, -zrou_prime[2], zrou_prime[1]],
                  [zrou_prime[2], 0, -zrou_prime[0]],
                  [-zrou_prime[1], zrou_prime[0], 0]])
    R = np.eye(3, dtype='float') + W * math.sin(theta) + np.dot(W, W) * (1 - math.cos(theta))

    return R

The purpose of calibrating the camera is to determine the internal parameters of the camera (including focal length, pixel spacing, principal point coordinates, etc.) and distortion parameters (including radial distortion and tangential distortion), so as to convert the image coordinates obtained from the camera into real-world coordinates.

The specific implementation process is as follows: according to the given real-world coordinates and corresponding image coordinates, using the least squares method, the camera internal parameters, distortion parameters and external parameters of each image are solved by an optimization algorithm to obtain the optimal camera calibration parameters. Among them, the camera intrinsic parameters and distortion parameters are shared by all images, while the extrinsic parameters of each image are calculated separately.

The program implements the following functions:

refinall_all_param: fine-tune all parameters

compose_paramter_vector: integrate all parameters into an array

decompose_paramter_vector: Decompose the parameter set to get the corresponding internal parameters, external parameters and distortion correction coefficients

get_single_project_coor: Returns image coordinates mapped from real-world coordinates according to camera intrinsics, distortion parameters, and extrinsics

h. Main function solution

import cv2
import numpy as np
import os

from parameter.homography import get_homography
from parameter.InExparam import get_ex_param,get_in_param
from parameter.distortion import get_distortion
from parameter.refine_all import refinall_all_param

def calibrate():
    # 求单应矩阵
    H = get_homography(pic_points, real_points_x_y)

    # 求内参
    intrinsics_param = get_in_param(H)

    # 求对应每幅图外参
    extrinsics_param = get_ex_param(H, intrinsics_param)

    # 畸变矫正
    k = get_distortion(intrinsics_param, extrinsics_param, pic_points, real_points_x_y)

    # 微调所有参数
    [new_intrinsics_param, new_k, new_extrinsics_param] = refinall_all_param(intrinsics_param,
                                                                             k, extrinsics_param, real_points,
                                                                             pic_points)

    print("intrinsics_parm:\t", new_intrinsics_param)
    print("distortionk:\t", new_k)
    print("extrinsics_parm:\t", new_extrinsics_param)


if __name__ == "__main__":
    file_dir = r'E:\pythonconda2\demarcate\calibration_pic'
    # 标定所用图像
    pic_name = os.listdir(file_dir)

    # 由于棋盘为二维平面，设定世界坐标系在棋盘上，一个单位代表一个棋盘宽度，产生世界坐标系三维坐标
    cross_corners = [7, 4]  # 棋盘方块交界点排列
    real_coor = np.zeros((cross_corners[0] * cross_corners[1], 3), np.float32)
    real_coor[:, :2] = np.mgrid[0:7, 0:4].T.reshape(-1, 2)

    real_points = []
    real_points_x_y = []
    pic_points = []

    for pic in pic_name:
        pic_path = os.path.join(file_dir, pic)
        pic_data = cv2.imread(pic_path)

        # 寻找到棋盘角点
        succ, pic_coor = cv2.findChessboardCorners(pic_data, (cross_corners[0], cross_corners[1]), None)

        if succ:
            # 添加每幅图的对应3D-2D坐标
            pic_coor = pic_coor.reshape(-1, 2)
            pic_points.append(pic_coor)

            real_points.append(real_coor)
            real_points_x_y.append(real_coor[:, :2])
    calibrate()

By calibrating a group of camera internal parameters (including focal length, principal point, etc.) and external parameters (including camera position and direction in space, etc.), the images captured by the camera can accurately reflect the geometric information of the real scene. In the specific implementation, the program establishes the corresponding relationship between the image and the real scene by finding the corner points of the checkerboard, and uses the corresponding points in multiple images to calculate the internal and external parameters of the camera, and also performs distortion correction and parameter Fine-tuning and other processing.

The specific process is as follows:

Define the folder path and checkerboard corner arrangement required for calibration.

Create a 3D world coordinate system, set the size of each grid, and generate corresponding coordinates.

Traverse the image folder used for calibration and read each image.

For each image, find the corner points of the chessboard through the findChessboardCorners function in OpenCV, and store them in the pic_points array.

Store the 3D coordinates of the world coordinate system in the real_points array.

Store the x and y coordinates of the world coordinate system in the real_points_x_y array.

Call the function calibrate() to calibrate. The implementation process of this function is as follows:

Obtain the corner coordinates of all pictures and the corresponding 3D coordinates of the world coordinate system through the findChessboardCorners function.

Use the get_homography function to get the homography matrix H of all images.

Use the get_in_param function to find the internal parameters.

Use the get_ex_param function to find the extrinsic parameters of each image.

Use the get_distortion function to correct the distortion of the image.

Use the refinall_all_param function to fine-tune intrinsic parameters, distortion parameters, and extrinsic parameters.

Output the internal reference, distortion parameter and external reference of the calibration result.

camera calibration error

a. Error source

The manufacturing accuracy of the checkerboard calibration board is not high, resulting in problems such as irregular grid size and inaccurate grid position.
Nonlinear distortion of camera imaging: The camera imaging process will be affected by optical factors, mechanical factors, etc., resulting in an error between the position of the pixel in the image and the actual position.
Camera attitude error: Camera calibration needs to take multiple pictures to calculate the internal and external parameters of the camera, and the difference in camera attitude when taking different pictures will also lead to errors.
Changes in the environment during the shooting process: such as lighting changes, camera movement, calibration board movement, etc., will affect the calibration results.

b. Error analysis

It can be analyzed by reprojection error and distortion corrected error. The reprojection error refers to applying the calibrated parameters to a new image to calculate the error between the projected position of the point in the new image and the position of the actual point.

The smaller the reprojection error, the more accurate the calibration result.

The distortion-corrected error is the error between the projected position of a point in the calculated new image and the position of the actual point after the distortion correction has been applied. If the error after distortion correction is smaller than the reprojection error, then the distortion correction has played a certain role.

c. Error correction

Improve the accuracy of the calibration board, making the size of the checkerboard more regular and the position of the grid more accurate.
Use a higher-precision camera to reduce the nonlinear distortion of camera imaging.
Improve the quality of captured pictures and ensure the stability of camera posture.
Carry out multiple sets of calibration, calculate the average value or optimize to improve the calibration accuracy.
For the distortion problem, distortion correction can be performed, and the distortion correction coefficient can be obtained through calculation, so as to correct the distortion of the image and reduce the influence of distortion on the calibration result.

reference article

I recommend everyone to take a look here. I am also learning for the first time. I refer to the articles of the big guys for learning, and find the information by myself to write.

(9 messages) Using Python-OpenCV and PS to make a checkerboard calibration board_opencv to create a chessboard_None072's blog-CSDN blog (7 messages) Computer Vision Tutorial 0-4: Manual Zhang Zhengyou Calibration Method, Detailed Image Distortion (with code)_Zhang Zhengyou Calibration Code_Mr.Winter`s Blog-CSDN Blog

(7 messages) (ultra-detailed) Zhang Zhengyou’s calibration principle and formula derivation_Zhang Zhengyou’s calibration formula reasoning_jiayuzhang128’s blog-CSDN blog The most detailed and complete explanation of camera calibration (6 messages) Zhang Zhengyou’s camera calibration principle and implementation _Zhang Zhengyou Calibration Method_Eating Lee's Blog-CSDN Blog

Industrial camera calibration (Zhang Zhengyou calibration method)

The concept of camera calibration

a. Definition of camera calibration