ChArUco calibration

Camera internal and external parameters and distortion parameters

ref1 jianshu
ref2 opencv

Distortion parameters

Cameras have radial distortion and tangential distortion.

Radial distortion:

Tangential distortion:

I don’t know where the formula came from, so I won’t worry about it for the time being. I only need to look at it when I’m writing a paper, and that’s it for engineering purposes.

Distortion coefficient

So the distortion coefficient is five parameters:

Camera internal parameters (fixed)

Camera coordinate system to image pixel coordinate system.

In fact, it is small hole imaging.
$f_x,f_y$is the focal length, $c_x,c_y$It is the optical center of the camera.

u,v are the image pixel plane coordinates, and Z is the distance from the point on the object to the optical center of the camera.

Camera external parameters

Refers to the rotation matrix and translation matrix. R&T.

Chessboard

ref
directly detects corners and performs camera calibration.

ArUco

Each block of ArUco is a binary code. The white block part of ChArUco uses binary encoding blocks.
ArUco can be used for pose estimation and calibration, but the effect is not as good as ChArUco.
ref
ok.

ChArUco

So what is ChArUco?
Introduction to opencv.
Detection of ChArUco Boards
A ChArUco board is a combination of Chessboard and ArUco.

Using the ArUco module, calibration can be performed based on ArUco markers corners or ChArUco corners. Calibrating using ArUco is much more versatile than using traditional chessboard patterns, since it allows occlusions or partial views.
As it can be stated, calibration can be done using both, marker corners or ChArUco corners. However, it is highly recommended using the ChArUco corners approach since the provided corners are much more accurate in comparison to the marker corners. Calibration using a standard Board should only be employed in those scenarios where the ChArUco boards cannot be employed because of any kind of restriction.
…
To calibrate using a ChArUco board, it is necessary to detect the board from different viewpoints, in the same way that the standard calibration does with the traditional chessboard pattern. However, due to the benefits of using ChArUco, occlusions and partial views are allowed, and not all the corners need to be visible in all the viewpoints.
——opencv

The yellow text describes the advantages of ChArUco. opencv recommends that the ChArUco board be used first for camera calibration. Only when the ChArUco board cannot be used anymore, the standard board or ArUco board should be used.

Calibration with ChArUco Boards

Detect markers

Code reference:
code Main reference
code Secondary reference
The version is not high. My opencv4.8.1 is higher than his, and I changed the names of several python APIs. Webpage 1 webpage 2 webpage 3 of
several opencv document web pages that may be frequently used show the detection effects from different angles. I have not included some detections with less than 6 grids. It seems that there are at least 6 grids.

corners, ids, rejectedImgPoints = aruco.detectMarkers(gray, aruco_dict, 
                                                      parameters=parameters)

Parameter reference

translation is:

Corners: The vector of the detected maker corner points. Each maker corner point is represented by 4 corner points, and the 4 corner points are arranged clockwise. Like this:
array([[155. , 383. ], [224. , 381. ], [208.58003, 402.61102], [132.88284, 405.2367 ]])
ids: maker's id, the value ranges from 0 to N, N is the total number of makers. The order is the same as the pixel order of the imgPoints array (input image).
rejectedImgPoints: seems to be incorrect points. Easy to debug.

Get corners

After detectMarkers gets Markers, ids,

retval, charucoCorners, charucoIds = cv2.aruco.interpolateCornersCharuco(corners,ids,gray,board) 

This function receives the detected markers and returns the 2D position of the chessboard corners from a ChArUco board using the detected Aruco markers. If camera parameters are provided, the process is based in an approximated pose estimation, else it is based on local homography. Only visible corners are returned. For each corner, its corresponding identifier is also returned in charucoIds. The function returns the number of interpolated corners.
Translation: This function receives the makers detected by detectMakers and returns the two-dimensional corner point coordinates of ChArUco. If camera parameters are provided, this function is calculated based on pose estimation, otherwise it is based on local homography. link

But I still don’t know cv2.aruco.interpolateCornersCharuco(corners,ids,gray,board)what I did.
Check cv2.aruco.interpolateCornersCharuco(corners,ids,gray,board)the number of corners before and after, some increase and some decrease.

use

aruco.drawDetectedCornersCharuco

Draw DetectedCornersCharuco:

This way you know that interpolateCornersCharuco detects the corners.
So I thought detectMarkers got corners, which is not entirely true. Corner is the corner of markers, so it is marker_corner.

That is, detectMarkers gets markers.
interpolateCornersCharuco gets Corners.
I won’t look at the detailed code. (Read it if you are interested).
The boss asked me to study the impact of the number of grids, the size of markers and the number of detections on the calibration results later.

But how is the algorithm principle of this part done?

The above are all parts of corners detection, and subsequent calibration is performed based on the detected corners.

Single target setting

calibrateCameraCharuco
ref1：Zhengyou Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330–1334, 2000.
ref2：Jean-Yves Bouguet. Camera calibration tool box for matlab [eb/ol], 2004.

Zhang Zhengyou calibration method:

The paper is only five pages, including a page of references, all full of mathematical formulas.
Awesome articles are just that, no background, no relevant technology, no experiments, just a few simple formulas.
There is a lot of content here, so let’s leave a hole first.