Robot hand-eye calibration and reprojection based on OpenCV-Python (9-point calibration method)
Article Directory
Foreword:
The previous article was reprinted and wrote the principle of nine-point calibration.
This article is to record how to calibrate, and calculate the calibration parameters and reprojection accuracy.
It seems that I haven't searched the related keywords of "9-point calibration" and "re-projection" in the Chinese community.
But in Zhang Zhengyou's calibration method, there is a concept of reprojection error.
The calculated transformation matrix M is substituted into the transformation formula to calculate the error between the new target coordinate and the original coordinate.
Expressed by the formula:
new_target_pos = origin_pos*M
e = MSE(new_target_pos, target_pos)
Intuitively, if our calibration parameters cover all samples and fit by the least square method, then the error obtained should be relatively small.
So I borrowed the commonly used methods of deep learning and divided the training set and test set.
And tested the impact of different training samples on the results.
The result is amazing!
- In fact, this function can be used with 3 points, but there must be at least 3 points, because 1 point has two equations, and there are 6 unknowns.
There may be some abnormal points in the data set, and the abnormal points have a great influence on the results! - As long as you find the right data points, the results will be better if optimization is not optimized.
- If there is a particularly large error, then use a little more data, the effect is also good.
- The reprojection error is 0.4mm or 0.4 pixels.
experiment process:
Eye-to-Hand Calibration: The
camera is fixed, and the relative position to the robot base coordinate system remains unchanged. And the end of the robot moves on a fixed plane, that is, only a transformation relationship of a homography matrix is required.
The experiment process is as follows:
- Set up the scene of the hand-eye system: the camera is fixed, and the robotic arm drives the needle tip to move on a fixed plane.
- Calibration sample collection. Including the camera image acquisition, and the corresponding robot joint configuration acquisition. –Calibration_data_collected_main.py
- The image processing extracts the calibration needle tip, and calculates the coordinates of the needle tip in the robot coordinate system. Record the tip pixel coordinates, tip world coordinates, and end coordinates of each location point
- Calculate the calibration parameter matrix M--calibration_class.py
- Calculate the reprojection error avg_e--calibration_class.py
The main environment configuration and tools used in the calibration experiment are:
-
Operating system: Windows 7 64bit
-
Image processing tool: OpenCV-Python 3.4.* If it can’t be installed, the version is 4.* or higher, and estimated2DAffine seems to work. I haven’t tested it.
-
Robot and camera: Siasun SCR5 seven-degree-of-freedom collaborative robotic arm, Hikvision industrial camera MV-CA013-21UC
The calibration_class.py can be used alone. As long as there is an independent set of calibration points.
Then go to the code?
Code:
Hang my github link first, this code is a bit too much.
https://github.com/kaixindelele/Eye-to-Hand-Calibration
If it helps, remember to order a star~
Core code:
Calculate the conversion matrix m
def get_m(self,
origin_points_set,
target_points_set):
# 确保两个点集的数量级不要差距过大，否则会输出None,看到这个输出，我直接好家伙。
# 明明回家前，还能输出一个好的转换矩阵，为什么一回家就报错？我错哪儿了...
m = cv2.estimateRigidTransform(origin_points_set,
target_points_set,
fullAffine=True)
return m
Reprojection error calculation:
def reproject(self,
origin_points_set,
target_points_set,
m):
error_list = []
for index in range(len(origin_points_set)):
p_origin = list(origin_points_set[index])
p_origin.append(1)
p_origin = np.array(p_origin)
p_tar = target_points_set[index]
new_tar = np.dot(m, p_origin)
error = np.linalg.norm(new_tar-p_tar[:2])
error_list.append(error)
print("avg_e:", np.mean(np.array(error_list)))
return np.mean(np.array(error_list))
Reprojection error-training sample number test:
Data analysis: the
horizontal axis is the number of training samples, starting from 9 and ending at 29, and the total sample size is 37; the
vertical axis is the error size, in millimeters;
title explanation: xy is the needle tip coordinates, uv is the needle pixel pixel coordinates, xy2uv Means to calculate the m in uv=m xy through nine-point calibration, and then substitute m into the test sample to calculate the new uv_pred_test=m xy_test, and the calculation error e=mse(uv_pred, uv_origin_test)
if it is mt, then it is , After calculating m by uv=m*xy, use a system of linear equations similar to the following:
t_rx= (A * t_px) + B * t_py + C);
t_ry= (D * t_px) + E * t_py + F) ；
Solve the mt in xy=mt*uv. Then calculate the reprojection error.
The results have been analyzed in the preface.
to sum up:
With so many records, in fact, the core content is nothing innovative.
As for the reprojection error of the nine-point calibration, I saw that there was an article in ICRA in 19 years that was actually used.
But they have an eye-in-hand structure. The two robotic arms are much more complicated in structure and principle than mine...
My small project took me close to a week. Where did the time go?
I reviewed it:
One is the doubt about the entire calibration process. I have never calibrated this kind of hand-eye structure, nor have I found a ready-made toolkit.
I didn't understand the entire information flow, so I didn't even know what data to collect at the beginning.
The second is that we do not have a calibration board. For the detection of feature points, we have to use the tip detection program written by myself. I have adjusted this program for a long time. Finally, using the scene information, I wrote an optimized and better detection module, which can be in 0.04 seconds. At the speed of one frame, an error within 2 pixels is achieved.
Then there is this rubbish OpenCV version problem, versions above 4 do not have cv2.estimateRigidTransform. I can only find the 3.4 version.
Finally, there is the problem of floating-point number accuracy, because the range of pixel coordinates is 0-280, and the range of initial needle-point coordinates is 0-0.8, which is three orders of magnitude difference. The point set is directly brought into the function. Different computers The return value is different for different hardware. I can get a matrix m with a very small value on the desktop computer, but in my laptop, the return value is None with the same code and data? I wondered if my laptop was broken.
Do a test: Test
on Google's colab, the maximum accuracy is float128 The
test script is very simple:
import numpy as np
print(np.finfo(np.longdouble))
out:Machine parameters for float128
---------------------------------------------------------------
precision = 18 resolution = 1e-18
machep = -63 eps = 1.084202172485504434e-19
negep = -64 epsneg = 5.42101086242752217e-20
minexp = -16382 tiny = 3.3621031431120935063e-4932
maxexp = 16384 max = 1.189731495357231765e+4932
nexp = 15 min = -max
And the maximum accuracy of my notebook is float64. Interesting, this pit must be remembered, maybe when it will be pitted by this thing.
Machine parameters for float64
---------------------------------------------------------------
precision = 15 resolution = 1.0000000000000001e-15
machep = -52 eps = 2.2204460492503131e-16
negep = -53 epsneg = 1.1102230246251565e-16
minexp = -1022 tiny = 2.2250738585072014e-308
maxexp = 1024 max = 1.7976931348623157e+308
nexp = 11 min = -max