When using aruco markers for pose estimation, the original internal parameter matrix obtained by cv2.calibrateCamera() should be used instead of the dedistorted internal parameter matrix obtained by cv2.getOptimalNewCameraMatrix().
The reasons are as follows: 1. The aruco.estimatePoseSingleMarkers() function needs to pass in the original camera internal parameter matrix, that is, the internal parameter matrix without past distortion operations. This internal parameter matrix will be used to calculate the coordinates of the marker in the image during marker detection.
2. The internal parameter matrix obtained by cv2.getOptimalNewCameraMatrix() has been de-distorted, and the coordinates have been transformed to the de-distorted image plane. This internal parameter matrix is not suitable for direct use in aruco's attitude estimation.
3. The coordinates after marker detection are still on the original image. It is necessary to use the original camera internal parameter matrix to convert them into the camera coordinate system, and then compare them with the true size of the marker to estimate the external parameters.
4. The general process is to first detect the mark to obtain the 2D coordinates, and then use the original internal parameter matrix to calculate the 3D coordinates of the mark in the camera coordinate system to restore the posture.
So the correct method is: use the original internal parameter matrix estimated by cv2.calibrateCamera() as the internal parameter matrix parameter of the aruco.estimatePoseSingleMarkers() function to perform correct pose estimation.
During pose estimation, the original internal parameter matrix self.mtx should continue to be used, and new_camera_mtx should only be used when de-distortion display is required.