Mainly use cv::solvepnp and cv::solvePnPRansac or calibrate to solve the camera external parameters, but the calculation of the external parameters of the fisheye camera is called
Reference link:
I:
You are an opencv expert, teach me to calculate the external parameters of the fisheye camera, and implement it with c++ code
openai(chatgpt4):
Calculating the extrinsic parameters of a fisheye camera requires two steps: camera calibration and pose estimation. The following are some basic steps and C++ code implementation.
1. Camera Calibration
Camera calibration is the process for estimating the intrinsic parameters and distortion coefficients of a camera. This can be achieved by taking a set of images with known 3D world coordinates and corresponding 2D image coordinates. OpenCV provides a function cv::fisheye::calibrate()for calibrating fisheye cameras. Here is a sample code:
cv::Mat cameraMatrix, distCoeffs;
std::vector<std::vector<cv::Point3f>> objectPoints;
std::vector<std::vector<cv::Point2f>> imagePoints;
cv::Size imageSize;
double rms = cv::fisheye::calibrate(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, cv::noArray(), cv::noArray());
Among them objectPointsis one vector<vector<Point3f>>, which contains the coordinates of all three-dimensional points on the calibration plate, imagePointsand is one vector<vector<Point2f>>, which contains the two-dimensional coordinates of these points in the calibration image. imageSizeis the size of the calibration image. cameraMatrixand distCoeffsare the output camera intrinsic parameters and distortion coefficients. rmsis the calibration error calculated from the calibration results.
2. Pose Estimation
Pose estimation is moving the camera from its initial position to a new position and determining the rotation and translation of the camera at that position. Here, we can use OpenCV's solvePnP function for pose estimation. Here is a sample code:
cv::Mat rvec, tvec;
std::vector<cv::Point3f> objectPoints3D;
std::vector<cv::Point2f> imagePoints2D;
cv::Mat cameraMatrix, distCoeffs;
cv::fisheye::projectPoints(objectPoints3D, rvec, tvec, cameraMatrix, distCoeffs, imagePoints2D);
Among them objectPoints3Dis a vector<Point3f>, which contains the coordinates of the three-dimensional points on the calibration plate, imagePoints2Dand is a vector<Point2f>, which contains the two-dimensional coordinates of these points in the current image. cameraMatrixand distCoeffsare the camera intrinsic parameters and distortion coefficients obtained in camera calibration. rvecand tvecare the output rotation vector and translation vector.
Using solvePnP function, we can get rotation vector and translation vector. Converting them into rotation matrix and translation matrix, the extrinsic parameters of the camera can be obtained. Here is a sample code:
cv::Mat R;
cv::Rodrigues(rvec, R);
cv::Mat T = tvec;
3. Calculating the external parameters of the fisheye camera can be divided into the following steps:
1) Calibrate the internal parameters and distortion parameters of the fisheye camera. This step can be done using functions in OpenCV calibrateCamera.
2) Collect the image of the fisheye camera, and use the corner detection algorithm (such as findChessboardCornersthe function in OpenCV) to detect the corners of the calibration board.
3) Calculate the external parameters of the fisheye camera according to the known spatial points on the calibration board and the corresponding image points. This step can be done using functions in OpenCV solvePnP.
4) Convert the obtained rotation vector and translation vector into a rotation matrix and a translation matrix to obtain the external parameter matrix of the fisheye camera.
Here is an example of implementing the above steps in C++ code:
#include <opencv2/opencv.hpp>
#include <iostream>
using namespace std;
using namespace cv;
int main()
{ // Read the calibration board image Mat boardImg = imread("board.png"); // Set the calibration board size and spatial point coordinates Size boardSize = Size(9, 6); vector<Point3f> objectPoints; for (int i = 0; i < boardSize. height; i++) { for (int j = 0; j < boardSize. width; j++) { objectPoints. push_back(Point3f(i, j, 0)); } }
// 标定相机内参和畸变参数
Mat cameraMatrix, distCoeffs;
vector<vector<Point2f>> imagePoints;
vector<Mat> rvecs, tvecs;
vector<float> reprojErrs;
double rms = calibrateCamera(vector<vector<Point3f>>{objectPoints}, vector<vector<Point2f>>{imagePoints},
boardImg.size(), cameraMatrix, distCoeffs, rvecs, tvecs);
// Detect the corners of the calibration board
vector<Point2f> corners;
bool found = findChessboardCorners(boardImg, boardSize, corners);
// Calculate camera extrinsic parameters
Mat rvec, tvec;
solvePnP(objectPoints, corners, cameraMatrix, distCoeffs, rvec, tvec);
// Convert rotation and translation vectors to rotation and translation matrices
Mat R;
Rodrigues(rvec, R);
Mat T = tvec.reshape(1, 3);
// Get the camera's extrinsic matrix
Mat extrinsicMat = Mat::eye(4, 4, CV_32F);
R.copyTo(extrinsicMat(Rect(0, 0, 3, 3)));
T.copyTo(extrinsicMat(Rect( 3, 0, 1, 3)));
cout << "Camera internal parameters:" << endl << cameraMatrix << endl;
cout << "Camera distortion parameters: " << endl << distCoeffs << endl;
cout << "Camera external parameters:" << endl < < extrinsicMat << endl;
return 0;
}
4. Calculating the external parameters of the fisheye camera can be divided into the following steps:
1) Acquire images and corresponding 3D coordinates. A calibration plate and known 3D coordinates can be used to acquire multiple images and corresponding 3D coordinates.
2) Perform camera calibration to obtain internal reference matrix and distortion coefficients. This can be achieved using the calibrateCamera function of OpenCV.
3) Transform the 3D coordinates into the camera coordinate system through the multi-view geometry theory.
4) Convert the 3D points in the camera coordinate system to the world coordinate system by solving the external parameter matrix.
The following is a sample code implemented in C++ for calculating the external parameters of the fisheye camera:
#include <iostream>
#include <opencv2/opencv.hpp>
using namespace std;
using namespace cv;
int main()
{ // Read the calibration board image and the corresponding 3D coordinates vector<vector<Point3f>> objectPoints; vector<vector<Point2f>> imagePoints; // ...
// Perform camera calibration
Mat cameraMatrix, distCoeffs;
vector<Mat> rvecs, tvecs;
calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs);
// Define 3D points and 2D points
vector<Point3f> objectPoints3D;
vector<Point2f> imagePoints2D;
// ...
// Convert 3D points to the camera coordinate system
Mat objectPointsMat(objectPoints3D);
Mat rvec, tvec;
solvePnP(objectPointsMat, imagePoints2D, cameraMatrix, distCoeffs, rvec, tvec);
// Convert the 3D point in the camera coordinate system to the world coordinate system
Mat R;
Rodrigues(rvec, R);
Mat extrinsicMat(3, 4, CV_64FC1);
hconcat(R, tvec, extrinsicMat);
// Output extrinsic matrix
cout << extrinsicMat << endl;
return 0;
}
Among them, the solvePnP function is used to solve the external parameter matrix, the Rodrigues function is used to convert the rotation vector into a rotation matrix, and the hconcat function is used to combine the rotation matrix and translation vector into a 3×4 external parameter matrix. Please modify the variables and parameters in the code according to the actual situation.
5 Calculating the external parameters of the fisheye camera requires the following steps:
1) Calculate the internal reference matrix of the camera, including parameters such as focal length, principal point, and distortion coefficient. These parameters can usually be obtained through documentation provided by the camera manufacturer or camera calibration.
2) Collect images from at least 4 different locations, and each location needs to take at least 3 images from different orientations. These images can be used to calculate the extrinsics of the camera.
3) To extract the feature points in each image, a feature point detection algorithm such as ORB or SURF can be used.
4) For each feature point, project it onto a spherical surface through the fisheye model. For details, please refer to the fisheye module of OpenCV.
5) Use algorithms such as RANSAC to estimate the extrinsic parameters of each image. For details, please refer to the solvePnP function of OpenCV.
6) Combine the external parameters of all images, and use algorithms such as Bundle Adjustment to optimize the results.
The following is a simple C++ code example that can calculate the external parameters of the fisheye camera:
#include <iostream>
#include <vector>
#include <opencv2/opencv.hpp>
#include <opencv2/features2d.hpp>
#include <opencv2/xfeatures2d.hpp>
#include <opencv2/core/types.hpp>
#include <opencv2/calib3d.hpp>
#include <opencv2/core/eigen.hpp>
using namespace std;
using namespace cv;
using namespace cv::xfeatures2d;
void calculateExtrinsics(const vector<vector<Point2f>>& imagePoints,
const vector<vector<Point3f>>& objectPoints,
const Mat& cameraMatrix,
const Mat& distCoeffs,
Mat& R,
Mat& t)
{
vector<Mat> rvecs, tvecs;
solvePnPRansac(objectPoints, imagePoints, cameraMatrix, distCoeffs, rvecs, tvecs);
Rodrigues(rvecs[0], R);
t = tvecs[0];
}
int main()
{
// load images and corresponding object points
vector<vector<Point2f>> imagePoints;
vector<vector<Point3f>> objectPoints;
// TODO: load imagePoints and objectPoints from file or capture them from camera
// calculate camera matrix and distortion coefficients
Mat cameraMatrix, distCoeffs;
// TODO: calculate cameraMatrix and distCoeffs using calibration data
// calculate extrinsics for each image
vector<Mat> Rs, ts;
for (int i = 0; i < imagePoints.size(); i++) {
Mat R, t;
calculateExtrinsics(imagePoints[i], objectPoints, cameraMatrix, distCoeffs, R, t);
Rs.push_back(R);
ts.push_back(t);
}
// bundle adjustment to refine extrinsics
TermCriteria criteria(TermCriteria::COUNT + TermCriteria::EPS, 100, 1e-5);
vector<Mat> Rs_opt, ts_opt;
vector<vector<Point2f>> imagePoints_opt;
bundleAdjustment(objectPoints, imagePoints, cameraMatrix, distCoeffs, Rs, ts, Rs_opt, ts_opt, imagePoints_opt, criteria);
// print results
for (int i = 0; i < Rs_opt.size(); i++) {
5.
Calculating the extrinsic parameters of the fisheye camera requires knowing the intrinsic parameters of the camera and some known scene information. The following are the steps to realize the calculation of the external parameters of the fisheye camera in C++:
1) Load the image and extract the feature points in the image. You can use cv::KeyPoint and cv::FeatureDetector in OpenCV to extract feature points in the image.
2) Select several known 3D points in the scene and calculate their coordinates in the camera coordinate system. These 3D points should be static and visible to the camera. You can use the cv::solvePnP function in OpenCV to calculate the transformation relationship between the camera coordinate system and the image coordinate system, that is, the external parameters of the camera.
3) Calculate the distortion parameters of the camera. You can use the cv::calibrateCamera function in OpenCV to calculate the internal parameters and distortion parameters of the camera. It is necessary to provide a set of known 3D points and their projections in the image, as well as the internal parameters of the camera.
4) Combine the internal and external parameters of the camera to obtain the projection matrix of the camera.
5) Use the projection matrix of the camera to project the 3D points in the scene into the image, and match them with the actually extracted feature points. You can use the cv::findHomography function in OpenCV to calculate the transformation relationship between the camera's projection matrix and feature points.
The following is a sample code for calculating the extrinsic parameters of a fisheye camera:
#include <opencv2/opencv.hpp>
using namespace cv;
int main()
{ // load image Mat image = imread("image.jpg");
// Extract the feature points in the image
Ptr<FeatureDetector> detector = FastFeatureDetector::create();
std::vector<KeyPoint> keypoints;
detector->detect(image, keypoints);
// Select several known 3D points in the scene
std::vector<Point3f> objectPoints;
objectPoints.push_back(Point3f(0, 0, 0));
objectPoints.push_back(Point3f(1, 0, 0));
objectPoints.push_back(Point3f(0, 1, 0));
objectPoints.push_back(Point3f(0, 0, 1));
// Calculate the camera's external parameters
Mat cameraMatrix = Mat::eye(3, 3, CV_64F); // Camera internal parameters
Mat distCoeffs = Mat::zeros(1, 5, CV_64F); // Distortion parameters
Mat rvec, tvec;
solvePnP(objectPoints, keypoints, cameraMatrix, distCoeffs, rvec, tvec);
// Calculate the camera's projection matrix
Mat R;
Rodrigues(rvec, R);
Mat projectionMatrix = Mat::zeros(3, 4, CV_64F);
hconcat(R, tvec, projectionMatrix.rowRange(0, 3).colRange(0 , 4));
// Project 3D points to the graph
6.
To calculate the external parameters of the fisheye camera, the internal parameters and distortion coefficients of the camera need to be obtained first, and these parameters can be obtained through camera calibration. Assuming that these parameters have been obtained, the functions in the OpenCV library can be used to calculate the external parameters of the fisheye camera.
First, you need to prepare at least 6 sets of correspondences between 2D points and 3D points. These correspondences can be obtained by manually marking or using feature points detected by other algorithms (such as SIFT, SURF, etc.). Here, we assume that these correspondences are already in place.
The following is a sample code that uses the OpenCV library function to calculate the extrinsic parameters of the fisheye camera:
#include <opencv2/opencv.hpp>
#include <iostream>
using namespace std;
using namespace cv;
int main()
{ // Read camera internal parameters and distortion coefficient Mat K, D; FileStorage fs("calibration.xml", FileStorage::READ); fs["K"] >> K; fs["D"] >>D;
// Read the correspondence between 3D points and 2D points
vector<Point3f> objectPoints;
vector<Point2f> imagePoints;
FileStorage fs2("points.xml", FileStorage::READ);
fs2["objectPoints"] >> objectPoints;
fs2["imagePoints"] >> imagePoints;
// Calculate external parameters
Mat rvec, tvec;
solvePnP(objectPoints, imagePoints, K, D, rvec, tvec);
// print rotation vector and translation vector
cout << "Rotation Vector: " << rvec << endl;
cout << "Translation Vector: " << tvec << endl;
return 0;
}
In this sample code, we first read the internal parameters and distortion coefficients of the camera, and then read the correspondence between 3D points and 2D points. Next, use the function in OpenCV solvePnPto calculate the extrinsic parameters of the camera, and finally print out the rotation vector and translation vector.
It should be noted that when reading the camera internal parameters and distortion coefficients and the corresponding relationship between 3D points and 2D points, you need to modify it according to your own file format. In addition, due to the relatively large distortion of the fisheye camera, it is recommended to use at least 10 sets of correspondences to calculate the external parameters.
Summarize:
Before finding the external parameters of the fisheye camera, it is necessary to determine the internal parameters + distortion parameters of the fisheye camera. You can use the solvePnP interface to use 3d coordinates, 2d coordinates, internal parameters + distortion parameters to find the external parameters of the fisheye camera, or you can first use undistortPoints to use internal parameters + distortion parameters to first undistort the 2d coordinates, and then use the undistorted 2d coordinates and 3d coordinates Find the external parameters of the fisheye camera through the solvePnP interface.