Since the measurement of an object needs to be implemented, but there is already a QT program, the final overall function needs to be integrated and implemented in C#.
First there are two options: (1) Use existing QT programs and interfaces to directly call QT or C++ programs in C#. However, after trying, we found that calling between the two is not that simple and involves many variable definitions. Not used and the data structure is different. Therefore, we decided on option (2) and re-implemented this function in C#.
Since this is my first time using a camera, I will take this opportunity to record it.
1. The first is camera calibration. This is very simple and there are a lot of related references:
Camera Calibration (1) - Internal Parameter Calibration and Program Implementation_Camera Internal Parameter Calibration_white_Learner's Blog-CSDN BlogCamera Calibration (1) - Internal Parameter Calibration and Program Implementation of the Camera Calibration (2) - Conversion of image coordinates and world coordinates Camera calibration (3) - Hand-eye calibration 1. Zhang Zhengyou calibration algorithm implementation process 1.1 Prepare the checkerboard Note: The black and white spacing of the checkerboard is known, you can use printing paper or purchase a black and white checkerboard for calibration Board (high accuracy requirements) 1.2 Take several pictures of the chessboard pattern. There are two situations here (1) Calibrating the distortion coefficient and camera internal parameters. The photos need to include the complete chessboard, and at the same time, different distances and different orientations are required... 
https://blog.csdn.net/Kalenee/article/details/80672785
Because OpencvSharp is originally a C# dynamic link library encapsulated by Opencv, the method used is essentially the same as the basic function implementation. You can directly refer to the C++ version of camera calibration. You only need to pay attention to modifying the specific physical size of the calibration grid and the number of calibration grids when using it personally.
2. Convert pixel coordinates to world coordinates
After calibrating the camera, you need to implement how to convert pixel coordinates to world coordinates. For the theory and implementation process, please refer to the C++ version.
public static Point3d GetWorldPoints(Point2f inPoints)
{
double s;
int zConst = 0;
Mat r = new Mat(3, 3, MatType.CV_64FC1);
Mat t = new Mat(3, 1, MatType.CV_64FC1);
using (var fs1 = new FileStorage("RT.yaml", FileStorage.Mode.Read))
{
r = (Mat)fs1["R"];
t = (Mat)fs1["t"];
}
Mat R_invert = new Mat(3, 3, MatType.CV_64FC1);
Mat cameraMatrix_invert = new Mat(3, 3, MatType.CV_64FC1);
Mat imagePoint = new Mat(3, 1, MatType.CV_64FC1);
imagePoint.At<double>(0, 0) = inPoints.X;
imagePoint.At<double>(1, 0) = inPoints.Y;
imagePoint.At<double>(2, 0) = 1;
Mat cameraMatrix = new Mat(3, 3, MatType.CV_64FC1, new double[3, 3] {
{ 1473.819, 0, 615.859 },
{ 0, 1474.14, 467.697 },
{ 0, 0, 1 } });
Cv2.Invert(r, R_invert, DecompTypes.SVD);
Cv2.Invert(cameraMatrix, cameraMatrix_invert, DecompTypes.SVD);
Mat tempMat = R_invert * cameraMatrix_invert * imagePoint;
Mat tempMat2 = R_invert * t;
s = zConst + tempMat2.At<double>(2, 0);
s /= tempMat.At<double>(2, 0);
//计算世界坐标
Mat wcPoint;
wcPoint = R_invert * ( cameraMatrix_invert*s * imagePoint - t);
Point3d world;
world.X = wcPoint.At<double>(0, 0);
world.Y = wcPoint.At<double>(1, 0);
world.Z = wcPoint.At<double>(2, 0);
//Point3f worldPoint(wcPoint.at<double>(0, 0), wcPoint.at<double>(1, 0), wcPoint.at<double>(2, 0));
return world;
}It can be seen that it can almost be achieved with the C++ version.
What needs to be noted here is the Cv2.Invert() matrix inversion function. Depending on its third parameter DecompTypes, the inversion method is also different. However, the results obtained by my own test are only more accurate using the DecompTypes.SVD method. I wonder if you have any experience in choosing the inverse method.
However, after the function was written, testing the pixel coordinates to the world coordinates still failed to meet the requirements. After the guidance of my senior brother, it was because the external parameters (rotation matrix and translation vector) I used here were wrong. The reason was that I used Cv2 directly. The external parameters obtained by .CalibrateCamera() are used for calculation. However, the shooting conditions have changed during my experiment, and the external parameters at that time are no longer applicable. Therefore, it is necessary to obtain the external parameters of the current camera before solving. Making this mistake means not having a good overall understanding of the entire implementation process and the camera coordinate conversion theory, which leads to problems with the correspondence of external parameters to objects. Therefore, the following is to solve the external parameters of the current camera.
3. Solving external parameters
Since the known object pixel coordinates need to be converted into world coordinates, the changes in the position and posture of the camera and the object at this time are required. Because solving the external parameters can infer the actual world coordinates of the object.
It should be noted that my actual environment is where the camera is stationary, that is, the camera is always parallel to the ground and the position of the camera is fixed. Therefore, if it needs to be used when the camera moves or the relative position changes at any time, it needs to be modified.
The functions used are also very simple. SolvePnP that comes with OpencvSharp is used for solution.
As you can see from the parameters of the function, we need to prepare objectPoints, imagePoints, Intrinsic, distCoeffs, rvec, tvec, SolvePnPFlags. I think these are the more critical parameters.
The first is objectPoints: a set of world coordinates needs to be provided
ImagePoints: A set of pixel coordinates corresponding to world coordinates needs to be provided
Intrinsic, distCoeffs: camera’s internal parameters and distortion coefficients
rvec, tvec: Output a rotation vector and translation vector. Usually the rotation vector needs to be converted into a rotation matrix using Cv2.Rodrigues()
SolvePnPFlags: Solution method selection, input points for different parameter selection requirements also have requirements, you can refer to related articles to make selections
Here I use two methods to obtain the values of objectPoints and imagePoints.
One is to use calibration paper, find the corner coordinates of the calibration paper through a program, and then use a ruler to actually measure the world coordinates of each corner point. What needs to be noted here is the selection of the X and Y axes of the world coordinates, based on the corner points. The upper left corner is the origin, forming the X-axis to the right and the Y-axis downward. Be sure to pay attention to the correspondence between world coordinate points and pixel coordinate points.
It can be seen from the obtained corner points that my calibration paper has five horizontal corner points and four vertical corner points, and the corner point coordinates obtained through Cv2.FindChessboardCorners() are based on the upper left corner of the picture as the origin of the coordinates, to the right Form the X axis and go down to form the Y axis. And the order of stored points is starting from the first point in the upper left corner, from left to right, from top to bottom, that is, the index of the first row of corner points is 0, 1, 2..., the second Platoon starts at 5.
The following is the solution process
public static void GetRvec()
{
double meanDistance = 0;
double sumDistance = 0;
int numPairs = 0;
Mat Intrinsic = new Mat(3, 3, MatType.CV_32FC1, new double[] { 1473.81, 0, 615.85 , 0, 1474.14, 467.69 , 0, 0, 1 });
Mat distCoeffs = new Mat(5,1, MatType.CV_32FC1, new double[] { 0.051, 0.44, -0.01, -0.009, -2.22 });
string calibImagesPath = "标定图片/"; // 标定图片所在目录
int boardWidth = 5; // 棋盘格宽度（内角点个数）
int boardHeight = 4; // 棋盘格高度（内角点个数）
float squareSize = 36.1F; // 棋盘格单个方格的边长（毫米）
int k=0;
Point3d[] objectPoints = new Point3d[20];
Point2d[] ww = new Point2d[20] ;
for (int i = 0; i < boardHeight; i++)
{
for (int j = 0; j < boardWidth; j++)
{
// objectPoints.Add(new Point3d(j * squareSize, i * squareSize, 0));
objectPoints[k].X = j * squareSize;
objectPoints[k].Y = i * squareSize;
objectPoints[k].Z = 0;
ww[k].X = j * squareSize;
ww[k].Y = i * squareSize;
k++;
}
}
// 提取图像中的角点
Mat gray = new Mat();
Cv2.CvtColor(calibImage, gray, ColorConversionCodes.BGR2GRAY);
Point2f[] corners;
Point2f[] imagepoint = new Point2f[4];
bool found = Cv2.FindChessboardCorners(gray, new Size(boardWidth, boardHeight), out corners);
if (found)
{
TermCriteria criteria = new TermCriteria(CriteriaType.MaxIter | CriteriaType.Eps, 30, 0.001);
Cv2.CornerSubPix(gray, corners, new Size(11, 11), new Size(-1, -1), criteria);
Cv2.Find4QuadCornerSubpix(gray, corners, new Size(5, 5));
}
foreach (Point2f p in corners)
{
Cv2.Circle(gray, (int)p.X, (int)p.Y, 5, new Scalar(0, 0, 255), 2);
}
Cv2.ImShow("Image with Corners", gray);
Mat rvec = new Mat(); // 旋转向量
Mat tvec = new Mat(); // 平移向量
InputArray imagePoints = InputArray.Create(corners);
InputArray objectPoints1 = InputArray.Create(objectPoints);
Cv2.SolvePnP(objectPoints1, imagePoints, Intrinsic, distCoeffs, rvec, tvec, false, SolvePnPFlags.Iterative);
Mat rotMatrix = new Mat(3, 3, MatType.CV_32FC1);
Cv2.Rodrigues(rvec, rotMatrix);
using (var fs1 = new FileStorage("test5.yaml", FileStorage.Mode.Write))
{
fs1.Add("rvecsMat").Add(rotMatrix);
fs1.Add("tvecsMat").Add(tvec);
}
}The second method is to manually obtain the pixel coordinates of the picture. This can use any picture and obtain the coordinates by manually clicking. The world coordinates are still obtained through actual measurement.
private void button6_Click(object sender, EventArgs e)
{
MouseCallback draw = new MouseCallback(draw_circle);
Mat src = Cv2.ImRead(@"你自己的图片", ImreadModes.AnyColor);
Cv2.ImShow("src image", src);
tempMat = new Mat(src.Size(), src.Type());
Cv2.CopyTo(src, tempMat);
System.Runtime.InteropServices.GCHandle handle = System.Runtime.InteropServices.GCHandle.Alloc(src);
IntPtr ptr = System.Runtime.InteropServices.GCHandle.ToIntPtr(handle);
Cv2.SetMouseCallback("src image", draw, ptr);
}
static Mat tempMat;
static Point2f[] a = new Point2f[4];
static int i = 0;
public static void draw_circle(MouseEventTypes @event, int x, int y,
MouseEventFlags flags, IntPtr userData)
{
System.Runtime.InteropServices.GCHandle handle =
System.Runtime.InteropServices.GCHandle.FromIntPtr(userData);
Mat src = (Mat)handle.Target;
if (@event == MouseEventTypes.LButtonDown)
{
a[i].X = x;
a[i].Y = y;
i++;
}
}The above code is to read the photo and create a mouse click event when the key is triggered. I am not familiar with the operation of the mouse, so I just found a routine and modified it to achieve the basic mouse click function. Due to the manual clicking method used here, the number of points is set relatively small. Only four points are used. Adjust according to your actual situation. It is best to have no less than 4 points.
After getting the external parameters of the camera, when I brought the known pixel coordinates into it, I found that I did not get the results I wanted. After several weeks of debugging and calculation, I finally compared them one by one through the C++ version of the solution process. I output the results at each step for comparison, and found that there was a problem with SolvePnP(). Before bringing in the parameters, everything was known to the results of the C++ version, but the external parameters solved by SolvePnP() did not match. I tried to change the input data format, the number of data, and the solution method, but it was always inconsistent with the C++ version, so I used the external parameters obtained by C++ and brought them into my own follow-up program, and finally I was able to get a more accurate world coordinates. The explanation is a problem with the SolvePnP() function.
But unfortunately, I still haven't found a solution. I don't know if you guys have any solutions or where I have the problem.
In order to achieve this step, I finally used C++ to call the C++ version of the SolvePnP() function, encapsulated it into a Dill dynamic link library, and let C# make the call.
4. C++ encapsulates DILL and calls it to C#
Refer to the C++ encapsulation process in other articles. Basically, there will be no big problems if you follow the steps.
#include <iostream>
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgcodecs/legacy/constants_c.h"
#include <opencv2/opencv.hpp>
#include <time.h>
#include"getRT.h" //这里对应你新建的那个头文件
#include <vector>
using namespace std;
using namespace cv;
void toCV()
{
vector<Point2d> imagepoint;
Mat imagepoint1 = Mat(4, 2, CV_64FC1);
Mat objectpoint1 = Mat(4, 3, CV_64FC1);
cv::FileStorage fs("point.yaml", FileStorage::READ);
fs.open("point.yaml", cv::FileStorage::READ);
fs["point"]>>imagepoint1;
fs["worldpoint"] >> objectpoint1;
fs.release();
//imagepoint1.
/* vector<Point3d> objP;
objP.clear();
objP.push_back(Point3d(0, 0, 0));
objP.push_back(Point3d(212.0f, 0, 0));
objP.push_back(Point3d(212.0f, 298.3f, 0));
objP.push_back(Point3d(0, 298.3f, 0));
*/
Mat intrinsic = (Mat_<double>(3, 3) << 1473.819, 0, 615.859 ,
0, 1474.14, 467.697,
0, 0, 1 );
Mat dis = (Mat_<double>(5,1) << 0.051, 0.44, -0.01, -0.009, -2.22);
Mat rvec = Mat(3, 1, CV_64FC1, Scalar::all(0));
Mat tvec = Mat(3, 1, CV_64FC1, Scalar::all(0));
Mat rotM = Mat(3, 3, CV_64FC1, Scalar::all(0));
solvePnP(objectpoint1, imagepoint1, intrinsic, dis, rvec, tvec,false,SOLVEPNP_EPNP );
Rodrigues(rvec, rotM); //将旋转向量变换成旋转矩阵
cv::FileStorage fd("RT.yaml", FileStorage::WRITE);
fd << "R" << rotM;
fd << "t" << tvec;
fd << "point" << imagepoint1;
fd.release();
}
void main()
{
toCV();
}Just reference the function in the C# project, remember to add these two lines of code
[DllImport("getRT.dll")]
private extern static void toCV();Finally, just call toCV where PNP solution is needed.
The following is the final effect. First, the object is detected, the smallest circumscribed circle is found, and the center and radius of the circle are obtained.
Finally, through calculation, I found that the diagonal length of my mobile phone is 166mm, which is converted to 162.56mm based on the screen size of 6.4 inches. Considering that the mobile phone is not a full screen, adding a little to the upper and lower chin, the error should be within 5mm.
It is my first time to come into contact with C#, so I may not be able to express myself clearly in many places, so I will record the whole process by the way, hoping to help those in need.
Another problem is SolvePnP(). I wonder if anyone knows why my solution is always wrong. I hope you can correct me if there are any deficiencies or unclear aspects in the article.