It is recommended to understand the camera model knowledge first:
https://zhuanlan.zhihu.com/p/71234078
https://blog.csdn.net/lqql2012/article/details/106127447
https://zhuanlan.zhihu.com/p/ 317673040
Kalibr is a toolbox (a toolbox).
The camera-imu calibration tool (one of the tools) estimates the spatial and temporal parameters of a camera system with respect to an intrinsically calibrated IMU. The calibration parameters (the parameters to be calibrated are estimated) are estimated
in a full batch optimization using splines to model the pose of the system.
kalibr——github：
https://github.com/ethz-asl/kalibr
kalibr——wiki：
https://github.com/ethz-asl/kalibr/wiki
So, Kalibr is exactly what we need!
There are two ways to obtain the Kalibr toolbox: 1. Compile from source code; 2. CDE package.
1. Source code compilation: it depends on ROS indigo and a catkin workspace. This method of calibration has two advantages: it is faster than the CDE package and provides all (tools) tools.
2. CDE package: All dependencies are packaged in the package without any other dependencies. But the Camera focus and Calibration validator tools cannot be used (because they depend on ROS).
Install kalibr from source
Installation environment: ubuntu18 + ros melodic
camera equipment: mynteye-s2110
installation method:
sudo apt-get install python-setuptools python-rosinstall ipython libeigen3-dev libboost-all-dev doxygen libopencv-dev ros-melodic-vision-opencv ros-melodic-image-transport-plugins ros-melodic-cmake-modules software-properties-common libpoco-dev python-matplotlib python-scipy python-git python-pip ipython libtbb-dev libblas-dev liblapack-dev python-catkin-tools libv4l-dev
Among them, ros-melodic-vision-opencv, ros-melodic-image-transport-plugins, ros-melodic-cmake-modules need to change melodic to the corresponding version required by the system
3. Install igraph
sudo pip install python-igraph (sudo apt install python-pip (python2.7))
4. Create a workspace:
mkdir -p ~/kalibr_workspace/src
cd ~/calibre_workspace
source /opt/ros/melodic/setup.bash
catkin hot
catkin config --extend /opt/ros/melodic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release
5. Download the kalibr source code:
cd ~/kalibr_workspace/src
git clone https://github.com/ethz-asl/kalibr.git
6. Compile
cd ~/kalibr_workspace
catkin build -DCMAKE_BUILD_TYPE=Release -j4
7. Set up the environment
source ~/kalibr_workspace/devel/setup.bash
8. Test whether the installation is successful
kalibr_calibrate_imu_camera --target april_6x6.yaml --cam camchain.yaml --imu imu_adis16448.yaml --bag dynamic.bag --bag-from-to 5 45
There is a problem when using kalibr_bagcreater, no module "ImageFile", the solution is to directly comment out the import ImageFile in kalibr_bagcreater.
Note:
upgrade pip, otherwise it is easy to find the installation package
sudo python -m pip install --upgrade --force-reinstall pip
Calibration binocular
Supported camera models:
https://github.com/ethz-asl/kalibr/wiki/supported-models
yaml file format:
https://github.com/ethz-asl/kalibr/wiki/calibration-targets
calibration image download :
https://github.com/ethz-asl/kalibr/wiki
official calibration document:
https://github.com/ethz-asl/kalibr/wiki/multiple-camera-calibration
Mimi's camera parameters:
Intrinsics of the camera: k represents the equidistant distortion coefficient, mu, mv correspond to the focal length, v, u correspond to the principal point coordinates. Camera extrinsics (from right eye to left eye): rotation indicates the rotation matrix, and translation indicates the translation matrix. D, K, R, and P are distortion parameters, internal reference matrix, correction matrix, and projection matrix, respectively. Reference link: ros CameraInfo
(1) Recording data
Data set requirements: fix the Mimi camera, and move the aprilgrid calibration board in the camera field of view.
Pay attention to turn off the IR light (I am afraid that the light may affect the matching), enter the MYNT-EYE-S-SDK folder, enter wrappers/ros/src/mynt_eye_ros_wrapper/config/device/standard.yaml, it should be ok.
Turn on the camera
source wrappers/ros/devel/setup.bash
roslaunch mynt_eye_ros_wrapper display.launch
Adjust the frequency
Kalibr recommends using 4Hz image frame rate, here use topic_tools/throttle to reduce the receiving frequency
How to use: throttle messages <msgs_per_sec> [outtopic]
rosrun topic_tools throttle messages /mynteye/left/image_raw 4.0 /mynteye/left/image_raw
rosrun topic_tools throttle messages /mynteye/right/image_raw 4.0 /mynteye/right/image_raw
Some reference blogs will be renamed directly to /left and /right. At this time, the yaml generated after calibration will also record the name of the topic. When calibrating the relationship between the binocular and IMU, pay attention to the unification of topics.
rosbag record -O stereo_calib.bag /mynteye/left/image_raw /mynteye/right/image_raw
Assume that it is saved under the ~/ path.
(2) Calibration
After collecting the data set, you can use the installed Kalibr to calibrate the binocular camera. The command is as follows:
3) Start calibration
Execute with the following command (take the commands in the follow-up routine as an example):
kalibr_calibrate_cameras --bag /home/laoli/stereo_calib_1.bag --topics /left /right --models pinhole-equi pinhole-equi --target /home/laoli/project_work/kalibr/kalibr_18_1/stereo_calib_1.yaml --show-extraction --approx-sync 0.08
4) Output
Calibration will generate the following output files:
– report-imucam-%BAGNAME%.pdf
– results-imucam-%BAGNAME%.txt
– camchain-imucam-%BAGNAME%.yaml
(3) Calibration results
The evaluation standard of the calibration results
is due to possible artificial installation problems, and the real transfer matrix from the IMU to the camera may be uncertain. Since there is no accurate judgment criterion, the current calibration results are mainly based on the following two aspect:
- The conversion between the real IMU coordinate system and the camera coordinate system, that is, the physical distance between the camera and the IMU
- On the stability of calibration results
Calibration verification was not successful
Verification method: Calibration validator can be used for calibration verification. The principle is to quantify and analyze the reprojection error. Use the camera calibration result of the previous step and the yaml file of the calibration board, the specific command is as follows: (refer to the link)
kalibr_camera_validator --cam camchain.yaml --target target.yaml
Calibrate the relationship between stereo-imu
Data collection
requirements: fix the aprilgrid calibration board in the camera field of view, move the camera, and rotate around the three axes (yaw-pitch-roll), and there are movements of up and down, left and right, push and push, and draw a figure of 8. The movement should not be too fast .
The best is image 20hz, IMU 200hz, others are also fine.
Here I enter the MYNT-EYE-S-SDK folder, enter wrappers/ros/src/mynt_eye_ros_wrapper/config/device/standard.yaml, and modify the frequency: It should be like this.
rosbag record -O stereo_imu_calib.bag /mynteye/left/image_raw /mynteye/right/image_raw /mynteye/imu/data_raw
Calibration
Calibration will use imu.yaml, mainly including frequency, noise, and random walk values, which should be given An approximate initial value.
I directly used the imu_adis16448.yaml file provided in kalibr, and now I think there should be a problem. I found that there are settings about imu parameters in MYNT-EYE-VINS-FUSION-Samples/config/mynteye/mynteye_stereo_imu_config.yaml of MYNT-EYE-VINS-FUSION-Samples.
Using this parameter should work.
Kalibr_calibiBi_imu_Camera-Target /Home/jmliu/mynt_calib/april_6x6_80x80cm.yaml-Cam/Home/jmliu/ Homejmliusteo_Calib.yamam l -IMU/HOME/jmliu/imu.yaml --bag /Home/jmliu/steeo_imu_calib.bag-- -bag-from-to 2 45
points to note:
Pay attention to modifying the topic of imu in imu.yaml. (This is imu_adis16448.yaml, just modify the value according to the format)
--bag-from-to parameter, if the data set has the action of raising the camera at the beginning and the end, please add this parameter to remove this part of the data. The action of lifting and lowering will have a certain impact on the calibration. There is no need to add.
Will use the above yaml to calibrate the relationship between the binoculars
and pay attention to modifying the path. You can add the following command during calibration to remove the jitter at the beginning and near the end
–bag-from-to 5 45
[Note]: The commands of the CDE version and the source code compilation version are slightly different.
Among them, the meaning of the parameters:
–bag contains the ROS package of camera image data and IMU data;
–cam saves the internal and extrinsic calibration parameters of the camera system (intrinsic and extrinsic calibration parameters). You can use the multiple-camera-calibration tool calibration results here, so the file name is so long! (See the second section for details);
–imu saves the file of the IMU statistics and the topic name (the IMU statistics and the IMU's topic);
–target saves the file of the target pattern size information.
result
The last –bag_from_to is to select the bag data of 5-45s, which removes the influence of the jitter generated by the pick-up prevention device. We recommend performing the best unbiased estimation of the calibration results after multiple sets of calibration. It is not recommended to take the average value of the R part directly. You can convert the R part of multiple sets of data into a quaternion. Process it afterwards.
else：
In the process of IMU and Cam calibration data acquisition, it is usually recommended that the camera use 20hz, and the imu use 200hz frequency. Avoid vibrations, avoid blurry photos, etc., especially at the beginning and end of the calibration.
After the image and IMU data are collected, the ROS bag format can be generated by the following command.
kalibr_bagcreater --folder dataset-dir --output-bag awsome.bag
You can use rosbag info awsome.bag to check whether the generated bag is qualified.
In addition to the bag package, the three parameters required by the command are –target, –cam, and –imu, which respectively refer to the standard of the calibration board used in the process of collecting data, the internal reference of the camera, and the noise data of the imu. The calibration board used here needs to be printed out in a ratio of 1:1. For example, the pdf downloaded from kalibr is a calibration board of 0.8m*0.8m. Of course, you can also make a calibration board yourself, such as making an A4-sized calibration board, and then choose not to zoom when printing, and check the production method through kalibr_create_target_pdf –h. The internal reference calibration of the camera is a relatively mature technology, which will not be introduced here. The noise parameters of the IMU can generally be obtained through the datasheet of the IMU device.
If there is no problem, by running kalibr_calibrate_imu_camera, we can get pdf, txt, and yaml3 result files.
We mainly focus on the results-imucam-%BAGNAME%.txt file, and check whether our calibration is valid based on this result.
Among them, the reprojection error should not exceed 5 pixels. The displacement in the T_ic/T_ci matrix should be based on the actual position of the device IMU and Camera. Generally, it should be a scale value of centimeters or millimeters.