Table of contents
3.1 Introduction to lidar_camera_calib
3.5 Using your own sensor settings
3.5.1 Collect camera pictures and radar bag data
3.5.2 Using multi-scene calibration
3.5.3 Acquisition of camera internal parameters
3.5.4 Running the Calibration Program
Source address: https://github.com/hku-mars/livox_camera_calib
Note: lidar_camera_calib and livox_camera_lidar_calibration are two different calibration programs. The two calibration codes are based on livox’s lidar. The code overlap is relatively high, and they should be the results of the same team. The self-calibration process of lidar_camera_calib calibration is relatively simple, and the accuracy will follow lower. This article introduces lidar_camera_calib, and also draws on livox_camera_lidar_calibration (projecting point cloud to photo projectCloud.cpp and point cloud coloring color_lidar_display.cpp code)
3.1 Introduction to lidar_camera_calib
lidar_camera_calib is a tool for accurate extrinsic calibration between high-resolution LiDARs (such as Livox) and cameras in targetless environments. Our algorithm can operate in both indoor and outdoor scenes, and only requires edge information in the scene. If the scene is suitable, we can achieve pixel-level accuracy similar to or even surpass object-based methods.
We color the point cloud with the calibrated extrinsic parameters and compare with the real image. A and C are partially zoomed-in views of the point cloud. B and D are the parts of the camera image corresponding to the point clouds in A and C.
lidar_camera_calib supports multi-scene calibration (more accurate and stable) Related papers Related papers can be found on arxiv: External self-calibration of high-resolution LiDAR and cameras in pixel-level targetless environments.
3.2 Environment preparation
(1) Ubuntu and ROS
Requires Ubuntu 64-bit 16.04 or 18.04. ROS Kinetic or Melodic. Installation of ROS and its additional ROS packages
sudo apt-get install ros-XXX-cv-bridge ros-xxx-pcl-conversions(2) Own
// 安装
cd eigen-git-mirror
mkdir build
cd build
cmake ..
sudo make install
// 安装后 头文件安装在/usr/local/include/eigen3/
// 移动头文件
sudo cp -r /usr/local/include/eigen3/Eigen /usr/local/includeRemarks: In many programs, #include <Eigen/Dense> is often used instead of #include <eigen3/Eigen/Dense> when including, so do the following processing
(3) Ceres Solver
(4) PCL
3.3 compile
Clone the codebase and run catkin_make
cd ~/catkin_ws/src
git clone https://github.com/hku-mars/livox_camera_calib.git
cd ../
catkin_make
source ~/catkin_ws/devel/setup.bashCompilation error 1: The following error will appear during compilation. The reason for the error is that the ceres version is too high. After changing ceres2.1.0 to ceres2.2.0, it will run normally
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_multi_calib.cpp:312:14: error: ‘LocalParameterization’ is not a member of ‘ceres’
312 | ceres::LocalParameterization *q_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_multi_calib.cpp:312:37: error: ‘q_parameterization’ was not declared in this scope
312 | ceres::LocalParameterization *q_parameterization =
| ^~~~~~~~~~~~~~~~~~
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_multi_calib.cpp:313:15: error: expected type-specifier
313 | new ceres::EigenQuaternionParameterization();
| ^~~~~
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_calib.cpp: In function ‘int main(int, char**)’:
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_calib.cpp:311:14: error: ‘LocalParameterization’ is not a member of ‘ceres’
311 | ceres::LocalParameterization *q_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_calib.cpp:311:37: error: ‘q_parameterization’ was not declared in this scope
311 | ceres::LocalParameterization *q_parameterization =
| ^~~~~~~~~~~~~~~~~~
/home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/src/lidar_camera_calib.cpp:312:15: error: expected type-specifier
312 | new ceres::EigenQuaternionParameterization();Compile error 2: opencv version conflict, opencv4.5.4 that comes with the system conflicts with opencv4.2.0 under ros.
/usr/bin/ld: warning: libopencv_features2d.so.4.2, needed by /usr/lib/aarch64-linux-gnu/libopencv_calib3d.so.4.2.0, may conflict with libopencv_features2d.so.4.5
/usr/bin/ld: warning: libopencv_imgproc.so.4.5, needed by /usr/lib/aarch64-linux-gnu/libopencv_imgcodecs.so.4.5.4, may conflict with libopencv_imgproc.so.4.2
[ 91%] Built target lidar_camera_calib
/usr/bin/ld: warning: libopencv_features2d.so.4.2, needed by /usr/lib/aarch64-linux-gnu/libopencv_calib3d.so.4.2.0, may conflict with libopencv_features2d.so.4.5
/usr/bin/ld: warning: libopencv_imgproc.so.4.5, needed by /usr/lib/aarch64-linux-gnu/libopencv_imgcodecs.so.4.5.4, may conflict with libopencv_imgproc.so.4.24Modify the cv_bridge configuration:
The opencv path installed by ros by default is in three directories: /usr/include, /usr/lib, and /usr/share. After compiling and installing from the source code of the opencv official website, opencv will be installed to the corresponding three subdirectories under usr/local by default.
#if(NOT "include;/usr/include/opencv4 " STREQUAL " ")
# set(cv_bridge_INCLUDE_DIRS "")
# set(_include_dirs "include;/usr/include/opencv4")
if(NOT "include;/usr/local/include;/usr/local/include/opencv4" STREQUAL " ")
set(cv_bridge_INCLUDE_DIRS "")
set(_include_dirs "include;/usr/local/include;/usr/local/include/opencv4;/usr/include")
set(libraries "cv_bridge;/usr/lib/aarch64-linux-gnu/libopencv_calib3d.so.4.2.0;/usr/lib/aarch64-linux-gnu/libopencv_dnn.so.4.2.0;........)3.4 Running the dataset
The officially provided datasets can be downloaded from OneDrive and BaiduNetDisk (Baidu NetDisk).
Baidu Netdisk: https://pan.baidu.com/s/1oz3unqsmDnFvBExY5fiBJQ?pwd=i964#list/path=%2F
(1) Single scene calibration
Download our pcd and image files to a local path, and change the file path in the calib.yaml file to your data path. Then run directly:
source ./devel/setup.bash
roslaunch livox_camera_calib calib.launchwill get the following result.
make[2]: *** No rule to make target '/usr/lib/aarch64-linux-gnu/libopencv_aruco.so.4.5.4', needed by '/home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/bag_to_pcd'. Stop.
make[1]: *** [CMakeFiles/Makefile2:2054: livox_camera_calib/CMakeFiles/bag_to_pcd.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: *** No rule to make target '/usr/lib/aarch64-linux-gnu/libopencv_aruco.so.4.5.4', needed by '/home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/lidar_camera_calib'. Stop.
make[1]: *** [CMakeFiles/Makefile2:488: livox_camera_calib/CMakeFiles/lidar_camera_calib.dir/all] Error 2
make[2]: *** No rule to make target '/usr/lib/aarch64-linux-gnu/libopencv_aruco.so.4.5.4', needed by '/home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/lidar_camera_multi_calib'. Stop.Running error 1: Can not load image.
[ERROR] [1689831836.845248880]: Can not load image from /home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib-master/data/calib_dataset/single_scene_calibratio/0.png
[lidar_camera_calib-1] process has died [pid 18393, exit code 255, cmd /home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/lidar_camera_calib __name:=lidar_camera_calib __log:=/home/zjlab/.ros/log/d53e9cfa-26ac-11ee-b53d-48b02d3db6e3/lidar_camera_calib-1.log].
log file: /home/zjlab/.ros/log/d53e9cfa-26ac-11ee-b53d-48b02d3db6e3/lidar_camera_calib-1*.logCheck whether the file path in the calib.yaml file is changed correctly, and it works normally after the change.
Running error 2: process has died. The opencv4.5.4 that comes with the system conflicts with the opencv4.2.0 under ros
[lidar_camera_calib-1] process has died [pid 25608, exit code -11, cmd /home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/lidar_camera_calib __name:=lidar_camera_calib __log:=/home/zjlab/.ros/log/d53e9cfa-26ac-11ee-b53d-48b02d3db6e3/lidar_camera_calib-1.log].
log file: /home/zjlab/.ros/log/d53e9cfa-26ac-11ee-b53d-48b02d3db6e3/lidar_camera_calib-1*.log
init.png
The calibration results generated under the file directory specified by the specified yaml are as follows:
-0.00261333,-0.999901,-0.0138569,0.014096
-0.00334576,0.0138656,-0.999898,0.0574687
0.999991,-0.0025667,-0.00338166,-0.0518364
0,0,0,1(2) Multi-scene calibration
Download our pcd and image files to a local path, and change the file path in the multi_calib.yaml file to your data path. Then run directly:
roslaunch livox_camera_calib multi_calib.launchProjected image obtained using initial extrinsic parameters. (Sensor Kit: Livox Horizon + MVS Camera)
Example of multi-scene calibration.
residual visualization results
rviz visualization results
Projection image obtained with initial extrinsics. (Sensor Kit: Livox Horizon + MVS Camera)
An example of multi-scene calibration. Projection image obtained with initial extrinsics .
Coarse calibration is used to handle bad extrinsics.
Projection image obtained with extrinsic parameters obtained after a rough calibration.
After final optimization, we end up with an exact extrinsic parameter.
Projection image obtained with extrinsics obtained after fine calibration.
The calibration results generated under the file directory specified by the specified yaml are as follows:
0.00943777,-0.999902,-0.0103278,-0.0507875
-0.0434641,0.00990826,-0.999006,0.0851625
0.99901,0.00987727,-0.0433664,-0.0231513
0,0,0,13.5 Calibration using own sensor data
3.5.1 Collect camera pictures and radar bag data
(1) Use the key 'T' to capture camera images, and implement the following script, which comes with the ZED camera, and modify it in the /usr/local/zed/samples/tutorials/tutorial2_image_capture directory.
#include </usr/local/zed/include/sl/Camera.hpp>
#include <iostream>
#include <opencv2/opencv.hpp>
using namespace std;
using namespace sl;
int main(int argc, char **argv) {
// Create a ZED camera object
Camera zed;
// Set configuration parameters
InitParameters init_parameters;
init_parameters.camera_resolution = RESOLUTION::HD1080; // Use HD1080 video mode
init_parameters.camera_fps = 30; // Set fps at 30
// Open the camera
auto returned_state = zed.open(init_parameters);
if (returned_state != ERROR_CODE::SUCCESS) {
cout << "Error " << returned_state << ", exit program." << endl;
return EXIT_FAILURE;
}
// Capture 50 frames and stop
// int i = 0;
int i = 0;
sl::Mat image;
cout << "Please 'T' or 't' capture picture, 'E' or 'e' Exit!" << endl;
while (true){
// Grab an image
returned_state = zed.grab();
// A new image is available if grab() returns ERROR_CODE::SUCCESS
if (returned_state == ERROR_CODE::SUCCESS) {
char key;
cin >> key;
if(key == 'E' || key == 'e'){
break;
}
if(key == 'T' || key == 't'){
// Get the left image
cout << "Capture picture" << to_string(i) << endl;
zed.retrieveImage(image, VIEW::LEFT);
string savePath = "../data/image/" + to_string(i) + ".bmp" ;
image.write(savePath.c_str());
// Display the image resolution and its acquisition timestamp
cout<<"Image resolution: "<< image.getWidth()<<"x"<<image.getHeight() <<" || Image timestamp: "<<image.timestamp.data_ns<< endl << endl;
i++;
}
}
}
// while(i < 10){
// returned_state
// }
// Close the camera
zed.close();
return EXIT_SUCCESS;(2) To collect radar bag data, start the radar through the roslaunch livox_ros_driver livox_lidar_msg.launch command. The type of the bag is livox_ros_driver2/CustomMsg, otherwise the format of the bag is sensor_msgs/PointCloud2, which is inconvenient for subsequent processing.
roslaunch livox_ros_driver livox_lidar_msg.launch # 启动雷达
# 在根目录下逐级创建用于保存 bag 文件的文件夹
mkdir -p src/livox_camera_lidar_calibration/data/lidar
rosbag record -O src/livox_camera_lidar_calibration/data/lidar/0.bag /livox/lidar # 录制点云，输出到指定文件夹下，便于后续操作，文件名从序号 0 开始标3.5.2 Using multi-scene calibration
Change the parameters in multi_calib.yaml to name the image files and pcd files from 0 to (total number of data - 1).
[ INFO] [1690958506.262467995]: Loading the rosbag /home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag/0.bag
[ERROR] [1690958506.437747485]: LOADING BAG FAILED: Bag unindexed
[bag_to_pcd-2] process has died [pid 6316, exit code 255, cmd /home/zjlab/perception/Calib/calib_ws/devel/lib/livox_camera_calib/bag_to_pcd __name:=bag_to_pcd __log:=/home/zjlab/.ros/log/a4eaa69a-30ff-11ee-99c8-48b02d3db6e3/bag_to_pcd-2.log].
log file: /home/zjlab/.ros/log/a4eaa69a-30ff-11ee-99c8-48b02d3db6e3/bag_to_pcd-2*.logError: The specific information of the error is:
rosbag.bag.ROSBagUnindexedException: Unindexed bagproblem solved:
When encountering this problem, we need to take a look at the bag and use the command:
rosbag info 0.bagAt this time, an error will be reported:
ERROR bag unindexed: 0.bag. Run rosbag reindex.Execute the recorded bag in sequence:
rosbag reindex 0.bagWait for the execution to finish, and then we can input. If the output can be normal, it means that the bag can be read normally.
zjlab@zjlab-zjrobot:~/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag$ rosbag info 0.bag
ERROR bag unindexed: 0.bag. Run rosbag reindex.
zjlab@zjlab-zjrobot:~/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag$ Run rosbag reindex
bash: Run: command not found
zjlab@zjlab-zjrobot:~/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag$ rosbag reindex 0.bag
0.bag 100% 44.1 MB 00:00
zjlab@zjlab-zjrobot:~/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag$ rosbag info 0.bag
path: 0.bag
version: 2.0
duration: 10.5s
start: Jul 28 2023 16:37:17.00 (1690533438.00)
end: Jul 28 2023 16:37:28.50 (1690533448.50)
size: 43.7 MB
messages: 106
compression: none [53/53 chunks]
types: sensor_msgs/PointCloud2 [1158d486dd51d683ce2f1be655c3c181]
topics: /livox/lidar 106 msgs : sensor_msgs/PointCloud2
3.5.3 Acquisition of camera internal parameters
The format of the internal reference matrix is shown in the figure below. Generally, there are corresponding values in the four positions (0,0);(0,2);(1,1);(1,2).
Obtain the internal parameters of the camera in the /usr/local/zed/settings directory file
[LEFT_CAM_2K]
fx=1055.98
fy=1054.75
cx=1109.22
cy=618.272
k1=-0.0436089
k2=0.0133655
p1=-0.000636837
p2=0.000189454
k3=-0.00610127
Error reporting: The internal and external parameters of the camera must meet the format requirements, otherwise the following error will be reported.
process[projectCloud-1]: started with pid [25817]
Get the parameters from the launch file
[ INFO] [1691026996.688365859]: Start to load the rosbag /home/zjlab/perception/Calib/calib_ws/src/livox_camera_calib/mydata/bag/0.bag
Can not convert a string to double
[projectCloud-1] process has finished cleanly
Solution: Modify the internal and external reference files, arrange them according to the requirements, and there must be spaces between the numbers
报错：Failed to load module "canberra-gtk-module"
Gtk-Message: 11:34:35.337: Failed to load module "canberra-gtk-module"
[projectCloud-1] process has finished cleanly
log file: /home/zjlab/.ros/log/83ec06a2-311b-11ee-a1d3-024273a4022a/projectCloud-1*.log
all processes on machine have died, roslaunch will exitSolution: reinstall it.
sudo apt-get install libcanberra-gtk-module3.5.4 Running the Calibration Program
After running, the external reference results will be saved in the specified directory
roslaunch livox_camera_calib multi_calib.launch3.5.5 Verification Results
Project point cloud to photo and point cloud coloring can refer to livox_camera_lidar_calibration, just modify the path under the launch file to verify.
(1) Project point cloud to photo
roslaunch livox_camera_calib projectCloud.launch(2) Point cloud coloring:
roslaunch livox_camera_calib colorLidar.launch