Calibration Tutorial way

Camera and IMU joint calibration

1. imu_utils calibrates the internal reference of the IMU

1. imu_utils calibrates the internal reference of the IMU, which can calibrate the noise density and random walk noise of the IMU

2. The kalibr package calibrates the internal and external parameters of the camera, and the external parameters between the camera and the IMU

1.1 Installation environment

The package used here is imu_utils , which can be used to calibrate the noise density and random walk noise of the IMU

Step1: Install the ceres library

sudo apt-get install liblapack-dev libsuitesparse-dev libcxsparse3.1.2 libgflags-dev 
sudo apt-get install libgoogle-glog-dev libgtest-dev

Download and compile ceres-solver

git clone https://github.com/ceres-solver/ceres-solver.git
cd ceres-solver
mkdir build
cd build
cmake ..
make 
sudo make install

step2: Install code_utils

build workspace

mkdir -p ~/kalibr_workspace/src
cd ~/kalibr_workspace
catkin_make

Notice:Here you need to modify the CMakeLists.txt file of code_utils, CMAKE_CXX_FLAGS "-std=c++11" is changed to CMAKE_CXX_STANDARD 14 and add include_directories (include/code_utils)

sudo apt-get install libdw-dev
cd kalibr_workspace/src
git clone https://github.com/gaowenliang/code_utils.git
cd ..
catkin_make

Step3: Install imu_utils (imu_utils depends on code_utils, first compile code_utils and then compile imu_utils.)

Notice:Here you need to also modify the CMakeLists.txt file of imu_utils, CMAKE_CXX_FLAGS "-std=c++11" is changed to CMAKE_CXX_STANDARD 14

cd kalibr_workspace/src
git clone https://github.com/gaowenliang/imu_utils.git
cd ..
catkin_make

1.2. Record IMU data set

step4: Collect IMU data at rest and record it as a ROS package. My collection time is 2 hours and 20 minutes.

rosbag record /imu/data -O imu_xsens.bag

step5: Calibrate and configure the xsens.launch file as follows: (specify the topic of the IMU)

<launch>
    <node pkg="imu_utils" type="imu_an" name="imu_an" output="screen">
        <param name="imu_topic" type="string" value= "/imu/data"/>		#话题名称
        <param name="imu_name" type="string" value= "xsens"/>
        <param name="data_save_path" type="string" value= "$(find imu_utils)/data/"/>
        <param name="max_time_min" type="int" value= "120"/>   #加载多长时间的数据
        <param name="max_cluster" type="int" value= "100"/>
    </node>
</launch>

Here is max_time_mina maximum time for using bag data, the unit is minutes, the default is 120 minutes,
step6 : Start imu_utils to calibrate the IMU

roslaunch imu_utils xsens.launch
rosbag play -r 200 imu_xsens.bag

This step is the key. After executing the previous step roslaunch imu_utils xsens.launch, the program will enter the state of waiting for the topic.
The following output results are obtained:
insert image description here
Execute as soon as possible rosbag play -r 200 自己的.bag, the program enters bag reading, and calculates the variance of allan. After the accelerated playback of the bag package is completed, the launch window will still display wait for imu data. Wait for a period of time for calculation, and the calculation result will be displayed after the calculation is completed.
insert image description here After done is displayed, catkin_ws/src/imu_utils/dataa series of data files will appear in this folder. Open xsens_imu_param.yamlthis file, and you will see the calculated noise and random walk coefficient values.
![Insert picture description here](https://img-blog.csdnimg.cn/319e45ea36804a1aa6bf7611742df814.png

%YAML:1.0
---
type: IMU
name: xsens
Gyr:
   unit: " rad/s"
   avg-axis:
      gyr_n: 5.0034702036714999e-02
      gyr_w: 2.4583978028785876e-03
   x-axis:
      gyr_n: 2.5852284735645492e-02
      gyr_w: 1.8303245177389645e-03
   y-axis:
      gyr_n: 9.8883151331650182e-02
      gyr_w: 4.8477797168896648e-03
   z-axis:
      gyr_n: 2.5368670042849317e-02
      gyr_w: 6.9708917400713272e-04
Acc:
   unit: " m/s^2"
   avg-axis:
      acc_n: 9.1440795637947525e-01
      acc_w: 2.1004351612340905e-02
   x-axis:
      acc_n: 2.0238764067980461e-01
      acc_w: 1.7432126284010131e-02
   y-axis:
      acc_n: 4.8601261496646220e-01
      acc_w: 2.0739830749315680e-02
   z-axis:
      acc_n: 2.0548236134921587e+00
      acc_w: 2.4841097803696900e-02

2. Kalibr calibration tool

2.1 Installation

sudo apt-get install python-setuptools python-rosinstall ipython libeigen3-dev libboost-all-dev doxygen libopencv-dev
sudo apt-get install libopencv-dev ros-kinetic-vision-opencv ros-kinetic-image-transport-plugins ros-kinetic-cmake-modules python-software-properties software-properties-common libpoco-dev python-matplotlib python-scipy python-git python-pip ipython libtbb-dev libblas-dev liblapack-dev python-catkin-tools libv4l-dev

sudo pip install python-igraph  (sudo apt install python-pip (python2.7))
//若出现问题，用以下命令安装
sudo apt-get install python-igraph

Compile kalibr, here to ensure that the computer is connected to the Internet, the compilation time is a bit long.

cd ~/kalibr_workspace/src
git clone https://github.com/ethz-asl/Kalibr.git
cd ~/kalibr_workspace
catkin build -DCMAKE_BUILD_TYPE=Release -j4
source ~/kalibr_workspace/devel/setup.bash

node: If the following problems occur during compilation, refer to the link [2].

2.2 Internal and external parameters of the calibration camera

It is possible to use the two-dimensional code Aprilgrid or the checkerboard for the calibration board, and it is enough to modify the corresponding yaml when using it. For the calibration board download, see https://github.com/ethz-asl/kalibr/wiki/downloads, which requires a Google account and access to Google Drive. If not (for example, I can't), you can refer to https://github.com/ethz-asl/kalibr/wiki/calibration-targets, use the kalibr_create_target_pdf command to create the specified calibration board yourself and print it.
This article uses the QR code calibration board to create or modify apriltag.yaml

target_type: 'aprilgrid'    #gridtype
tagCols: 6                  		#number of apriltags
tagRows: 6                  			#number of apriltags
tagSize:  0.088           		#size of apriltag, edge to edge [m]
tagSpacing: 0.3          		#ratio of space between tags to tagSize

Among them, tagsize and tagspacing are to be modified according to the actual printed calibration board. The meaning of the parameters can be referred to the figure below, which is also from https://github.com/ethz-asl/kalibr/wiki/calibration-targets.
insert image description here

step1 Run the camera node

rosrun sensor_driver stereo_left_node 
rosurn sensor_driver stereo_right_node

step2 Modify the frequency of the topic to 4
and start the camera node, because it is recommended that the image frequency should not be too high during kalibr processing. Here, use topic_tools/throttle to reduce the frequency of the output image topic, and 4Hz is also the officially recommended frequency.

rosrun topic_tools throttle messages /stereo_left_node/left 4.0 /left 
rosrun topic_tools throttle messages /stereo_right_node/right 4.0 /right

node: Explanation of the command
rosrun topic_tools throttle messages 修改后的话题名频率原始话题名
This command can modify the name and frequency of the topic. This article modifies the frequency of the topic, and the topic name is changed to /left/right

step3 record bag data

rosbag record -0 stereo_calibar.bag /left_img  /right_img

node: Description of binocular data collection
Point the camera at the calibration board, move the camera continuously to obtain images from different angles, collect about 200 pictures (about 50s)

step4 calibrate with Kalibr binocular camera

// 进入到kalibr工具目录下
source devel/setup.bash
rosrun kalibr kalibr_calibrate_cameras --bag xxx/stereo_calibra.bag --topics /left_img /right_img --models  omni-radtan omni-radtan --target xxx/april_6x6_80x80cm_A0.yaml

rosrun kalibr kalibr_calibrate_cameras --bag src/data/2021-09-02-16-24-01.bag --topics /left_img /right_img --models omni-radtan omni-radtan --target src/data/april_6x6_80x80cm_A0.yaml --show-extraction --approx-sync 0.05

node: explain the command

rosrun kalibr kalibr_calibrate_cameras --bag xxx/stereo_calibra.bag --topics /left_img /right_img --models omni-radtan omni-radtan --target xxx/april_6x6_80x80cm_A0.yaml --bag-from-to 5 105

xxx/stereo_calibra.bag bag path recorded by stpe3
/left_img /right_img topic name
omni-radtan After omni-radtan is the camera/distortion model, there are several cameras to write a few, here are two omni-radtan model cameras, so there are two omni-radtan; correspondingly, other supported models can be viewed https://github.com/ethz-asl/kalibr/wiki/supported-models.
xxx/april_6x6_80x80cm_A0.yaml Calibration plate configuration file (Use a standard calibration plate, the parameters of the yaml file downloaded from the official website do not need to be modified, this file can be used directly in the notebook)
--show-extraction is a display interface during the calibration process, you can see the process of image extraction, you can not;
--approx-sync 0.05 Timestamp misalignment problem
–bag-from-to is followed by the start time and end time of the data time period you want to use, unit: second (s), this parameter can eliminate some screens that enter and exit the field of view at the beginning and end of the recording

Records of problems encountered by individuals:

An error was reported during the calibration run, RuntimeError: Optimization failed!

You can refer to the author's answer on github https://github.com/ethz-asl/kalibr/issues/41 to improve timeOffsetPadding (https://github.com/ethz-asl/kalibr/blob/master/aslam_offline_calibration/kalibr/ python/kalibr_calibrate_imu_camera#L171) try again.

After the calibration is completed, the following 3 files will be generated, among which the .yaml file named camchain will be used for subsequent joint calibration, which contains the required internal and external parameters of the camera.

insert image description here

You can view the pdf of the report. It is better to calibrate the reprojection errors within 1 pixel.
insert image description here

3. Joint calibration of camera and IMU

step1 recording data set

Fix the IMU and the camera together to record the ROS bag. When recording, pay attention to follow the official statement - fully stimulate the IMU - rotate around 3 axes and translate in 3 directions. Here is an [official video 3] for reference
recording data The frequency of image data should also be controlled when packetizing.

rosrun topic_tools throttle messages /camleft/video_image 4.0 /caml
rosrun topic_tools throttle messages /camright/video_image 4.0 /camr
rosbag record -O stereo_imu_calibra.bag /stereo_left_node/left  /stereo_right_node/right /imu_node/imu0

Precautions:

采集数据的起始和结束阶段注意别晃动太大，如从桌子上拿起或者放下。如果有这样的动作，在标定阶段应该跳过bag数据集的首尾的数据.
采集数据的时候应该给imu各个轴足够的激励，如先依次绕各个轴运动，运动完后来个在空中画8字之类的操作，当然也要注意别运动太剧烈，图像都模糊了。

The yaml format of the imu internal reference in the joint calibration is as follows, and the yaml file of the calibration result of imu_utils cannot be used directly, otherwise an error will be reported RuntimeError: [ImuConfig Reader]: Could not read configuration from …

#Accelerometers
accelerometer_noise_density: 4.8641695361661035e-03   #Noise density (continuous-time)
accelerometer_random_walk:   2.3944306307068487e-04   #Bias random walk
 
#Gyroscopes
gyroscope_noise_density:     5.4583665041817392e-04   #Noise density (continuous-time)
gyroscope_random_walk:       6.1968037914410386e-06   #Bias random walk
 
rostopic:                    /imu      #the IMU ROS topic
update_rate:                 100.0      #Hz (for discretization of the values above)

step2 Calibration

source devel/setup.bash 
 
rosrun kalibr kalibr_calibrate_imu_camera --target src/data/april_6x6_80x80cm_A0.yaml --cam src/data/camera.yaml --imu src/data/A1_ium.yaml --bag src/data/stereo_imu_calibra.bag
rosrun kalibr kalibr_calibrate_imu_camera --target april_6x6_55x55mm.yaml --bag ~/bagfiles/zed_xsens.bag --bag-from-to 5 50 --cam src/data/camera.yaml  --imu src/data/A1_ium.yaml --imu-models scale-misalignment --timeoffset-padding 0.1

node: explain the command

1. --target src/data/april_6x6_80x80cm_A0.yaml　: The parameter file of the calibration board;

2. --bag images_imu.bagSpecify the data package

3. –bag-from-to 5 50Set the start time and end time of the bag to avoid the data in the time period of picking up and putting down the IMU

2. --cam src/data/camera.yaml　: camera parameter file, refer to [ Document 1: camera.yaml ] for the format. Use the calibrated camera parameters to replace the corresponding content;
3. --imu src/data/ium.yaml 　: imu configuration file, refer to [ Document 2: imu.yaml ] for the format, replace the corresponding parameters with the calibrated imu parameters;
4. –imu-models scale-misalignmentIMU parameter model

The content of the camera.yaml file is as follows:

cam0:
  cam_overlaps: [1]
  camera_model: pinhole
  distortion_coeffs: [-0.1734857772863602, 0.026545178121976657, 0.0004291887376674085,
    -3.4873170616746686e-05]
  distortion_model: radtan
  intrinsics: [693.131838769146, 692.5498277671763, 616.3486206381017, 379.6677572220899]
  resolution: [1280, 720]
  rostopic: /stereo/left/image_raw
cam1:
  T_cn_cnm1:
  - [0.9999658061828064, 0.0005632950824424241, 0.0082504038578218, -0.11947602055948268]
  - [-0.0006621128372211408, 0.9999280240823567, 0.011979493367486592, 0.0004870068672051519]
  - [-0.008243062037729159, -0.011984546441186855, 0.9998942056912012, -0.0028910358303400464]
  - [0.0, 0.0, 0.0, 1.0]
  cam_overlaps: [0]
  camera_model: pinhole
  distortion_coeffs: [-0.17456713089475154, 0.027410444232267236, 0.0006360696559962682,
    -0.0002450168896166665]
  distortion_model: radtan
  intrinsics: [694.2107729740508, 693.480347118504, 617.3114354961933, 380.800130116761]
  resolution: [1280, 720]
  rostopic: /stereo/right/image_raw

The content of the imu.yaml file is as follows:

#Accelerometers
accelerometer_noise_density: 5.43036e-03   #Noise density (continuous-time)
accelerometer_random_walk:   1.44598e-04   #Bias random walk

#Gyroscopes
gyroscope_noise_density:     4.9700e-03   #Noise density (continuous-time)
gyroscope_random_walk:       6.8522e-05   #Bias random walk

rostopic:                    /imu/data      #the IMU ROS topic
update_rate:                 100.0      #Hz (for discretization of the values above)

After the calibration is completed, a report will also be generated and the calibration information will be printed in the terminal;

insert image description here

The reprojection error of the calibration result should be within a few tenths of a pixel.
The calibrated displacement is consistent with the actual measured or estimated IMU and camera center displacement.

Camera and IMU Calibration Tutorial