ORB-SLAM2 to

ORB-SLAM is a visual SLAM system written by Raul Mur-Artal of Zaragoza University in Spain. His paper "ORB-SLAM: a versatile and accurate monocular SLAM system" was published in IEEE Trans. on Robotics in 2015. The open source code includes ORB-SLAM in the early stage and ORB-SLAM2 in the later stage. The first version is mainly used for monocular SLAM, while the second version supports monocular, binocular and RGBD interfaces.

Features

ORB-SLAM is a complete SLAM system, including visual odometry, tracking, and loop detection. It is a monocular SLAM system based entirely on sparse feature points, and its core is to use ORB (Orinted FAST and BRIEF) as the core feature in the entire visual SLAM. It is embodied in several aspects:

• The feature points extracted and tracked use ORB. The extraction process of ORB features is very fast, suitable for real-time systems.
• Loopback detection uses a bag-of-words model, and its dictionary is a large ORB dictionary.
• The interface is rich, supports monocular, binocular, and RGBD sensor input, and ROS is optional when compiling, making its application very portable. The price is to support various interfaces, and the code logic is slightly more complicated.
• Real-time calculation is performed on a PC at a speed of 30ms/frame, but it performs poorly on an embedded platform.

principle

The overall process of ORB-SLAM is shown in the figure below.

It is mainly composed of three threads: tracking, Local Mapping (also known as small map), Loop Closing (also known as large map).

track

The tracking thread is equivalent to a visual odometer, and the process is as follows:

• First, extract the ORB feature from the original image and calculate the descriptor.
• According to the feature description, feature matching between images.
• Estimate camera movement based on matching feature points.
• According to the key frame criterion, it is judged whether the current frame is a key frame.

Compared with most visual SLAMs that use the size of inter-frame motion to take key frames, ORB_SLAM's key frame discrimination criteria are more complicated.

1 The purpose of the experiment

(1) Configure the ORB-SLAM environment and run it in the real environment
(2) Understand the process and framework of project implementation
(3) Familiar with the development under Ubuntu and ROS systems

2 Experimental environment

2.1 Hardware

(1) HP laptop with the following configuration:
Processor: Intel® Core™ i5-7300HQ CPU @ 2.50GHz × 4
Memory: 8 GB
(2) The sensor is a Logitech C170 webcam, the parameters are as follows:
pixels: 1.3 million ; Capture format: 1024*768; Interface: USB 2.0

2.2 Software

(1) Operating system: Ubuntu 14.04
(2) Software platform: ROS indigo

3 Experimental steps

3.1 First install the necessary dependencies

(1) Update the apt library and update the software list

$ sudo apt-get update

(2) Install git, which is used to clone the project from Github to the local

$ sudo apt-get install git

(3) Install cmake for program compilation

$ sudo apt-get install cmake

(4) Install Pangolin as visualization and user interface
a. Install dependencies

$ sudo apt-get install libglew-dev libpython2.7-dev

b. Download the project from Github to the local

$ git clone https://github.com/stevenlovegrove/Pangolin.git

c. Compile and install

$ cd Pangolin
$ mkdir build
$ cd build
$ cmake ..
$ make –j
$ sudo make install

(5) Install OpenCV to process images and features. The
minimum required version is 2.4.3, and it has been tested on OpenCV 2.4.11 and OpenCV 3.2.
Note: The version issue here is very important. Different version numbers may be incompatible, and various errors will occur. You can only recompile. For inexperienced developers, they will spend a lot of time looking for errors and solving problems.
In this experiment, I used OpenCV 3.2 for the first time, but an error occurred at the end, and it succeeded after switching to OpenCV 2.4.11.
a. Install dependencies
[compiler]

sudo apt-get install build-essential

[required]

sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev

[optional]

sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev

b. Download the source version of OpenCV 2.4.11 on the OpenCV official website ( http://opencv.org ), and then unzip it to the local
c. Compile and install

$ cd ~/opencv
$ mkdir build
$ cd build
$　cmake -D CMAKE_BUILD_TYPE=Release –D CMAKE_INSTALL_PREFIX=/usr/local ..
$　make –j8
$　sudo make install

The compilation process will take a long time. Computers with lower CPU performance are recommended not to be used make -j, which are easy to freeze; higher CPU performance can be used make -jx, x stands for thread, which can speed up compilation.
(6) Install Eigen3, which is an open source linear library that can perform matrix operations

$ sudo apt-get install libeigen3-dev

(7) Install DBoW2 and g2o
DBoW2 is an improved version of the DBow library. The DBow library is an open source C++ library for indexing images and converting them into word representations.
g2o is an open source C++ framework for optimizing graph-based nonlinear error functions.
These two libraries are in the third-party folder of the ORB-SLAM2 project. They will not be compiled separately here, but will be compiled in a unified manner.
(8) Install ORB-SLAM2
a. Clone the warehouse

$ git clone https://github.com/raulmur/ORB_SLAM2.git ORB_SLAM2

b. Compile ORB-SLAM2, DBoW2 and g2o in the third-party library, and decompress the ORB dictionary

$ cd ORB_SLAM2
$ chmod +x build.sh
$ ./build.sh

3.2 Monocular example

There are three datasets: TUM, KITTI, and EuRoC. This experiment uses the TUM dataset. Download the sequence from http://vision.in.tum.de/data/datasets/rgbd-dataset/download and decompress it.

$ ./Examples/Monocular/mono_tum Vocabulary/ORBvoc.txt Examples/Monocular/TUMX.yaml PATH_TO_SEQUENCE_FOLDER

Among them PATH_TO_SEQUENCE_FOLDERis the storage path tumx.yamlof the data set and will correspond to the downloaded data set. For example, TUM1.yaml, TUM2.yaml and TUM3.yaml correspond to freiburg1, freiburg2 and freiburg3 respectively.
The running effect diagram is shown below. The small blue squares in the left window of the figure are the extracted image ORB features, and the right window shows the sparse map of the environment and the camera's movement trajectory.

The open data set can provide a large amount of data to make up for the lack of data. Even if there is no RGB-D camera, you can also use the data of the open data set to experiment. Of course, if you want to do well, you must first sharpen your tools, whether you are doing scientific research or engineering. Hardware equipment is necessary; in addition, public data sets can also be used as evaluation criteria for algorithm performance comparison.

3.3 ROS example

In order to run ORB-SLAM2 online and in real time, ROS is needed. Regarding the installation of ROS, the official website ( http://wiki.ros.org/indigo/Installation/Ubuntu) has a detailed installation tutorial, so I won't repeat it here.
This experiment is developed on the basis of the previous blog-the reading and display of camera image information , which realizes the reading and display of camera data. The collected image data can be used as the input of the ORB-SLAM2 system.
The previous blog created a workspace catkin_wsand moved the ORB-SLAM2 project to its subfolder src.
(1) Add the included Examples/ROS/ORB_SLAM2path to the ROS_PACKAGE_PATHenvironment variable. Open the .bashrcfile and add the following line at the end.

export ROS_PACKAGE_PATH=${ROS_PACKAGE_PATH}:PATH/ORB_SLAM2/Examples/ROS

Where PATH is the path where the ORB-SLAM2 project is located, and my path is /home/wb/catkin_ws/src. In addition, ./bashrcwrite in the file

source /home/wb/catkin_ws/devel/setup.sh

Finally execute the command in the terminal

source ./bashrc

(2) Compile ORB-SLAM2 under ROS. The
default subscription topic of ORB-SLAM is /camera/image_raw, and usb_camthe topic published by the node is /usb_cam/image_raw, so you need ros_mono.ccto modify the topic of subscription in it. Pay special attention to this point. Because the source file changes must be recompiled, which is very time-consuming.

$ chmod +x build_ros.sh
$ ./build_ros.sh

(3) Run the
monocular node Before running the monocular node, first connect the camera to the computer, and then start the usb_camnode.

$ roslaunch usb_cam usb_cam-test.launc
$ rosrun ORB_SLAM2 Mono PATH_TO_VOCABULARY PATH_TO_SETTINGS_FILE

4 Experimental results

After starting the monocular node, it must first be initialized, and the camera will be shifted left and right until the ORB feature points are extracted from the image. Then the handheld camera moves in the laboratory space. The ORB-SLAM2 system extracts ORB features from the input image at a rate of 30 frames per second. After a series of calculations, the camera's movement trajectory and a sparse map of the surrounding environment are finally output.
note

1. In order to be able to initialize smoothly, a scene with rich textures must be selected and the camera must be shifted to a certain extent.
2. It cannot move fast, it is better to keep moving at a constant speed, and keep the current frame and the previous frame with a sufficient number of matching feature points, otherwise the tracking will fail.
3. After the tracking fails, you should go back to the previous scene and perform relocation.
4. The performance of the computer should not be too low and keep the battery fully charged.

I held the camera in my hand and moved around my workstation in the laboratory for a while, and the final result of the operation is shown below.

The small green squares in the left window of the figure are the extracted image ORB features, the green lines in the right window represent the movement trajectory of the camera, and the blue squares represent the spatial position (key frame) during the movement of the camera, and the black dots And the red dot represents a sparse map of the environment (black represents historical road signs, and red represents current road signs).

5 Summary

At this point, I can compile and run such open source projects. Although I have experienced various error reports, I am familiar with the operation of Linux system and ROS system, cmake, shell, OpenCV, etc., in order to be able to develop by myself, I need more In-depth study.
As a beginner, starting with an open source project, learning its theory and engineering implementation, is a shortcut to getting started, and then make certain improvements on the basis of it.
However, engineering implementation is only part of scientific research. As a graduate student, it is more important to grasp a small problem in SLAM and see if we can improve or compare existing algorithms. So we still have to start from the paper, find inspiration, realize it, and verify the effect.
There is still a long way to go, and the road of scientific research has just begun.