Overview of Fusion Algorithms

Multi -sensor Information Fusion (MSIF): Information processing that uses computer technology to analyze and synthesize information and data from multiple sensors or multiple sources under certain criteria to complete the required decisions and estimates. Processes, in short, are used for information processing techniques involving multiple or multiple sensors at different locations.

Preface

The necessity of multi-sensor fusion : mainly to use strengths and avoid weaknesses, redundant design, and improve the safety factor of the entire vehicle. Multi-sensor fusion systems can achieve far more than the sum of the functions that these independent systems can achieve. Using different sensor types provides additional redundancy in the event that all sensors of a particular type fail. Such errors or malfunctions may be caused by natural causes (such as a thick fog) or man-made phenomena (such as electronic or human interference with cameras or radars). The advantages and disadvantages of each sensor are as follows:

• Camera: It is sensitive to the color and texture of the target and can complete tasks such as target classification, detection, segmentation, and identification. However, it cannot obtain accurate detection distance and is easily affected by lighting and weather conditions.

• LiDAR: Accurate 3D information of the target can be obtained, and the detection range can reach 150 meters. It is not sensitive to light and can work normally at night. However, the angular resolution is large, the targets are sparse, the target texture cannot be obtained, the classification is inaccurate, and the performance will decrease in severe weather such as rain, fog, and snow. It is also sensitive to dust and water mist and prone to noise.

• Radar: It can provide accurate distance and speed information, and the detection range is relatively long. It can work around the clock, but the resolution is low and it cannot provide object height information.

1. Concept

• Hardware synchronization (hard synchronization): The same hardware issues trigger collection commands at the same time to achieve time synchronization of collection and measurement of each sensor, so as to collect the same information at the same time.
• Software synchronization : time synchronization, space synchronization.
  Time synchronization (timestamp synchronization, soft synchronization): The collection periods of sensors are independent of each other, and there is no guarantee that the same information will be collected at the same time.
  Spatial synchronization: Convert the measurement values ​​of different sensor coordinate systems into the same coordinate system. When the laser sensor moves at high speed, the intra-frame displacement calibration at the current speed needs to be considered.

2. Principle

(1) Multiple different types of sensors collect data on the observation target;
(2) Feature extraction is performed on the sensor output data and feature vectors representing the observation data are extracted;
(3) Feature vectors are identified and processed to complete the sensor's description of the target;
(4) Association: Group the description data of each sensor about the target according to the same target; (
5) Use the fusion algorithm to synthesize the sensor data of the target to obtain a consistent explanation and description of the target.

3. Prerequisites for integration

Three identical differences : the same target appears at the same world coordinates of different categories of sensors at the same time.
Unified clock : synchronize timestamps from different sensors.

• GPS timestamp synchronization method : If the sensor hardware supports this method, the data packets given by the sensor will have a global timestamp, using the same clock based on GPS.
  There is a problem: the data frequencies of different sensors are different, such as lidar 10Hz and camera 25/30Hz, so there is still a delay between different sensors. Even if the most recent frame can be found by finding adjacent timestamps, if the difference between the two timestamps is large, is large, the obstacle is moving, which may eventually lead to a large synchronization error.
  

As shown in the figure above, at time T1, sensor 2 generates a data. If we want to perform time synchronization, we need to find the data of sensor 1 and sensor 3 at the corresponding time. The actual search method is to find the closest time difference to sensr 2. Sensor data packet.

• Hard synchronization: LiDAR can be used as a source to trigger other sensors. When the LiDAR turns to a certain angle, the camera at that angle is triggered. This method can alleviate the error caused by searching for timestamps and improve the data alignment effect.

Unified coordinate system : one is motion compensation, and the other is sensor calibration.

• Motion compensation: For long-period sensors, the lidar period is 100ms. When the sensor collects data, the vehicle is in different positions at the beginning and end of the period, resulting in different coordinate systems for the data collected at different times. Therefore, it needs to be based on the vehicle body. Movement compensates for the sensor data.
  As shown in the figure below: the dotted line part can be considered as the world coordinate system, the red point represents a static obstacle, and there is a stable coordinate (5, 5) in the coordinate system. The blue part represents the self-driving car's own local coordinate system, which means (4,0) of the world coordinate system is the origin of the local coordinate system. At time T+1, this local coordinate system moves to the position (6, 0), that is, the self-driving car moves forward by 2 along the X direction. That is to say, at time T, the coordinates of the obstacle in the local coordinate system are (1, 5), and at time T+1, its coordinates become (-1, 5).
  

• Sensor calibration: internal parameter calibration and external parameter calibration.
  Internal parameters: solve the transformation between each sensor and the world coordinate system;
  external parameters: the world coordinate system, solve the transformation between different sensors. Sensor external reference calibration relies on the accurate internal reference calibration of the sensor.

4. Fusion classification

According to the different positions of fusion, it can be divided into pre-fusion and post-fusion; according to the abstract level of information processing, it can be divided into three levels: data level, feature level and decision-making level.

4.1. Pre-fusion and post-fusion

Before fusion
, there is only one perception algorithm to perceive the fused multi-dimensional comprehensive data.

The data is fused at the original layer. The fused data is like a super sensor, which can simultaneously observe infrared, camera, RGB, and lidar point cloud information.

post-fusion

Each sensor independently processes the generated target data.

Each sensor has its own independent perception. After all sensors complete the generation of target data, the main processor performs data fusion.

4.2. Data-level fusion, feature-level fusion and decision-level fusion

Data-level fusion,
also called pixel-level fusion, belongs to the underlying data fusion: raw observation data from multiple sensors are directly fused, and then feature vectors are extracted from the fused data for judgment and recognition.

The advantage of data-level fusion is that there is no problem of data loss and it provides subtle information that other fusion levels cannot provide. The disadvantage is that the calculation results are more accurate, but the calculation amount is larger, the processing time is longer, and the real-time performance is poor.

Feature-level fusion
belongs to the intermediate level fusion: first extract representative features from the original observation data provided by each sensor, and then fuse the features into a single feature vector; selecting appropriate features for fusion is the key; feature information includes edges, directions , speed, shape, etc.

Feature-level fusion can be divided into two categories: target state fusion and target feature fusion .

• Target state fusion : Mainly used in the field of multi-sensor target tracking. The fusion system first pre-processes sensor data to complete data registration, and then the fusion process mainly realizes parameter correlation and state estimation.
• Target feature fusion : joint recognition of feature layers, which is essentially a pattern recognition problem. Before fusion, features must be associated and then classified into meaningful combinations.

Among the three levels of fusion, feature-level fusion is relatively well developed, and because feature correlation technology has been established in the sub-feature layer, the consistency of fused information can be ensured, and the computational load and communication bandwidth are relatively low. However, due to the partial data The rejection accuracy has decreased.

Decision-level integration

High-level fusion, the output is a joint decision result. In this method, the information collected by each sensor is transformed, including preprocessing, feature extraction, identification or judgment, to establish a preliminary conclusion about the observed target, and finally make a decision based on certain criteria and the credibility of each judgment. Make the optimal decision. Decision-level fusion starts from the needs of specific decision-making problems and makes full use of various feature information of the measurement objects extracted by feature-level fusion.

This fusion is theoretically more precise or specific than any single-sensor approach. It has high flexibility in information processing. The system has very low requirements for information transmission bandwidth, can effectively integrate different types of information reflecting various aspects of the environment or targets, and can process asynchronous information.

This fusion method condenses the sensor data and produces relatively least accurate results, but it requires the least communication bandwidth. Due to the temporal changing characteristics of the environment and goals, the difficulty in obtaining prior knowledge, the huge characteristics of the knowledge base, object-oriented system design requirements, etc., the development of this fusion theory and technology is still subject to certain restrictions.

Comparison of data-level fusion, feature-level fusion and decision-level fusion.
Data-level fusion is the lowest level of fusion. It is performed on the basis of no or minimal processing of the original sensor information. It requires each fused sensor information source to have accurate Fusion at any level of abstraction down to one pixel registration accuracy. Its advantage is that it can provide detailed information that the other two levels of fusion do not have, but it also has limitations in the following aspects.

(1) Due to the large amount of sensor information it has to process, the processing cost is relatively high.

(2) Due to the poor stability of sensor information, especially during target detection and classification, high error correction processing capabilities are required during fusion.

(3) Since the information at this level requires a pixel-level registration relationship between the sensor information, each sensor information is required to come from homogeneous sensors.

(4) Due to its large communication volume, its anti-interference ability is poor.

   决策层融合的优缺点正好与数据层融合相反。其传感器可以是异质传感器，预处理代价较高，而融合中心处理代价小，整个系统的通信量小，抗干扰能力强。由于处理效果很大程度取决于各个传感器预处理的性能，而传感器预处理一般是简单的处理，其性能一般不太高，故融合中心的性能要比数据层融合性能差些。特征层融合是上述两种信息融合的折中形式，兼容了两者的优缺点。各层次融合的优缺点如下图所示：

Which level of data fusion method a system adopts depends on the specific requirements of the system. There is no universal structure that can be applied to all situations or applications. For specific engineering applications of multi-sensor fusion systems, factors such as sensor performance, system computing power, communication bandwidth, expected accuracy, and financial capabilities should be comprehensively considered to determine which level is optimal. In addition, in a system, fusion may also be performed at different fusion levels at the same time. An actual fusion system is a combination of the above three fusions. The higher the level of fusion, the faster the processing speed and the greater the amount of information compression. Big losses are bigger.

5. Typical fusion algorithms

(1) Weighted average method
The most simple and intuitive method of signal level fusion method is the weighted average method, which weights the redundant information provided by a group of sensors and averages the result as the fusion value. This method is a method that directly operates on the data source.

(2) Kalman filtering method
Kalman filtering is mainly used to fuse low-level real-time dynamic multi-sensor redundant data. This method uses the statistical properties of the measurement model to determine the optimal fusion and data estimation in a statistical sense. If the system has a linear dynamics model and the errors between the system and the sensor conform to the Gaussian white noise model, the Kalman filter will provide the only statistically optimal estimate for the fused data.

The recursive nature of Kalman filter eliminates the need for large amounts of data storage and computation for system processing. However, when using a single Kalman filter to perform data statistics on a multi-sensor combined system, there are many serious problems, such as: ① When the combined information is largely redundant, the amount of calculation will increase dramatically by the cube of the filter dimension, and in real time Sexual insatiability. ② The increase in sensor subsystems increases the probability of failure. When a system fails and is not detected in time, the failure will contaminate the entire system and reduce reliability.

(3) The multi-Bayesian estimation method
treats each sensor as a Bayesian estimate, and synthesizes the associated probability distribution of each individual object into a joint posterior probability distribution function. By minimizing the likelihood function of the joint distribution function, Provides the final fusion value of multi-sensor information, and fuses the information with a priori model of the environment to provide a characteristic description of the entire environment.

(4) DS evidence reasoning method
This method is an expansion of Bayesian reasoning and includes three basic points: basic probability assignment function, trust function and likelihood function.

The reasoning structure of the DS method is top-down and is divided into three levels: the first level is target synthesis, whose function is to synthesize the observation results from independent sensors into a total output result (ID); the second level is inference , its function is to obtain sensor observation results and make inferences, and expand the sensor observation results into target reports. The basis of this reasoning is: a certain sensor report will logically produce certain target reports that are credible with a certain degree of credibility; the third level is update, and each sensor generally has random errors, so it is fully independent in time. A set of consecutive reports from the same sensor is more reliable than any single report. Therefore, before inference and multi-sensor synthesis, the sensor observation data must be combined (updated) first.

(5) Fuzzy logic reasoning
Fuzzy logic is a multi-valued logic that represents the degree of truth by specifying a real number between 0 and 1 (equivalent to the premise of the implicit operator), allowing the uncertainty in the process of fusion of multiple sensor information Direct representation in the reasoning process. If some systematic method is adopted to inferentially model the uncertainty in the fusion process, consistent fuzzy inference can be produced.

Compared with probabilistic and statistical methods, logical reasoning has many advantages. It overcomes the problems faced by probability theory to a certain extent. The representation and processing of information is closer to the human way of thinking, and is generally more suitable for high-level applications ( such as decision-making). But logical reasoning itself is not mature and systematic enough. In addition, because there are many subjective factors in the description of information by logical reasoning, the representation and processing of information lack objectivity.

The actual value of fuzzy set theory for data fusion lies in its extension to fuzzy logic. Fuzzy logic is a multi-valued logic, and the degree of membership can be regarded as an inexact representation of the true value of the data. In the MSF process, existing uncertainties can be directly represented by fuzzy logic, and then multi-valued logic reasoning is used to merge various propositions according to various calculations of fuzzy set theory, thereby achieving data fusion.

(6) Artificial neural network method
Neural network has strong fault tolerance and self-learning, self-organizing and adaptive capabilities, and can simulate complex nonlinear mapping. These characteristics and powerful nonlinear processing capabilities of neural networks just meet the requirements of multi-sensor data fusion technology processing. In a multi-sensor system, the environmental information provided by each information source has a certain degree of uncertainty. The fusion process of these uncertain information is actually an uncertainty reasoning process. The neural network determines the classification standard based on the similarity of samples accepted by the current system. This determination method is mainly reflected in the weight distribution of the network. At the same time, a learning algorithm can be used to obtain knowledge and obtain an uncertainty reasoning mechanism. By utilizing the signal processing capabilities and automatic reasoning functions of neural networks, multi-sensor data fusion is achieved.