Summary
Self-driving cars rely on various sensors to gather information about their surroundings. The behavior of the vehicle is planned based on the perception of the environment, so its reliability is of paramount importance for safety reasons. Active LiDAR sensors are able to create an accurate 3D representation of a scene, making them a valuable addition to autonomous vehicle environment perception. Due to light scattering and occlusion, the performance of lidar can change under adverse weather conditions such as fog, snow or rain. This limitation has recently fueled a large body of research on methods to mitigate perceived performance degradation. This paper collects, analyzes and discusses different aspects of LiDAR-based environmental perception for adverse weather conditions. Topics such as the availability of appropriate data, raw point cloud processing and denoising, robust perception algorithms, and sensor fusion to mitigate disadvantages caused by adverse weather are discussed. In addition, the paper further identifies the most pressing gaps in the current literature and identifies promising research directions.
introduce
LiDAR sensors have recently gained increasing attention in the field of autonomous driving [1]. It provides sparse but accurate depth information, making it a valuable complement to more deeply researched sensors such as cameras and radars. A lidar sensor is an active sensor, meaning it emits pulses of light that are reflected by the environment. The sensor then captures the reflected light and measures the distance of the environment based on the elapsed time. In addition to time, other characteristics can be evaluated, such as the amount of light and the prolongation of the signal. In most cases there are mechanical components combined with multiple laser diodes to create a sparse point cloud of the complete scene [1]. There are a variety of different sensors on the market.
LiDAR sensors have different disadvantages in adverse weather conditions. First, sensor freezing or other mechanical complications may occur at freezing temperatures. Internal and structural factors such as sensor technology, type and mounting location play a role in the degree of degradation. Additionally, adverse weather can affect intensity values, point counts, and other point cloud features (see Figure 1). Generally speaking, when particulate matter is encountered in the air due to dust or severe weather, the emitted light is scattered or diverted backwards. This results in noisy distance and reflectance measurements in the point cloud, as some laser pulses return to the sensor prematurely or are lost in the atmosphere. Noise is especially detrimental when applying scene understanding algorithms. Maintaining reliably high predictive performance is especially important in such safety-critical use cases. Therefore, countermeasures are needed to minimize the degradation of LiDAR perception performance under adverse weather conditions, or at least to detect the sensor's limitations in real-world scenarios.
Most state-of-the-art algorithms rely on deep learning (DL) algorithms, which rely on large amounts of data, to derive general characteristics of the environment. While there is a line of research focusing on unsupervised perception, most recent work requires corresponding labeling of raw data. This includes bounding boxes for object detection and pointwise class labels for semantic segmentation. Manually labeling sparse and additionally noisy point clouds is not only difficult, but also costly and error-prone. Therefore, the question of how to simulate or augment existing point clouds with weather-specific noise is of particular interest.
Although there is a large body of research on analyzing the performance degradation of LiDAR sensors under adverse weather conditions, there is a lack of comprehensive summaries on algorithmic coping strategies to improve perception. Additionally, the investigation into autonomous driving in adverse weather conditions addressed weather-induced sensor degradation but did not identify weather-related issues specific to lidar sensors. This article summarizes and analyzes various approaches to coping with lidar-sensed adverse weather conditions. The paper therefore addresses this topic from three different perspectives:
- Data availability: real-world and synthetic datasets for developing robust lidar perception algorithms;
- Point cloud manipulation: sensor-specific weather robustness and perception-independent point cloud processing (e.g. weather classification, point cloud denoising);
- Robust Perception: Robust perception algorithms that can handle weather-induced noise in point clouds by fusing multiple sensors, making adjustments during training, or improving the overall robustness of the perception model.
Finally, the missing gaps in current technology and the most promising research directions are summarized.
adverse weather data
To train a DL model on any kind of perception task, a large amount of data is required. For supervised methods that still dominate, these data even have to be labeled by automatic labeling methods or manually. Either way, obtaining accurately labeled sparse lidar data is expensive and cumbersome, and is hampered even more when raw point clouds are corrupted by weather-induced noise.
Therefore, valuable datasets with high-quality labels are required. In general, there are three options to obtain a LiDAR point cloud with weather-characteristic noise patterns: real-world recording, augmented point cloud, and simulated point cloud. The first one was generated using a test car with an appropriate sensor setup in adverse weather conditions. The latter approach requires physical models or DL-based methods to create partial or entire point clouds.
real world dataset
Most existing datasets for lidar perception benchmarks are recorded under favorable weather conditions. In order to use the developed perception algorithms in the real world, the underlying dataset must reflect all weather conditions. In addition to sunny weather conditions, there are some extensive datasets that explicitly include rain, snow, and fog.
Table I shows an overview of publicly available datasets used to study lidar perception in severe weather conditions. The datasets were recorded under different conditions and vary widely in size. Most of these were actually recorded in real-world driving scenarios, while two of them were (in part) from the weather room. Meteorological chambers have the advantage of having complete control over weather conditions and surroundings, i.e. in terms of obstacles. Still, they don't adequately reflect real-world situations.
Furthermore, each dataset uses a different sensor setup. [27] specifically benchmark LiDAR makes and models under severe weather conditions. With the exception of lidar sensors, all datasets provide RGB camera recordings, and some datasets even include radar, stereo, event, gated or infrared cameras.
These datasets are designed to solve different perception and driving tasks for autonomous vehicles. Almost all sensor setups (except [21]) include localization and motion sensors, namely GPS/GNSS and IMU. Therefore, they are suitable for developing and testing SLAM algorithms. All datasets provide labels for object detection or pointwise segmentation, except [29] which only provides motion GT.
Finally, all datasets include some metadata about weather conditions. This is critical for developing almost any type of perception model in adverse weather conditions. Knowing the intensity and nature of the surrounding weather conditions is crucial, at least for a thorough validation. Only one dataset provides point-by-point weather labels, snowfall and snow accumulation on roadsides.
A dataset consisting of real-world records has the advantage of being highly realistic. The downside is that labels for recorded scenes are only partially available (point by point), or, if the data is recorded in a weather chamber, limited to more complex real world scenarios. Manually labeling lidar point clouds point-by-point in adverse weather conditions is especially challenging because in many cases it is impractical to distinguish clutter or noise from actual reflected signals.
enhanced weather
Extending adverse weather effects to existing datasets provides an efficient way to generate large amounts of data, rather than tediously collecting and labeling new datasets of different adverse weather effects. Often, physics-based or empirical augmentation models are used to augment certain adverse weather effects into sunny-weather point clouds, whether they are driven from the real world or from simulations such as . This allows obtaining scenes corrupted by weather-specific noise while preserving all the interesting edge cases and annotations already present in the dataset.
The augmentation method defines the mapping from sunny weather points to corresponding points under adverse weather conditions. For this, reference is often made to the theoretical lidar model in [32], which models the effects of unfavorable rain, fog, and snow. It models the received intensity distribution as a linear system by convolving the transmitted pulse with the scene response. Scene Response models reflections on solid objects as well as backscattering and attenuation due to bad weather.
A more practical fog enhancement is introduced in [9], which can be directly applied to point clouds. It is based on the maximum line-of-sight distance as a function of measured intensity, LiDAR parameters, and optical visibility in fog. If the distance of the clear weather point is lower than the maximum viewing distance, random scattered points will appear, or the point will be lost with a certain probability. The model is adapted to rainfall by converting visibility parameters and scatter probabilities into rainfall rates.
However, these models ignore the considered beam divergence of emitted lidar pulses for rain enhancement. Here, the number of intersections of oversampled beams simulating beam divergence with spherical raindrops is calculated. If the number of intersections exceeds a certain threshold, a scatter point is added. The augmentation method in [35] extends this approach such that missing points are possible. Also, it works well with snow and fog.
Another augmentation for fog, snow and rain is introduced in [36]. The model operates in the power domain and does not rely on, for example, computing intersections like the previously discussed methods. Furthermore, beam divergence is modeled using a computationally more efficient scatter point distance sampling strategy. Typically, the model starts by comparing the attenuated power reflected by solid objects and randomly sampled scatterers with a distance-dependent noise threshold. Scatter points are added if their power exceeds that of the solid object. A point is lost if it falls below the distance-dependent noise threshold.
In addition to physics-based models, empirical models can also be used for augmentation. Empirical augmentation of spray rolled up by other vehicles can be found in [38]. Central to this model is the observation from specialized experiments that sprays are organized into clusters. Another data-driven approach is proposed in [39], which relies on spray scenarios from the Waymo dataset. In [40], a more computationally expensive spray enhancement method is proposed, which relies on a renderer with a physics engine.
Finally, DL-based methods can be applied to adverse weather enhancement. In [41], inspired by image-to-image translation, a generative adversarial network (GAN) based method is proposed, which is able to transform point clouds from sunny to foggy or rainy. They qualitatively compared their results to real fog and rain point clouds from weather chambers.
However, assessing the quality and realism of augmentation methods is challenging. Some authors use weather chambers or other controlled environments to allow comparisons with real-world weather effects. Furthermore, an augmentation method is generally considered realistic if it contributes to perceived performance in real-world adverse weather conditions.
Point cloud processing and denoising
This section presents methods on how to deal with adverse weather conditions, which are based on sensor technology or point clouds, i.e. independent of the actual perception task. Therefore, the paper analyzes general sensor-related weather robustness and the possibility to estimate the degree of performance degradation depending on weather conditions. Furthermore, there are numerous studies using classical denoising methods and DL to remove weather-induced noise from lidar point clouds.
Sensor Dependent Weather Robustness
Depending on the technology, features and configuration, different lidar models are more or less affected by weather conditions. Due to eye safety constraints and suppression of ambient light, two operating wavelengths for lidar sensors dominate: 905nm and 1550nm, with 905nm being the majority of available sensors. This is partly due to better performance in adverse weather conditions, i.e. lower absorption of raindrops, better reflectivity in snow and less degradation in fog. For a comprehensive discussion of lidar techniques and wavelengths in adverse weather conditions, we refer to [17].
In addition, the performance of full waveform lidar (FWL) under severe weather conditions is investigated. FWL measures not just one or two echoes, but all the weaker ones, effectively measuring more noise, but also gathering more information about the surrounding environment. Although FWL requires high computational resources, it has proven useful for analyzing the surrounding medium, which can form the basis for understanding even changing conditions and adjusting them dynamically.
Sensor Degradation Estimation and Weather Classification
Since lidar sensors degrade differently under different weather conditions, estimating the degree of sensor degradation is the first step in dealing with corrupted lidar point clouds. Progress has been made in developing methods to better identify sensing limits to prevent false detections from propagating into downstream tasks.
First, several studies on characterizing the degradation of sensors under various weather conditions have established a solid basis for the calibration and further development of sensors in severe weather conditions, although their weather classification capabilities have not yet been evaluated.
The first work to actually simulate the effect of rainfall on LiDAR sensors is presented in [33]. The authors propose a mathematical model derived from the lidar equations and allows performance degradation estimation based on rainfall rate and maximum sensing range.
In subsequent research work, the estimation of sensor degradation under severe weather conditions was formulated as an anomaly detection task and a verification task. The former employs a DL-based model that aims to learn a latent representation that distinguishes clear LiDAR scans from rainy LiDAR scans, enabling quantification of performance degradation. The latter approach proposes the use of reinforcement learning (RL) models to identify failures in object detection and tracking models.
While the methods described above aim to quantify the degradation in sensor performance itself, another line of research focuses on the classification of surrounding weather conditions (i.e., clear, rainy, fog, and snow). With the help of classical machine learning methods (k-Nearest Neighbors and Support Vector Machines) based on hand-crafted features3 from lidar point clouds, satisfactory results are achieved: [10] proposed a feature set to perform pointwise weather categories.
[51] developed a probabilistic model for frame-by-frame regression of rainfall rate. Working with experts, they accurately inferred the rainfall rate from the lidar point cloud.
It should be noted that most of the methods are trained and evaluated on data collected by weather chambers. While the ability to carefully control weather conditions allows for high reproducibility, the data often do not accurately reflect real-world conditions. In order to evaluate the classification ability of each method, a thorough study on real-world data is necessary [50].
Point cloud denoising
Weather effects are reflected in lidar point clouds in specific noise patterns. As mentioned in Section 1, they may affect factors such as the number of measurements in the point cloud and the maximum sensing range. Instead of enhancing point clouds with weather-specific noise, point clouds can be denoised by various methods to reconstruct clean measurements. In addition to classical filtering algorithms, some DL-based denoising work has recently emerged.
In addition to applying perceptual tasks such as object detection on denoised point clouds, metrics such as precision (preserving environmental features) and recall (filtering out weather-induced noise) are crucial for evaluating the performance of classical filtering methods. To calculate these metrics, point-by-point markers are required to account for weather categories such as snow particles.
Radius Outlier Removal (ROR) filters out noise based on the neighborhood of any point. This becomes problematic for LiDAR to measure distant objects, as point clouds become naturally sparse. State-of-the-art methods address this by dynamically adjusting the threshold based on the sensing distance (Dynamic Radius Outlier Removal (DROR)) or by taking into account the average distance of each point’s neighbors in the point cloud (Statistical Outlier Removal). Both methods exhibit high runtimes, making them almost unsuitable for autonomous driving. Both Fast Cluster Statistical Outlier Removal (FCSOR) and Dynamic Statistical Outlier Removal (DSOR) propose ways to reduce computational load while still removing weather artifacts from point clouds.
Denoising methods for roadside lidar rely on background models of historical data (which can be used for fixed roadside sensors), combined with the rationale used in classical denoising to identify dynamic points. [57] filter weather noise from real objects with the help of intensity thresholding. Unfortunately, this is not readily applicable to lidar sensors mounted on moving vehicles.
Contrary to classical denoising methods, DL-based lidar point cloud denoising is popular because the model can directly understand the underlying structure of weather noise: First, a convolutional neural network (CNN)-based model has been used to Effective weather denoising. Using temporal data for differentiation further exploits weather-specific noise removal, since, naturally, weather noise changes more frequently than the scene background or even objects within the scene. CNN-based methods (especially voxel-based methods) outperform classical denoising methods in terms of noise filtering. Also, since GPUs compute faster, they have lower inference times.
In addition to supervised CNN methods, unsupervised methods like CycleGANs are able to transform noisy point cloud inputs into clean LiDAR scans. However, they are still noisy in nature and the resulting point clouds are difficult to verify with respect to their authenticity.
Robust lidar perception
Although there are good efforts to reduce domain shifts from adverse weather, there are several possible ways to make lidar perception models more robust to adverse weather conditions, independent of the quality and noise level of the data. Here are three workflows: leveraging sensor fusion, augmenting training with data augmentation using weather-specific noise, or compensating for performance degradation with a general approach to model robustness against domain shift. It should be noted that, apart from object detection, sensor fusion approaches are the only ones that solve multiple perception tasks. As far as the paper is aware, there is no literature on other perceptual tasks such as semantic segmentation.
Using Sensor Fusion to Fight Severe Weather
In general, it can be said that each sensor in the autonomous driving sensor set has its advantages and disadvantages. The most common sensors in this sensor group are RGB cameras, radar, and lidar. As mentioned in Section 1, LiDAR perception suffers when it encounters visible airborne particles such as dust, rain, snow, or fog. The camera is more sensitive to strong light incidents and halo effects. Radar, in turn, is immune to both, but lacks the ability to detect static objects and fine structures. Thus, it forces itself to fuse different sensors in order to mitigate their respective shortcomings and facilitate robust perception under different environmental conditions.
Early work on sensor fusion to combat adverse effects of weather on sensor perception focused on developing robust data association frameworks. A recent stream of research utilizes DL-based methods for robust multimodal perception and mainly addresses early versus late fusion for robustness in harsh weather conditions.
The choice of pre- or post-fusion seems to depend on sensor choice, data representation, and expected failure rate. Assuming not all fused sensors are degraded equally and at least one of them is fully functional, late fusion seems to be better than early fusion. In this case, the model is able to process sensor streams independently, it is able to rely on working sensors and ignore faulty sensors. Instead, early fusion of radar and lidar depth maps helps filter false detections for clean scans.
Data representation is another factor that partially helps answer the question of early versus late fusion. Bird's eye view (BEV) of lidar sensors greatly facilitates object detection by improving the resolvability of obejcts. Consequently, any model that has learned to rely on the respective lidar features will suffer a performance loss when the lidar data is corrupted. A complete failure of the sensor was successfully resolved using a teacher-student network.
Ultimately, some sensor fusion methods rely on combining early and late fusion into a single model and exploit temporal data and concepts such as region-based fusion [72] or attention maps [73]. Another possibility is the fusion of adaptive, entropy control proposed in [21].
In addition to predictive performance, model runtime should also be considered when developing new perception methods. [68] introduced a new metric that combines predictive performance for driveable spatial segmentation with inference runtime. Interestingly, the lidar-only model scored the best on this metric.
There is no doubt that it is convenient to compensate for sensor failures with unaffected sensors in adverse weather conditions. However, safety-critical applications such as autonomous driving can become more reliable by working to improve perception in adverse weather conditions using only lidar.
Enhance training with data augmentation
While data augmentation is widely used in DL training strategies, what is particularly challenging is the generation of weather-specific noise. Section II presents various methods for generating weather-specific noise in lidar point clouds. Utilizing data augmentation during the training of perception models is a radial approach to point cloud denoising, which has been discussed in Section 3. The goal is not to remove the noise caused by the weather, but to get the model used to this exact noise. Weather augmentation has been shown to be more effective than denoising in terms of robustness, which provides valuable hints as to which research directions should be emphasized in the future.
In general, several works demonstrate the benefit of such data augmentation at training time by evaluating such data on the task 3D object detection.
Much work has addressed the problem of selecting the best feature extractor for robust LiDAR perception under harsh weather conditions. Point-based and voxelization methods seem to be less susceptible to enhanced weather effects, at least for object detection, suggesting that some robustness can be achieved through careful choice of perception models. Furthermore, there appears to be an interaction between the model architecture and point cloud corruption due to severe weather. The wetland extension proposed in [4] only helped some models, showing that the detection problem caused by ray scattering is more or less severe, depending on the model architecture.
Furthermore, object size and shape seem to play a role in the degree of performance degradation of any detection model. This means that smaller and under-represented classes (such as the bicycling class in the STF dataset) are more susceptible to weather enhancement than better-represented classes, such as cars and pedestrians. Thus, the number of annotated objects in the (clear) training set is a good indicator of object detection performance even under adverse weather conditions. This shows that weather augmentation training not only helps improve detection performance in sunny weather conditions, interestingly, it also seems to have the opposite effect.
Robust Perception Algorithm
While fusion methods with complementary sensors can alleviate the weather-induced performance degradation of each sensor, they can only serve as solutions to current practical problems. Changes in weather conditions can be viewed as a special case of domain transfer, and thus the methods developed to bridge the domain gap can be applied to domain transfer from weather to weather (e.g., rain/fog/snow). [77] provides a comprehensive overview of adaptive methods in the current state of the art, but they mainly address issues related to different sensor resolutions or available data and their labels.
In [78], the authors propose dataset-to-dataset domain transfer, which indirectly includes weather changes. They employ a teacher-student setup for object detection, where the teacher is trained on Waymo Open (sunny), generates labels for part of Waymo Open and part of Kirkland (rainy), and the student is trained on all labels and applied to Kirkland. Interestingly, the students seemed to generalize better to the target area, suggesting they were able to cope with inclement weather. However, it should be noted that domain gaps are not limited to variations between weather conditions, and other factors such as sensor resolution and labeling strategies may mask gaps caused by weather.
The authors of [79] propose a robust object detection pipeline including attention mechanism and global contextual feature extraction, enabling the model to ignore weather-induced noise while understanding the entire scene. Although their method cannot perform well on both domains (KITTI, sunny and CADC, rainy), the joint training based on the maximum difference loss yields promising results and demonstrates good performance on both source and target domains. high performance. Here, again, it is not clear which elements of the model are attributable to variations in the weather conditions themselves, as dataset-to-dataset variations appear to be very strong.
[80] focused on mitigating weather-induced sensor degradation for RGB cameras and lidars. Although they utilize sensor fusion (derived from entropy fusion proposed in [21]) as well as data augmentation from two sensors, their work strongly promotes the utilization of a set of methods to bridge the gap with multiple unknown target domains, for target detection. They achieve this by introducing domain discriminators and self-supervised learning via pre-trained policies. Their results show that their multi-modal, multi-object domain adaptation method generalizes well to e.g. foggy scenes.
Discussion and conclusion
In this survey paper, the paper provides an in-depth analysis and discussion of the availability of training data for deep learning algorithms, perception-independent point cloud processing techniques for detecting weather conditions and denoising lidar scans, and robust lidar perception the latest method. In the following, the most promising research directions are summarized and the remaining gaps are identified.
Adverse weather data: There are several autonomous driving datasets, including lidar sensors, that also cover adverse weather conditions. Most of them provide object labels, but only one has pointwise class labels. Clearly, suitable real-world datasets are needed to train and validate the growing number of deep learning-based lidar perception algorithms. Some works employ weather-specific data augmentation to model adverse weather effects, however, a method to evaluate the realism of the generated augmentations is lacking.
Point cloud processing and denoising: Different lidar technologies respond differently to adverse weather conditions. While the degradation of sensors under severe weather conditions has been intensively studied, there is a lack of systematic analysis of the impact on perception algorithms. Here, a method for sensor degradation estimation would be useful. Also, research is currently being done on cloud denoising, but existing statistical methods have been shown to be less effective than using weather augmentation in training. Modern methods, such as those based on CNNs or GANs, may bridge this gap.
Robust lidar perception: A lot of research has focused on mitigating sensor degradation with sensor fusion. While this yields convincing results, improving lidar-only perception in adverse weather conditions should not be overlooked. Sophisticated domain adaptation methods such as anomaly detection or uncertainty modeling may help to address this issue. Observing the presence of weather-induced noise in lidar point clouds from a different perspective may open new research streams bridging gaps in the field caused by adverse weather conditions. The quality of investigating gaps in this domain will suggest the potential of general domain adaptation methods.
Source| Heart of Autopilot
reference
[1] Survey on LiDAR Perception in Adverse Weather Conditions