The previous article shared two reports about Oculii 4D radar. The data set is a very important research tool. For the 4D radar field, it is at the forefront of research. There are few public data sets. The currently available data sets are: 
The data sets in these articles have been written by many bloggers, but they are often translations of the original texts of the authors. The actual acquisition and use are basically not covered. This article will explain these data sets in more depth and concretely, and give How to get it, how to use it, precautions, etc.
Table of contents
4. TJ4DRadSet (2022) is currently not open source
Update log:
The Tj4D data set is already available, and it needs to sign a non-disclosure agreement and send it to the mailbox to obtain it from the other party (it will start processing after February 7). -------2023. 0207
1. Astyx dataset: Time 2019
原文：Automotive Radar Dataset for Deep Learning Based 3D Object Detection
It is written in the article that its acquisition method is: www.astyx.net (I did not find the data set entry after I opened it)
Current acquisition method: https://github.com/under-the-radar/radar_dataset_astyx
The data set is relatively small and can be obtained directly.
Next, briefly introduce this data set:
The author's purpose of establishing this dataset is to provide the research community with high-resolution radar data, and to facilitate and facilitate research on algorithms utilizing radar sensor data. Because there are few data sets that pay attention to radar, the only autopilot data set that contains radar data at that time is the nuScense data set, but the radar data in it is 2D and very sparse (about 100), the point cloud in this data set can be Reaching about 1000 points, the equipment used to collect this data set is:
At present, the performance of the radar is average, the ranging range is 100m, and the field of view is 110*10; the lidar uses Velodyne 16 lines and the camera uses an 8-bit 120W pixel gray point camera.
Calibration adopts the classic checkerboard and corner inversion method, and the calibration of internal parameters is carried out offline. Calibration of external parameters is performed in pairs.
For the annotation of the GT box: the accuracy of the lidar to reflect the real physical coordinates is the highest among them, and the camera is very important. Here, the author regards it as a fine-tune for the laser; for the radar, the angular resolution and detection of the lidar Due to distance limitations, long-distance targets may not appear in the "common view area" of the sensor (this is similar to don't care in kitti). Here, as long as the radar and camera can see the common view, they will be marked . As shown below:
At the same time, invisible is also labeled, which means that those physical objects do not have any lidar or camera measurements, but are clearly visible in the radar data (for example, through multipath reflections propagating under other cars), and can be related by time reference Up - Becomes visible before or after during data logging. ( It means that objects that do exist, but cannot be captured by cameras and lidar, but can be found by radar through some methods are also marked ).
The authors extract the data for which the network is most uncertain about its decision-making from N frames of automatically pre-labeled data, and correct these data by manual fine-tuning to maximize the information gain of the network in the next training and pre-labeling rounds. (automatic labeling + manual)
This data set provides a total of 7 categories: Bus, Car, Cyclist, Motorcyclist, Person, Trailer, Truck
Annotated target attributes are:
The storage format of annotation is txt.
The point distribution of the target's lidar and radar: (different frames correspond to multiple targets and sizes)
Data distribution:
It can be seen that it is still very unbalanced. It can be seen from the picture on the right that the frames of the three cars are the most. The scene is also relatively simple.
The orientation distribution of the car car:
As can be seen from the figure below, the lidar is no longer visible to the AB and AB vehicles. According to the image and radar point cloud, it is distinguished according to the orientation.
Evaluation (without mentioning which network is used): Randomly split the dataset into training and test data using a 4:1 ratio, train two networks (radar camera and lidar camera) for 22k iterations, mini-batch of 16 and Classification, localization and localization accuracy of the results are evaluated by using an IoU threshold of 0.5.
evaluation result:
For car, the result of radar+camera is even better than that of Lidar+Camera.
Looking at the dataset shows that the dataset is very small, with only 546 annotated frames of short clips, each clip containing less than 10 frames.
The overall structure is:
The camera data format is jpg, the lidar data format is txt, which contains six dimensions of information (xyz, reflectivity, from which laser head and time stamp), and the radar data is also given in txt, which contains five dimensions of information (xyz, speed, and mag (should be intensity)) In addition, the calibration information is given in the form of json to transform each frame. GT annotations also use the json format, content examples:
{
"frame_flag": "",
"frame_index": 0,
"header": "Astyx Version_01_00",
"objects": [
{
"center3d": [
15.28,
4.24,
-0.46
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.66,
1.8,
1.45
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7020386802113796,
-0.03287019905010674,
0.005682565197980395,
0.711357118439284
],
"score": -1.0
},
{
"center3d": [
57.75593697820277,
-8.95030337350099,
1.7819997024076009
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.34,
1.81,
1.5098232605535091
],
"label_certainty": 0,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 0,
"orientation_quat": [
0.7194398517786975,
-0.03301020608906757,
0.0048577576720212606,
0.6937528581252859
],
"score": -1.0
},
{
"center3d": [
11.73,
4.59,
-0.64
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.15,
1.8,
1.45
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7058035625491359,
-0.03289943574160317,
0.005508833819764444,
0.7076217993891781
],
"score": -1.0
},
{
"center3d": [
9.145197640200534,
5.274276650959328,
-0.7580683445170724
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.77,
1.8,
1.45
],
"label_certainty": 1,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7199364568101033,
-0.03300232389003559,
0.004846831272768512,
0.6932379483266801
],
"score": -1.0
},
{
"center3d": [
6.479817221419508,
5.247589043270934,
-0.9141820375574657
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.48395807023072,
1.8,
1.45
],
"label_certainty": 1,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.6898415239772513,
-0.004692602780714069,
-0.033023180724797714,
-0.72319162108718
],
"score": -1.0
},
{
"center3d": [
9.994624230481369,
-6.017542527527623,
-0.1464149366341636
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.77,
1.8,
1.3121440590076265
],
"label_certainty": 1,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7130302072899429,
-0.03295292607233075,
0.005173472708666147,
0.7003393915348938
],
"score": -1.0
},
{
"center3d": [
13.167237991660787,
-6.162813630332312,
-0.04708915908300637
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.9561293466010117,
1.8,
1.3121440590076265
],
"label_certainty": 1,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.6950121632719635,
-0.004933161718357779,
-0.03298636965611393,
-0.7182239596646692
],
"score": -1.0
},
{
"center3d": [
18.42,
-6.81,
0.14
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.96,
1.8,
1.31
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7151992798301935,
-0.03296572963883116,
0.005076129151335868,
0.6981242609396919
],
"score": -1.0
},
{
"center3d": [
20.97984695193026,
-6.536436330155294,
0.23764169100863264
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.96,
1.8,
1.37
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7151992798301935,
-0.032965729638831244,
0.005076129151335691,
0.6981242609396919
],
"score": -1.0
},
{
"center3d": [
25.99999984503129,
-6.413659054009911,
0.3289432028830479
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.96,
1.8,
1.37
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
-0.7068649813345358,
0.03289508407018742,
-0.005475805274628736,
-0.7065619768736047
],
"score": -1.0
},
{
"center3d": [
28.57,
-6.72,
0.47
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.96,
1.8,
1.41
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
-0.7071073056108934,
0.005505677989137722,
0.03288670936708505,
0.7063196232645582
],
"score": -1.0
},
{
"center3d": [
17.9417990603285,
4.047919592487037,
-0.4298381193087091
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.66,
1.8,
1.4034827588118874
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7061942370819255,
-0.005447702826512753,
-0.03290636115848807,
-0.7072320647718813
],
"score": -1.0
},
{
"center3d": [
20.45,
4.51,
-0.31
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.18,
2.0,
1.5
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7072691145493584,
-0.032907514186323296,
0.005445972583162032,
0.706157090523962
],
"score": -1.0
},
{
"center3d": [
24.182930691745177,
3.957377988250483,
-0.21057144426054156
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.9935178142256045,
1.8765534659735825,
1.3706025545077207
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7072691145493584,
-0.032907514186323296,
0.005445972583162032,
0.706157090523962
],
"score": -1.0
},
{
"center3d": [
26.56403718760992,
3.4141810443770053,
-0.07728835626150013
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.99,
1.88,
1.37
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7072691145493584,
-0.032907514186323296,
0.005445972583162032,
0.706157090523962
],
"score": -1.0
},
{
"center3d": [
28.9087287526382,
2.7749496885068696,
0.061472400426277635
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.99,
1.88,
1.37
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7072691145493584,
-0.032907514186323296,
0.005445972583162032,
0.706157090523962
],
"score": -1.0
},
{
"center3d": [
31.80376110462618,
2.413184128432301,
0.20248894322203623
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.025175213620685,
1.9424920472975067,
1.37
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
0.7072691145493584,
-0.032907514186323296,
0.005445972583162032,
0.706157090523962
],
"score": -1.0
},
{
"center3d": [
69.01,
0.68,
1.73
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
4.39,
1.83,
1.62
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
-0.7174576529992662,
0.03299615793779559,
-0.0049519706657062896,
-0.695802592479113
],
"score": -1.0
},
{
"center3d": [
42.98202899707613,
-7.929520596676596,
0.993298589588324
],
"classname": "Car",
"created_by": "g.kuschk",
"dimension3d": [
3.7507295994605845,
1.8,
1.4635864010778035
],
"label_certainty": 2,
"measured_by": {
"Camera": 1,
"Lidar": 1,
"Radar": 1
},
"object_id": -1,
"occlusion": 1,
"orientation_quat": [
-0.7071073056108926,
0.005505677989137767,
0.032886709367085126,
0.706319623264559
],
"score": -1.0
}
]
}You can use it through: https://github.com/wzan0001/Astyx-radar-dataset-convert-to-kitti-format
Convert the data set to kitti format for use, this method realizes the conversion for radar .
Supplement: It is worth noting that the author uses this data set to use the AVOD network to fuse the camera and radar as follows:
Deep learning based 3D object detection for automotive radar and camera.
2. VoD Dataset 2022
原文：Multi-Class Road User Detection With 3+1D Radar in the View-of-Delft Dataset
Current access method: https://docs.google.com/forms/d/e/1FAIpQLSdKvkuKbzmJTn8raJBAWgekAJCpaQLS_ED63sUS89Ezo61RCQ/viewform
Fill out the application form, which needs to be a student email address for scientific research purposes. Authors who meet the requirements will send the dataset link to your email for download. It can be said that it is a relatively good dataset currently available. The introduction is on the official website, which is very detailed: The View of Delft dataset | Documentation and development kit
Explained that the next-generation radar (4D) will have great significance in the field of autonomous driving. In the ablation research, the job also discussed the additional height information (elevation) in multi-level road user detection, as well as Doppler and radar cross section. , The benefits of time accumulation. It also compares the point cloud forms of lidar and radar.
The dataset contains 8693 frame-synchronized and calibrated 64-line lidar, binocular camera and 3+1D radar data acquired in complex urban traffic. It consists of 123,106 3D bounding box annotations of moving and static objects, including 26,587 pedestrian, 10,800 cyclist, and 26,949 car labels . The results show that object detection on 64-line lidar data is still better than 3+1D radar data, but adding elevation information and integration of consecutive radar scans helps close this gap.
For the application of Doppler information, when the Doppler channel is added to PointNet++, there is a certain improvement.
Doppler dimensionality can benefit 3D object detection in two ways:
1) Classification, since classes may have different speed patterns;
2) In direction estimation, the general velocity (moving direction) of an object as it is highly correlated with its direction.
Dataset comparison:
Three contributions of this paper:
1. The author also uses PointPillars as the baseline to test the influence of adding the radar Doppler dimension, RCS and pitch angle dimensions on the results;
2. Comparing the performance of lidar and radar in target detection, the results show that lidar is still better, but it is foreseeable that radar detection still has great potential in the future, and radar is the only one for speed dimension detection;
3. Propose VoD dataset.
Next is the description of the dataset:
It mainly includes Velodyne's 64-line lidar, IDS binocular camera (230W pixels), and ZF's 4D radar, with IMU, GPS, RTK services and wheel speed.
Provided is the same synchronization as Kitti in the form of frames: Lidar point cloud, rectified mono image, radar point cloud and transformation information. Choose the lidar timestamp as the lead (the main one), the selection principle is the nearest neighbor timestamp, and the maximum tolerance is 50 milliseconds (the error is not small). The frames are temporally ordered at 10 Hz (after synchronization), and they are organized into segments with an average length of ∼40 s. The lidar is motion compensated , and the definition of the system also refers to Kitti, including the definition of the coordinate system and the organization of the file . In addition to this synchronous version of the dataset, "raw" (raw) asynchronously recorded data is provided, including all radar scans at 13 Hz, and rectified camera images at 30 Hz obtained from the left and right cameras. This could benefit researchers looking for richer temporal data for detection, tracking, prediction or other tasks.
Labeling rules: Within 50 meters of the lidar sensor, part or all of the camera's field of view (horizontal field of view: ±32°, vertical field of view: ±22°) are marked with 6 degrees of freedom (6 degrees of freedom) 3D bounding box . There are 13 categories , and for each object two types of occlusion ("Space" and "Lighting") and an activity attribute ("Stop", "Move", "Park", "Push", "Sit") are provided. ”) occlusion level.
(9+4 categories)
Annotation is done by https://understand.ai, a subsidiary of DSpace .
Use PointPillars as the Baseline. For the encoding of points, take p = [ x, y, z, v rel , RCS ] , where the speed is the compensated absolute speed. Time to merge point clouds, using lidar as the standard, merge (merge) to the last frame. It is also proposed that conventional data enhancement may not be applicable. For example, RCS and velocity dimension have a certain relationship with spatial position and cannot be arbitrarily enhanced.
For the performance test of this data set, the author only selected three categories: car, pedestrian, and cyclist. Split the dataset into training, validation, and test sets at a ratio of 59%/15%/26% , so that frames from the same clip will only appear in one clip (reasonably divided according to different clips).
The evaluation index uses the same two APs and AOS as kitti (one evaluation classification and one evaluation angle). The car's large target IoU uses 50%, while pedestrians and cyclists use 0.25 IoU. Results are derived from 1) the entire annotated area (camera field of view 50m) and 2) a more safety-related area called the "driving corridor", defined as a rectangle of ground front [−4m<x<+4m, z<25m] camera coordinates .
It can be seen that when the dimensions are full, the best results are generally obtained, and the results are better when 5 scans are accumulated. The author only did 3 and 5 scans and can do more sets. baseline is the PP of lidar, it can be seen that the effect of lidar is better. The overall AP drops with the distance.
For a cyclist, all RCS properties work best, the circular motion of the wheels and pedals, combined with the highly reflective metal frame near the center, produces a clear and distinctive reflection pattern that the radar can more reliably detected.
Examples of false detections: (a) merge small objects (two pedestrians as one cyclist), (b) large objects split into small ones (one cyclist is two pedestrians), (c) strong reflections and nearby clutter (metal extreme and high limits) and (d) too little reflection from distant objects (distant pedestrians).
data set:
3. RADIal data set
原文：Raw High-Definition Radar for Multi-Task Learning
Dataset acquisition method: GitHub - valeoai/RADIal
The download method is Google Drive: (Reminder: the large data set contains 91 clips with a duration of 1 to 4 minutes, a total of 2 hours.) https://drive.google.com/drive/folders/1vCis5mvO5qg7QlM69MqJ3Ni-gFunm5uz?usp = sharing
This data set was made by Valeo (they also did the WoodScape data set).
A model FFT-RadNet is proposed, along with a two-hour-long original dataset (raw, camera, laser, HD radar) RADIal for "Radar, LiDAR et al .
It can be seen that the feature of this data set is that the original data is retained, and the results of different processing methods are available. .RADIal is the only dataset that provides high-definition radar, combining cameras, lidar and odometry, and proposes both detection and free-space segmentation tasks.
Data set classification:
Device Information:
4. TJ4DRadSet (2022) is currently not open source
Original: TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving
The article strongly related to this article is (recommended to read together) : 3D Object Detection for Multi-frame 4D Automotive Millimeter-wave Radar Point Cloud
( Update --- has been released, need to sign a non-disclosure agreement ) Get the link: GitHub - TJRadarLab/TJ4DRadSet: TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving
The work of the article is mainly to use Oculii's 4D radar to create a data set. From other articles, it can be seen that the research group has published several extended articles based on this data set.
TJ4DRadSet contains a total of 40K frames of synchronized data, of which 7757 frames, 44 sequences have high-quality annotated 3D bounding boxes and tracking ids. The 3D labeling system uses joint multi-sensor labeling and multiple rounds of manual inspection labeling.
The platforms are as follows: LiDAR ( Velodyne-32C LiDAR can be identified through parameter comparison and pictures ), 4D radar (Oculii Eagle) and a computing platform, which uses inertial navigation and is equipped with RTK.
Then use PointPillars as Baseline to test it, the result:
Supplement: RadarScenes 2021
原文：RadarScenes: A Real-World Radar Point CloudData Set for Automotive Application
Get the link: RadarScenes - RadarScenes
As the name suggests, it consists of multiple scenes. It consists of 11 object categories, 5 large main categories and a total of more than 7000 road participants, manually labeled on 100 km of different street scenes. Roughly, there are five categories: cars, large vehicles, two-wheelers, pedestrian groups and pedestrians . The different scene columns combine factors such as weather conditions, traffic density or road class (highway, suburban, city center), and the column's sequence data indicates whether a temporally coherent sequence is available.
Sensor configuration: 4 radars at 77GHz:
Labeled 11 categories car, large vehicle , truck , bus, train , bicycle , motorized two-wheeler , pedestrian, pedestrian group, animal, and other
Distribution of the dataset:
Summarize:
Generally speaking, data sets are still relatively lacking, especially high-quality data sets. The above four data sets have made their contributions from different perspectives. When using data sets, analyze the scientific nature of data collection and the underlying principles The scientific nature is also very important. If there is a corresponding update in the future, it will be updated to the article.