Exchange group | Enter "sensor group/skateboard chassis group", please add WeChat ID: xsh041388
Exchange group | Enter the "Automotive Basic Software Group", please add WeChat account: Faye_chloe
Remarks: group name + real name, company, position
The author has been curious about many knowledge points in autonomous driving simulation (the number one is "simulation with real road data") for more than two years, but I have never had the opportunity to learn it before. During the epidemic in April last year, I occasionally got a chance to chat with the founder of a simulation company, and the author took the opportunity to ask him a lot of questions.
Since then, for cross-validation, the author has successively consulted nearly 20 experts in the front line of autonomous driving simulation business.
Experts who provide support for this series of study notes include but are not limited to An Hongwei, CEO of Zhixing Zhongwei, Yang Zijiang, founder of Shenxin Kechuang, Li Yue, CTO of Zhixing Zhongwei, Bao Shiqiang, CTO of 51 World, and simulations of Momo Zhixing, Qingzhou Zhihang, and Cheyou Intelligent experts etc. Thanks for this.
Question 1: Scene source - from synthetic data to real road data
According to Li Manman, the author of the public account "Che Lu Slowly", and Li Yue, CTO of Zhixing Zhongwei, there are generally two ways of thinking about the source of the simulation test scene:
The first idea is a three-layer system of functional scenarios-logic scenarios-specific scenarios proposed by the German PEGASUS project: 1) Obtain different types of scenarios (that is, functional scenarios) through real road data collection and theoretical analysis; 2) , and then analyze the key parameters in these different scene types, and obtain the distribution range of these key parameters (that is, logical scenes) through methods such as real data statistics and theoretical analysis; 3), and finally select the value of a group of parameters as a Test scenarios (i.e. concrete scenarios).
As shown below:
For example, the functional scenario can be described as, "the self-vehicle (vehicle under test) is running in the current lane, there is a vehicle in front of the self-vehicle accelerating, and the self-vehicle follows the preceding vehicle." The logical scene extracts the key scene parameters, And give the scene parameters a specific value range. For example, in the scene described above, parameters such as the speed of the vehicle in front, the speed and acceleration of the vehicle in front, and the distance between the vehicle in front and the vehicle in front can be extracted. Each parameter has a certain value range and distribution characteristics. There may also be dependencies between parameters. For a specific scene, specific scene parameter values need to be selected to form a scene parameter vector and expressed in a specific scene language.
This is actually the so-called "virtual construction/algorithm-generated scene". Although the understanding of the scene still comes from the real road scene, in practice, it is more based on this understanding to "artificially draw up" a scene in the software. Driving trajectory, a set of scenes, therefore, the data behind this scene is also called "synthetic data".
In practice, the main challenge with this approach is whether the simulation engineer has a deep enough understanding of the normal driving scenarios of the vehicle. If the engineer does not understand the scene and willfully "draws up" a scene, of course it cannot be used.
The second way of thinking is: collect traffic flow data in the predetermined working area of the autonomous driving vehicle, and input these data into the traffic simulation tool to generate traffic flow, and use this traffic flow as the "surrounding traffic vehicle" of the autonomous driving vehicle to realize the test Automatic generation of scenes.
According to Yang Zijiang, the founder of Shenxin Kechuang, in order to ensure a more accurate "true value", usually, the sensor configuration on the engineering collection vehicle is much higher than that of ordinary self-driving cars. For example, the positioning system will use more than 20W equipment and High-beam lidar will produce more accurate data.
Waabi, a simulation company founded by Raquel Urtasun, the former chief scientist of UberACT, is said to use the data collected by the camera for simulation without the need for high-precision sensors such as lidar.
The biggest advantage of using real road data for simulation is that the diversity of scenarios will not be limited by the engineers' lack of understanding of the scenarios. Therefore, it is easier to "salvage" those unknown scenarios that "no one can think of".
In addition, the person in charge of the simulation of an autonomous driving company said: In order to improve the realism of the simulation, we will use as little synthetic data as possible and use more real road data. In fact, the current simulation is already developing in this direction-there are more and more real data and modules.
However, engineers with front-line simulation practice generally reflect that this idea is too idealistic. Specifically, using real road data for simulation has the following limitations:
1. The data needs to be checked manually
In fact, the data collected by the sensor cannot be directly used for simulation - the data type and format need to be converted, there is a lot of invalid data that needs to be cleaned, and valid scenes must be identified from it, and some specific elements need to be marked. Different sensors The data between needs to be synchronized and fused in real time, etc.
Under normal circumstances, the perception data of autonomous driving vehicles does not need to be manually checked, but is directly given to the decision-making algorithm. However, if it is a simulation, manual checking of the perception data is an essential step.
2. The reverse process is more difficult to realize than the forward process
A simulation engineer of an unmanned truck company said: Simulation with synthetic data is a positive process, that is, you first know what tests you need to do, and then take the initiative to design such a scene; simulation with real data is a positive process. A reverse process, that is, you first encounter a problem, and then solve it. Comparing the two, the latter is much more difficult.
3. Unable to solve the interaction problem
Jame Zhang, head of Furui Microelectronics, mentioned in a public sharing that WorldSim (using virtual data for simulation) is like playing a game, while LogoSim (using real road data for simulation) is more like a movie, you can only watch, Unable to participate, therefore, LogoSim naturally cannot solve the problem of interactivity.
4. Unable to do closed loop
Jame Zhang, head of Furui Microelectronics, also mentioned another difference between the two simulation methods: using real road data for playback, the fragments that can be collected are always limited, and often, when the collection starts, danger may have already occurred It's been a while, and it's hard to get the previous data, but if you use virtual data (synthetic data), you don't have to face this problem.
The person in charge of the simulation of an OEM said: "The above-mentioned experts described the acquisition process. Indeed, considering the capacity of the acquisition equipment and the definition of effective scenes, the scenes of acquisition and management have lengths, generally before and after function triggering. Time, especially the cache before the trigger will not be particularly long. On the other hand, when the data is collected and used for refilling, only the scene before the function trigger is valid, but the real scene after the function trigger is invalid of."
The OEM expert said: It is possible to use real road data to train the perception algorithm, but to test the entire algorithm link, it still has to rely on synthetic scene data.
However, the simulation director of the main engine factory also emphasized at the end: "The so-called 'closed loop cannot be achieved' is also relative. There are already suppliers who can complete the parameterization of the elements in the collected scenes, so that the closed loop can be achieved. But the price of such equipment is very expensive."
5. The authenticity of data is still difficult to guarantee
Simulation with real traffic flow data, also known as "recharging".
According to Yang Zijiang, the founder of Shenxin Kechuang, there are two core technologies that need to be used for "recharging": one is to restore the road network structure of the road mining data in the simulation environment, and the other is to integrate the dynamic traffic participants in the road mining data ( Pedestrians, vehicles, etc.) The pose information in different coordinate systems is mapped to the global coordinate system under the simulated world road network.
The tools that need to be used in this process are SUMO or openScenario-used to read in the location information of traffic participants.
A simulation expert of an OEM said: "The refilling of original data cannot guarantee 100% authenticity, because after the original data is injected into the simulation platform, vehicle dynamics simulation must be added. But in this way, whether the scene is still the same as that on the real road The scenes are the same, so it’s hard to say.”
The reason is that the existing traffic flow simulation software often still has the following major defects:
The generated traffic flow is not fidelity enough, often only supports the import of vehicle trajectories, and the two-way interaction between vehicles is not realistic enough;
The data transmission interface between the simulation module (self-vehicle) and the traffic flow module (other road participants) is limited (for example, the road network format is different, and road network matching is required), and third-party operability is limited;
The rule-based traffic flow model is oriented to the evaluation of traffic efficiency, and there may be problems of oversimplification (one-dimensional models are often used, assuming that the establishment is driven along the center line, and the lateral impact is less considered), and it is difficult to meet the requirements of interactive safety evaluation. need.
A Tier 1 simulation engineer said that it is quite difficult to use real traffic flow data to generate simulation scenarios, how to choose a traffic flow model (such as how to define the car-following model and lane-changing model), and how to define the interface of the traffic flow simulation module. At the same time, how to synchronize the data from the own vehicle with the data of other road users will also be a big problem.
6. The universality of the data is low and the generalization is difficult
Both An Hongwei, CEO of Zhixing Zhongwei, and Li Yue, CTO, specifically mentioned the "universality" of simulation data. The so-called data versatility means that the parameters of the vehicle and the scene can be adjusted. For example, when the data is collected by a car, the angle of view of the camera is very low, but after it becomes a simulation scene, the angle of view of the camera can be adjusted higher, and this set of data can be used for the test of the truck model.
If the scene is virtual construction/algorithm generated, each parameter can be adjusted arbitrarily according to needs; then, what if the scene is based on real road data?
The person in charge of the simulation of a tool chain company said that in the case of using real road data for simulation, once the position or model of the sensor is changed, the value of this set of data will be reduced, or even "obsolete".
The simulation experts of Qingzhou Zhihang said that the neural network can also be used to adjust the parameters of real road data. This kind of parameter adjustment will be more intelligent, but the controllability will be weaker.
Using real traffic flow data for simulation, also known as "recharging", and recharging can be divided into two types, direct recharging and model recharging——
The so-called "direct recharge" refers to directly feeding the sensor data to the algorithm without processing. In this mode, the parameters of the vehicle and the scene cannot be adjusted. The data collected by a certain model can only be used for the same vehicle. The simulation test of the model;
"Model refilling" refers to abstracting and modeling the scene data first, and expressing it with a set of mathematical formulas. In this mathematical formula, the parameters of the vehicle and the scene are adjustable.
According to Li Yue, direct recharge does not require the use of mathematical models. "It is relatively simple. Basically, as long as there is big data capability, it can be done." The trajectory and speed of the vehicle are all done through mathematical formulas.
The technical threshold of model recharge is very high, and the cost is not low. A simulation engineer said: "It is very difficult to convert the data recorded by the sensor into simulation data. Therefore, at present, this technology mainly stays in PR level. In practice, each company’s simulation tests are based on scenarios generated by algorithms, supplemented by scenarios from real road sets.”
The person in charge of the simulation of an autonomous driving company said: It is still very cutting-edge technology to use real traffic flow data for simulation. It is very difficult to adjust the parameters of these data (the parameters can only be adjusted within a small range). Because road mining is a bunch of logs and records one by one, it records how the car operates in the first second and second, unlike some scenes edited by humans, which are composed of a series of formulas.
The simulation expert said that the biggest challenge of model refilling is: in the case of complex scenarios, it is extremely difficult to formulate the scenarios. This process can be realized in an automated way, but in the end it is Whether the scene can be used is also a question.
Waymo announced in 2020 that "by directly generating realistic image information from the data collected by the sensor for simulation", ChauffeurNet is actually using a neural network in the cloud to convert the original road data into a mathematical model, and then refill the model. But a simulation expert who has been in Silicon Valley for many years said that this is still in the experimental stage, and there is still some time before it becomes a real product.
More meaningful than refeeding, the simulation expert said, is the introduction of machine learning, or reinforcement learning. Specifically, the simulation system trains some of its own logic on the basis of fully learning the behavior habits of various traffic participants, formulates these logics, and then adjusts parameters in these formulas.
However, according to Li Yue, CTO of Zhixing Zhongwei, and Feng Zonglei, deputy general manager, they have already been able to realize model recharging.
Feng Zonglei believes that whether a simulation company has the ability to refill the model mainly depends on the tools they use and their scene management capabilities.
"In scene management, slicing is a very important part-not all data is valid. For example, in 1 hour of data, the real effective data may be less than 5 minutes. When doing scene management, the simulation company The effective part needs to be cut out, and this process is called 'slicing'.
"After the slicing is completed, the simulation company needs to create a corresponding management environment with semantic information (such as which is a pedestrian and which is an intersection) to facilitate the next screening. Specifically, it is necessary to classify the data slices first, and then Then refine the dynamic target list, and then import it into the model of the simulation environment. In this way, the model has corresponding semantic information. With the semantic information, you can adjust the parameters, and then, the data It can be reused.
"The reason why most companies cannot adjust parameters based on real traffic flow data is because they have not done a good job in scene management."
Yang Zijiang, the founder of Shenxin Kechuang, said: "If you want to generalize the road mining data and maintain the authenticity of the data, you can play back the road mining data at the scene initialization and the beginning stage, and at a certain point the smart-npc model will take over the road The background vehicles in the system will prevent the background vehicles from running according to the road sampling data. After the smart-npc takes over, it records the generalized scenes so that the generalized key scenes can be played back.”
A simulation engineer of an OEM believes that although model recharge sounds "unclear", it is actually "not necessary". The reason is: Modeling the data does not match the original intention of refilling—the original intention of refilling is to want real data, but since the model is modeled and the parameters are adjustable, it is not the most real; time-consuming and laborious, data format conversion Very troublesome and thankless.
The engineer said: "Since you want more scenarios, you can directly use the simulator to generate generalized scenarios on a large scale. You don't have to take the path of modeling real data."
In response, Feng Zonglei responded:
"Using algorithms to directly generate scenes is of course no problem in the early stages of development, but the limitations are also obvious - what about those scenes that the engineers 'unexpected'? Real traffic conditions are ever-changing, and your imagination cannot be limited Give it all.
"More importantly, in the scene imagined by the engineer, the interaction relationship between the objects is often unnatural. For example, if there is a vehicle in front of you, what angle does it insert at? When it is 10 meters away from you, it is still 5 meters away. Time insertion? In the practice of using algorithms to generate scenes, the formulation of scene parameters is often very subjective and arbitrary. Engineers took their brains and came up with a set of parameter injection models, but is this set of parameters representative? "
Feng Zonglei believes that when unmanned driving is still in the Demo stage, virtual scenes generated by algorithms can meet the needs, but in the era of pre-installed mass production, scene generalization is based on large-scale natural driving data (real traffic flow data). Still very necessary.
According to a person who has had contact with Momenta: "Momenta already has the ability to use real road data for scene generalization (parameter adjustment), but their technology is only for their own use and not for the outside world."
Bao Shiqiang, head of vehicle simulation at 51 World, believes that the generalization of natural driving data is still relatively forward-looking, but it will definitely become a very important direction in the future, so they are also exploring.
Summary: The two routes penetrate each other, and the boundaries become increasingly blurred
James Zhang, the person in charge of Furui Micro-simulation, mentioned in a sharing some time ago that there are two methods of Tesla’s simulation: the scene is completely virtual (generated by algorithm) called WorldSim, and the real data playback is called LogSim for the algorithm to see. "However, the road network in WorldSim is also generated on the basis of automatic standardization of data from real roads. Therefore, the boundaries between WorldSim and LogSim are becoming increasingly blurred."
The simulation expert of Qingzhou Zhihang said: "After the real scene data is converted into standard formatted data, it can be de-generalized through rules, thereby generating more valuable simulation scenarios."
Bao Shiqiang, head of 51 World's in-vehicle simulation business, also believes that the future trend is that the two routes of simulation using real road data and simulation using algorithm-generated data will interpenetrate.
Bao Shiqiang said: "On the one hand, using algorithms to generate scenes also depends on the engineer's understanding of real road scenes. The more thorough the understanding of real scenes, the closer the modeling can be to reality. On the other hand, using real road data as scenes, It is also necessary to slice and extract the data (screen out the effective part), then set parameters, trigger rules, and then perform refined classification, and then they can be logicalized and formulated.”
Question 2: Scene Generalization and Scene Extraction
The "parameter adjustment" of scene data mentioned repeatedly in the above paragraphs is also called "scene generalization"-usually mainly refers to the generalization of virtual scenes. In the words of a system engineer of an OEM, the advantage of scene generalization is that we can "create" some scenes that have never been seen in the real world.
The stronger the scene generalization ability of a simulation company, the more available scenes can be obtained after adjusting the parameters of a certain scene. Therefore, the scene generalization ability is also a key competitiveness of a simulation company.
However, Qingzhou Zhihang's algorithm experts said that scene generalization can be achieved through mathematical models, machine learning and other methods, but the key issue is how to ensure that the generalized scene is real and more valuable.
What are the key factors that determine whether a company's scene generalization ability is strong or weak?
Yang Zijiang, the founder of Shenxin Kechuang, believes that a big difficulty in scene generalization is how to abstract the trajectory into higher-level semantics and express it in a formal description language.
A Tier 1 simulation engineer said: It mainly depends on what language (such as openscenario) is used by the simulation tool used by the company to describe different traffic scenarios. details, while being scalable).
There are corresponding scenario languages for functional scenarios, logical scenarios, and specific scenarios: For the former two, there are advanced scenario languages such as M-SDL; for the latter, there are OpenSCENARIO, GeoScenario, etc.
Another level may be the simulation of interference behaviors, the degree of generalization of various driving behaviors and driving "personality".
△The chart is taken from the book "Autonomous Driving Virtual Simulation Test Evaluation Theory and Method" by Sun Jian, Tian Ye and Yu Rongjie
Yang Zijiang, the founder of Shenxin Technology, said: "Based on the generalization of traffic flow and the intelligence of drivers, if the model is good enough, due to the existence of random factors, running the scene 10 times is equivalent to generalizing 10 times."
However, Li Yue, CTO of Zhixing Zhongwei, believes that generalization cannot be done for the sake of generalization. "We must have a deep understanding of the function under test, and then design a generalization plan, not generalization for the sake of generalization, let alone generalization without bounds. Although scene generalization is virtual, we must also respect Reality."
Another simulation expert also said: "At the end of the day, simulation should serve testing. We have already encountered a problem on the road, and then we will see how to solve it through simulation, instead of saying that I have a simulation technology first, and then Let's see what it's used for?"
A simulation expert mentioned above said that as far as he knows, there are not many companies that can truly achieve the generalization of scenarios. In most cases, parameter adjustment is done manually. "Scene generalization ability is very important, but at this stage, no company can really do it well."
Bao Shiqiang, head of the vehicle simulation business at 51 World, believes that the most important thing for scene generalization is to have a deep understanding of what kind of scenarios are needed for autonomous driving simulation tests. In fact, the problem now is not that there are too few generated scenarios, but too many, and many of them will not actually happen, so they are not considered effective test scenarios. This is caused by a lack of understanding of requirements.
According to some experts, the biggest challenge faced by third-party simulation companies is that they have insufficient understanding of what kind of simulation is required for autonomous driving because they have not personally participated in autonomous driving.
And those L4 autonomous driving companies that are capable and have a deep understanding of simulation requirements do not have enough motivation to generalize the scene very deeply. Because Robotaxi usually only runs in a small area of a certain city, they only need to collect the scene data of this area for training and testing, there is not much need to generalize a lot of them for a long time Scenes that no one will touch.
Bao Shiqiang believes that OEMs like Wei Xiaoli have a lot of real road data, and there is no strong demand for scene generalization. On the contrary, for these companies, what is more urgent than scene generalization is to fine-tune the classification and management of scenes and screen out the truly effective scenes.
The simulation experts of Qingzhou Zhihang also believe that with the increase of fleet size and the rapid expansion of data from real roads, for simulation companies, how to fully mine effective scenarios in these data is indeed much more important than scene generalization . "We may explore a more intelligent generalization method, which can perform large-scale verification of the algorithm faster."
Yang Zijiang said: "Aiming at the generalization at the parameter level, such as the number of lanes, the number of types of traffic participants, weather, and key parameters such as speed and TTC, each company's ability to generate generalized scenarios is similar, but the core of the generalization ability of the scene is It lies in how to identify valid scenes and filter invalid scenes (including repeated and unreasonable ones); and the difficulty of scene recognition is that complex scenes need to identify the relationship between multiple objects."
The above-mentioned "identifying valid scenes and filtering invalid scenes" is also called "scene extraction".
The premise of scene extraction is to first figure out what is a "valid scene". According to several simulation experts, in addition to the scenarios that should be tested according to the law, the effective scenarios also include the following two types: when doing the forward design of the system, the scenarios defined by the engineers according to the development requirements of the algorithm; Can't get it right" scenario.
Of course, effectiveness and inefficiency are relative, which is related to the development stage of the company and the maturity stage of the algorithm—in principle, as the algorithm matures and the problem is solved, many original effective scenarios will become invalid scenarios.
So how do you efficiently screen out effective scenarios?
There is an idea in the academic community: set some entropy values in the perception algorithm, and when the complexity of the scene exceeds these values, the perception algorithm will mark the changed scene as a valid scene. But how to set this entropy value is a big challenge.
A simulation company adopts the "elimination method", that is, if an algorithm that originally performed very well has "problems frequently" in some generalization scenarios, then this scenario has a high probability of being an "invalid scenario" and can be ruled out.
A system engineer from an OEM said: "At present, there is no good method for scene screening. If you are not sure, then put it on the cloud simulation to calculate. After all, you can calculate these extreme scenarios, and then use these extreme conditions in your own If the verification is done on the HIL bench or VIL bench, the efficiency will be much higher.”
Question 3: Where is the difficulty in simulation?
In the process of communicating with experts from many simulation companies and their downstream users, we learned that one of the most difficult aspects of autonomous driving simulation is sensor modeling.
According to Li Yue, CTO of Zhixing Zhongwei, sensor modeling can be divided into functional information level modeling, phenomenon information level/statistical information level modeling and full physical level modeling. The difference between these concepts is as follows -
Functional information-level modeling simply describes the specific functions of the camera output image and millimeter-wave radar detecting targets within a certain range. The main purpose is to test and verify the perception algorithm, but it does not pay attention to the performance of the sensor itself;
Phenomena information and statistical information level modeling is a hybrid, intermediate level modeling that includes part functional information level modeling and part physical level modeling;
Full-physics-level modeling refers to the simulation of the entire physical link of the sensor's work. The goal is to test the physical performance of the sensor itself, such as the filtering ability of the millimeter-wave radar.
Sensor modeling in a narrow sense refers to modeling at the full physical level. This kind of modeling, few companies can do well, the specific reasons are as follows:
1. The efficiency of image rendering is not high enough
From the perspective of computer graphics imaging principles, sensor simulation includes light (input and output simulation), geometry, material simulation, image rendering and other simulations, and the difference in rendering capabilities and efficiency will affect the authenticity of the simulation.
2. Too many types of sensors & the "impossible triangle" of model accuracy, efficiency and versatility
It is not enough to have a single sensor with high accuracy, you also need all the sensors to reach an ideal state at the same time, which requires a wide coverage of modeling, but under the pressure of cost, it is obviously impossible for the simulation team Radar does 10 or 20 versions of modeling, right? On the other hand, it is difficult to use a general model to express various sensors of different styles.
The accuracy, efficiency, and versatility of the model are an "impossible triangle" relationship. You can improve one side or two corners, but it is difficult for you to continuously improve the three dimensions at the same time. When the efficiency is high enough, the model accuracy must decrease.
The simulation expert of Cheyou Intelligence said: "No matter how complicated the mathematical model is, it may only simulate the real sensor with 99% similarity, and the remaining 1% may be the factor that will cause fatal problems."
3. Sensor modeling is subject to the parameters of the target
Sensor simulation requires external data, that is, the external environment data is strongly coupled with the sensor. However, the modeling of the external environment is actually quite complicated and the cost is not low.
There are too many buildings in urban scenes, which will seriously consume computing resources for image rendering. Some buildings will block the traffic flow, pedestrians and other target objects on the road, but if they are blocked or not, the amount of calculation is completely different.
In addition, the reflectivity and material of the target are difficult to figure out through sensor modeling. For example, it can be said that a target is in the shape of a barrel, but it is difficult to express clearly through modeling whether it is an iron barrel or a plastic barrel; even if it can be expressed clearly, it is another problem to adjust these parameters in the simulation model. Super big project.
If the physical information such as the material of the target object is not clear, it is difficult to choose a simulator for simulation.
4. It is difficult to determine how much noise is added to the sensor
A Tier 1 simulation engineer said: "The recognition of objects by deep learning algorithms is a process from the collection of real-world sensor data to signal denoising. In contrast, sensor modeling is based on an ideal physical model. Add noise, and the difficulty lies in how to add noise to be close enough to the real world, so that it can be recognized by the deep learning model and effectively improve the generalization of model recognition.”
The implication is that the sensor signal generated by the simulation must be "sufficiently similar" to the sensor signal in the real world (can identify the corresponding object), but not "too similar" (simulating the corner case allows the perception model to achieve recognition in more situations— — generalization). However, the problem is that in the real world, sensor noise is random in many cases, which means that how to simulate these noises in the simulation system is a big challenge.
From the perspective of the sensor principle, the process of camera modeling also needs to do camera blurring (first generate an ideal model, and then add noise), distortion simulation, vignetting simulation, color conversion, fisheye effect processing, etc. The model can also be divided into an ideal point cloud model (the steps include scene clipping, visibility judgment, occlusion judgment and position calculation), power attenuation model (including acceptance laser power, reflected laser power, reflection antenna gain, target scattering cross section, interface aperture, Target distance, atmospheric transmission coefficient, optical transmission coefficient, etc.) and physical models considering weather noise, etc.
5. Resource constraints
An Hongwei, CEO of Zhixing Zhongwei, mentioned the limitation of resources on perception virtual simulation: "We need to do a complete physical level modeling of the sensor, such as the optical and physical parameters of the camera, etc., and we also need to know the target (sensing object) Materials, reflectivity and other data, the amount of this project is huge - with enough manpower, the construction period of a one-kilometer scene takes about one month. Even if it can be built, the complexity of the model is extremely high, and it is difficult to It runs on a physical machine (too much computing power)."
"In the future, all simulations will go to the cloud. It seems that the computing power of the cloud is 'infinite', but when it is allocated to a single model of a single node, the computing power of the cloud may not be as good as that of a physical machine—and, in When doing simulation on a physical machine, if the computing resources of one machine are not enough, three machines can be installed, one is responsible for the sensor model, one is responsible for dynamics, and one is responsible for regulation and control, but running simulation on the cloud can be used in a single scene The computing power on a single model is not endless, so this limits the complexity of our model."
6. It is difficult for simulation companies to obtain the underlying data of sensors
Full physical level modeling needs to construct various performances of sensors with mathematical models. For example, a specific performance of the signal receiver, the propagation path (influenced by air in the middle, the entire link of reflection and refraction) is expressed in mathematical formulas. However, at the stage when software and hardware have not been truly decoupled, the perception algorithm inside the sensor is a black box, and the simulation company cannot understand what the algorithm looks like.
Full physical modeling needs to obtain the underlying parameters of sensor components (such as CMOS chips, ISPs), and model these parameters. Moreover, it is also necessary to know the underlying physical principles of sensors, and to analyze the laser waves of lidar and millimeter wave radars. Modeling of electromagnetic waves.
In this regard, a simulation expert said: "To do a good job in sensor modeling, you must have a deep understanding of the underlying hardware knowledge of the sensor, which is basically equivalent to knowing how to design a sensor."
However, sensor vendors are generally reluctant to open up the underlying data.
Li Yue, CTO of Zhixing Zhongwei, said: "If you get these underlying parameters and use them to do modeling, then you can basically make this sensor."
An Hongwei, CEO of Zhixing Zhongwei, said: "Usually when OEMs deal with sensor suppliers, it is not easy to get the interface protocol, not to mention the details of material physical parameters. If the OEM is strong enough, the sensor Suppliers are also actively cooperating, and they can get interface protocols, but not all of them. It is even more difficult for OEMs to obtain things that are difficult for simulation companies.”
In fact, the physical level simulation of sensors can only be done by sensor manufacturers themselves. Many domestic sensor manufacturers use external chips and other components for integration. Therefore, it is actually upstream suppliers such as TI and NXP that can simulate the physical level of sensors.
A simulation engineer of a commercial vehicle unmanned driving company said: "Simulation of sensors is difficult, which makes the process of sensor selection very complicated. When we want to select sensors, basically the sensor company sends me the samples first, and we Then install various types on the car for testing. If sensor manufacturers can cooperate with simulation companies, they can connect all the interfaces and provide accurate sensor modeling. Knowing the sensor information will greatly reduce the workload of sensor selection.”
However, Bao Shiqiang, CTO of 51 World, said: "Perceptual simulation is still in its infancy, and it is far from reaching the stage where the modeling inside the sensor needs to be done so finely. I disassemble the inside of the sensor and model those things. It feels pointless."
In addition, according to the person in charge of the simulation of an unmanned driving company, the inability to do sensor simulation does not mean that the simulation of perception cannot be done at all.
For example, hardware-in-the-loop (HIL) can be connected to real sensors (sensors and domain controllers, both are real) for testing. Connecting to the real sensor can not only test the perception algorithm, but also test the function and performance of the sensor itself. In this mode, the sensor is real, and the simulation accuracy is higher than sensor simulation.
However, because it involves supporting hardware, it is complicated to integrate, and this method still requires a sensor model to control the generation of environmental signals, and the cost is higher. Therefore, this method is rarely used in practice.
Attachment: Two stages of autonomous driving simulation test
(Excerpt from the article "Introduction to Virtual Simulation Test of Autonomous Driving" published by the official account "Car Road Slowly" on March 26, 2021)
Considering the recent actual situation, autonomous driving simulation can be roughly divided into two stages of development (of course, these two stages may not have a clear time limit).
(1) Stage 1:
The perception and identification module of the sensor is tested in the laboratory and the closed test field, and the decision-making control module is tested in the virtual simulation environment. The simulation environment directly provides the target list to the decision-making control module.
This is mainly because current modeling of sensors has many limitations that prevent efficient (or even correct) simulations. For example, the pictures output by the camera are easier to simulate, but the simulation of characteristics such as stains and strong light is more difficult; and for millimeter-wave radar, if a model with high accuracy is established, the calculation speed is slow, which cannot meet the needs of simulation testing.
Complete control and data recording of the test environment can be carried out in the laboratory and closed test field. For example, arrange pedestrians and vehicles of different categories, positions and speeds, and even simulate environmental elements such as rain, snow, fog and strong light, and compare the target list output by sensor processing with the real environment, so as to give the perception recognition Module evaluation results and recommendations for improvement.
The advantage of this is that, in the case of many limitations in sensor modeling, the decision-making control module can still be tested in a simulation environment, and enjoy the advantages of simulation testing in advance.
(2) Stage two:
Perform high-precision sensor modeling in a virtual simulation environment to test complete autonomous driving algorithms.
In this way, not only can testing be performed in the same environment, thereby improving test efficiency, test scenario coverage and complexity; but also end-to-end testing can be performed on some AI-based algorithms.
The difficulty at this stage is, on the one hand, the sensor modeling that meets the test requirements mentioned above, and on the other hand, the interfaces for direct interaction between different sensor manufacturers and OEM manufacturers may be inconsistent (in some cases, they may not exist).
Question 4: What are the differences between lower and higher levels of autonomous driving simulation testing?
For the low-level autonomous driving stage, simulation is only an auxiliary means, but when it comes to high-level autonomous driving, simulation becomes a barrier to entry-L3 needs to do enough mileage simulations to get on the road.
A simulation expert of an OEM said: Usually, autonomous driving companies are more capable of L4 simulation, while third-party simulation companies mainly focus on L2 simulation. So, what are the specific differences between the two phases of simulation?
1. Functional boundaries
Qingzhou Zhihang simulation expert: "The product definition of L2 is mature and the functional boundaries are clear. Therefore, the services provided by simulation service providers to various OEMs are highly versatile; and where the functional boundaries of L4 are, everyone is still exploring, so , customers have a high degree of customization for simulation needs.”
2. Scale of scene library
Yang Zijiang, founder of Kechuang: "From the perspective of test scenarios, because of the higher complexity of ODD in L4, the order of magnitude of the scene library is much higher than that of L2."
3. Requirements for scene reproducibility
A simulation expert of an OEM said: "L4 simulation has higher requirements for scene reproducibility, that is, whether a problem found on the road can be reproduced in the simulation environment; but many companies that do L2 simulation have not thought about this. question."
4. Attention to data mining capabilities
For low-level autonomous driving simulation, everyone mainly fights for the authenticity of the scene, while high-level autonomous driving pays more attention to data mining.
5. Data composition
Bao Shiqiang, person in charge of vehicle simulation business at 51 WORLD: "L2 has a relatively clear definition of functions, and the simulation can be mainly based on synthetic data, supplemented by real road data; and at the L4 stage, the importance of data-driven will be higher. Therefore, it is necessary to Mainly based on real road data, supplemented by data generated by algorithms."
6. Perception
High-level self-driving vehicles have a large number of cameras and high pixels, which put forward higher requirements for the image rendering ability, data synchronization ability and stability of the simulation engine of the simulation system.
7. HD map
Li Yue, CTO of Zhixing Zhongwei: "Basically, low-level autonomous driving does not require high-precision maps, but high-level autonomous driving is highly dependent on high-precision maps at the current stage, which is one of the reasons why it is necessary to build a digital twin when constructing a scene. , compared with the real world.”
8. Decision making
Li Yue, CTO of Zhixing Zhongwei: "The L2 plan pays more attention to the strategic logic of decision-making and the testing of the executive agency, but does not focus on the planning algorithm. However, in the L4 plan, how to avoid obstacles and how to bypass There are more considerations in path planning algorithms such as roads."
9. Is a driver model required?
For low-level automatic driving, the system will not fully control the behavior of the vehicle, but only play an auxiliary role. Therefore, simulation companies need to design many driver models when doing scene design; Said that vehicle control is realized through automatic driving, therefore, there is no need to design a driver model in the simulation scene design.
10. Whether to set the test process in advance
Regarding this logic, the public account "Car Road Slowly" explained in more detail in an article:
The complexity and range of working conditions faced by lower-level automatic driving are relatively small, or because the driving behavior is mainly responsible for human drivers, the automatic driving system only needs to deal with a limited number of definite working conditions; The driving behavior of high-level automatic driving is mainly in charge of the automatic driving system, and the complexity and range of working conditions it handles are very large, and it cannot even be predicted in advance.
Based on this difference between the two, lower-level autonomous driving can be better tested using use-case-based testing methods, while higher-level autonomous driving requires scenario-based testing methods.
The test method based on the use case is to preset the test input and test process, and evaluate whether the test is passed by checking whether the tested algorithm achieves the expected function. For example, in the test of ACC, the initial speed of the vehicle under test and the vehicle in front is preset, as well as the deceleration time and deceleration of the vehicle in front, and check whether the vehicle under test can follow the deceleration to stop.
The scenario-based test method is to preset the test input , but does not pre-set the test process, only sets the behavior of traffic vehicles, gives the tested algorithm a greater degree of freedom, and checks whether the tested algorithm achieves the expected goal. Evaluate whether to pass the test . For example, in the test of straight road driving, the initial speed of the vehicle under test and the vehicle in front, as well as the deceleration time and deceleration of the vehicle in front are preset, but it is not limited whether the vehicle under test avoids collision with the vehicle in front by decelerating or changing lanes. collide.
One of the reasons for using different test methods for different levels of automatic driving functions is that low-level automatic driving can generally be decomposed into simple and independent functions, and a single function can be used as the test object; while high-level automatic driving is more difficult. Difficult to decompose into simple and independent functions, so the entire automatic driving system or a relatively large part thereof has to be taken as the object under test.
11. Industrial ecology
Yang Zijiang, the founder of Kechuang: "From the perspective of industrial ecology, for L2, car companies will basically not develop their own research, but directly adopt outsourcing solutions, and the test will be based on HIL or even road tests; for the simulation of L4, Many car companies will tend to start their own research from SIL."
Question 5: How to understand the "thousands of kilometers per day" in the simulation?
Similar to the real road test, some simulation companies also emphasize "driving mileage", for example, "hundreds of thousands of kilometers per day", so what is the real meaning behind this number? How does it compare to mileage on real roads?
The virtual mileage refers to the sum of the mileage of a massive simulation platform in parallel simulation nodes per unit time. The simulation mileage per unit time depends on the number of nodes supported by the computing power of the entire platform and the super real-time index under different simulation scene complexity.
To put it simply, a simulation node is a vehicle, that is, how many "test vehicles" the simulation platform can support running in parallel at the same time.
According to An Hongwei, CEO of Zhixing Zhongwei, explained: To put it simply, if a simulation platform has the computing power of 100 GPU servers, and each deploys 8 simulation instances, then the simulation platform has the ability to parallelize 800 simulations at the same time. The simulation mileage depends on the daily mileage of each instance.
How many instances can run on a GPU server depends on the performance of the GPU and whether the simulation solver can be simulated in parallel on a server .
An Hongwei said: "The simulation nodes of our cloud simulation platform have realized a variety of deployment methods, which can flexibly meet the conditions of various cloud resources of customers, and can achieve large-scale and flexible node deployment. Currently we are building in Xiangcheng, Suzhou. Its cloud simulation platform has achieved the deployment of more than 400 nodes."
Combined with the daily mileage of each instance, the total daily simulation mileage on the simulation platform can be roughly calculated. If one instance (virtual car) runs an average of 120 kilometers per hour and runs 24 hours a day, then it is nearly 3,000 kilometers per day. If there are 33 instances, then there are almost 100,000 kilometers per day on this server.
However, according to An Hongwei, the simulation "thousands of thousands of kilometers per day" that the industry usually refers to is not very rigorous. " It needs to be supported by a reasonable simulation test plan and a large number of scenarios, and the coverage and effectiveness of the scenarios should be continuously expanded. Finally, the effective scenarios that can be run out are fundamental. "
Question 6: Super real-time simulation
During the interview, the author repeatedly asked a question: Are the cars running on the simulation platform in the same time dimension as the cars in the real world? Put another way: Is 1 hour on the simulation platform equal to 1 hour in the real world? Will there be a situation of "one year on earth, ten years in heaven"?
The answer is: it can be equal to (real-time simulation), or not equal to (super real-time simulation). Ultra-real-time simulation can be divided into two cases of "time acceleration" and "time deceleration" - time acceleration means that the time on the simulation platform is faster than the time in the real world, and time deceleration means that the time on the simulation platform is slower than the real world.
Simulation is faster than real-world time for efficiency, so why is it slower than real- world time ?
An Hongwei's explanation is: "For example, some simulation tests require very high accuracy in image rendering. In order to pursue accuracy, the rendering of a single frame image may not be completed in real time. This kind of simulation that is slower than real time, Instead of doing real-time closed-loop testing, it is doing offline testing.”
Specifically, in real-time simulation, after the image is generated, it is directly sent to the algorithm for recognition. This process may be completed within 100 milliseconds, but in offline simulation, the image is saved first after generation, and sent to the algorithm under offline conditions. deal with.
According to An Hongwei's explanation, the following two prerequisites need to be met for ultra-real-time simulation on the simulation platform: the computing resources of the server are sufficiently powerful; the algorithm under test can receive virtual time.
Algorithms can accept virtual time, how do you understand this? An Hongwei's explanation is that some algorithms may need to read the time service on the hardware or the network time service under the condition of combining the hardware running platform, but cannot read the virtual time provided by the simulation system.
A Tier1 simulation expert said: Accurate time alignment and synchronization can be achieved in the engineering framework of the simulation system, PoseidonOS, and then the algorithm can be deployed on cluster servers, so that the time in the simulation space can be decoupled from the time in the real physical world. Once you untie it, you can "accelerate at will".
So, when doing time acceleration, can it be accelerated by 2 times or 3 times, what does this acceleration factor depend on?
An Hongwei's answer is: the server's computing resources, the complexity of the test scenario, the complexity of the algorithm, and the operating efficiency of the algorithm. That is to say, in theory, under the conditions of the same scene complexity and the same algorithm, the more powerful the computing resources of the server, the more possible acceleration times can be achieved.
What is the upper limit of the time acceleration multiple? We have to combine the principle of time acceleration to answer this question.
According to the person in charge of the simulation of an autonomous driving company, due to the inconsistency of the algorithm complexity and other reasons, the calculation speed of the training module, the vehicle control module and other modules is different, and the most conventional method of super real-time is to use the calculation of each module involved in the calculation. Do unified scheduling. The so-called acceleration means that the module with a faster calculation speed "cancels the waiting time"-no matter if you have not finished calculating another module, I will synchronize when the time is up.
If the difference in calculation period between modules is too small, the waiting time for cancellation is very small, so the acceleration factor will be very low; on the other hand, if the difference in calculation period of each module is particularly large, for example, it takes 1 second, And the other one takes 100 seconds, so there is no way to "cancel the wait".
Therefore, the multiple of time acceleration is often limited - 2-3 times is considered very high.
Even, many experts said that in practice, it is difficult to really speed up time.
Yang Zijiang, the founder of Shenxin Kechuang, said that if the algorithm of the automatic driving system has been compiled and deployed to the domain controller or industrial computer (this is the case in the HIL stage), it can only run in real time in the simulation system—— At this time, super real-time simulation is not feasible.
An Hongwei also said: "Hardware-in-the-loop (HIL, hardware-in-the-loop simulation) itself must be a real-time simulation. There is no concept of 'super real-time', and the terms 'parallel simulation' or 'time acceleration' are not applicable."
Bao Shiqiang said: "The premise of time acceleration is precise control of time and time synchronization. It is difficult to accelerate perception because the frequencies of different sensors are different. The camera may be 30 Hz, and the lidar is 10 Hz, similar to this, How do you ensure that the signals from different sensors can be strongly synchronized?"
In addition, a simulation expert who has worked in Silicon Valley for many years believes that no company can truly achieve ultra-real-time simulation. In the opinion of this expert, to improve simulation efficiency, massively parallel simulation is a more desirable solution.
An Hongwei believes that the time acceleration capability depends on the super real-time level of each instance, the total number of instances and the quality of the scene. "Actually, for cloud computing power simulation, the ultra-real-time level on a single instance is not very important. The core is to focus on the quality of the simulation on this instance."
Qingzhou Zhihang simulation experts even believe that the term "acceleration multiple" is actually not true. Because, between the time in the simulation and the time in the real world, there is not a simple multiple relationship, they don't even have a relationship. In practice, more technical means are used to reduce the occupation of computing power and improve the efficiency of timing scheduling to achieve the improvement of computing time.
In the real road test, the vehicle drives continuously. You would not say that this is a corner case. I will run it. It is not a corner case. corner case; on the simulation platform, engineers usually only capture those fragments related to the corner case (that is, "effective scenes"). After processing this matter, the clock will jump to the next time period without the need to Waste of time on the scene.
Therefore, when doing simulation, how to efficiently screen out effective scenarios is more important than the time acceleration factor.
Speaking of this, we can find that although the acceleration of time does not seem to be obvious, but to increase the virtual mileage on the simulation platform, in fact, we cannot mainly rely on the acceleration of time. The key is to rely on "multi-instance concurrency", which is actually to Do cloud computing power simulation and increase the number of servers and simulation instances .
Question 7: Large-scale concurrent testing
Can it support high concurrency in the cloud, and how large a scale of concurrency is supported? Where is the difficulty? Is it enough to just rely on heap servers?
Sounds right, but the problem is that every order of magnitude increase in the size of the server brings new problems -
(1) The cost of servers is quite high. Each server is hundreds of thousands. If there are 100 servers, the direct cost is tens of millions. The ideal solution is to go to the public cloud, but domestic OEMs still need to accept the public cloud. a period of time;
(2) In the case of large-scale concurrency, the raw data of the sensor is extremely large. The storage cost of these data is very high, and the transmission is also difficult - the synchronization of data on different servers will cause delays, which will affect the efficiency of Zhixing ;
(3) What runs on each road is not a continuous traffic flow scene, but a very short segment, maybe only 30 seconds, but usually thousands of roads run in parallel, if 1,000 roads have 1,000 algorithms running on 1,000 scenes , which poses a serious challenge to the architecture design of the simulation platform. (CEO of a simulation company)
However, regarding the last item above, An Hongwei said: This is a basic requirement for cloud computing power simulation, and it is not a challenge for us. The cloud simulation platform in Xiangcheng District, Suzhou has solved this problem as early as 2019. In addition, the scenes run on the cloud simulation platform will also have several kilometers of continuous complex/combined scenes. Xiangcheng's Robo-X simulation evaluation system includes such (group) scenes. Based on such scenarios, a "takeover" test under virtual simulation can be carried out.
Question 8: What is the most critical indicator to measure the strength of a company's simulation capabilities?
At the current stage, the simulations of different companies are quite different from the tool chain to the scene data used, from the methodology to the source of the data. Everyone is talking about "simulation", but they are not necessarily talking about the same concept. So, what are the most critical indicators to measure the strength of a company's simulation capabilities? After this round of interviews, we got the following answers:
1. Reproducibility
That is, whether the problems found in the real road test can be reproduced in the simulation environment. (Smart sailing with light boats, smart travel at the end of the day)
This issue will be discussed in more detail in the second half of this article.
2. Scene definition ability
That is, whether the simulation scenario defined by the company can really help improve the actual passing ability of autonomous driving.
3. Scene data acquisition ability
Scene data acquisition, production capacity, data versatility and reusability.
4. The quality and quantity of scene data
That is, how close the simulation scene is to the real scene, the accuracy, confidence, and freshness of the scene data, and the number of valid scenes, and whether there is enough massive simulation scene data to support multi-instance parallel simulation.
5. Simulation efficiency
That is, how to automate and efficiently do data mining to generate the environment model required for simulation, so as to quickly find real problems.
6. Technical Architecture
That is, whether there is a complete technical closed-loop system suitable for the needs of the tested object. (IAE Zhixing Zhongwei Li Yue)
7. Whether it has the ability of large-scale concurrent testing
Only in a large-scale test (the number of instances and scenarios are large enough), a company can build an evaluation system for model accuracy, system stability, etc., which tests a company's data management, data mining, resource scheduling and other capabilities. (Qingzhou Zhihang)
8. Simulation accuracy
Regulation-oriented simulation and perception-oriented simulation have different accuracy requirements - the former may depend on the vehicle dynamics model, what abstraction levels are there, and the granularity of interference behavior in traffic flow; the latter may depend on different sensors based on Noise added by different imaging principles, etc.
Usually, due to cost considerations, users hope that the technical architecture can be used universally. However, an overly general solution will sacrifice accuracy in some aspects - the accuracy of the model, the efficiency of the model, and the versatility of the model are a triangular relationship.
When it comes to the authenticity of the simulation data, we need to add another question: MANA has introduced real traffic flow scenarios into the simulation system. The real traffic flow at each moment is recorded, and then imported into the simulation engine through log2world. After adding the driver model, it can be used for debugging and verification of the intersection scene. So, how to guarantee the accuracy of this kind of data?
In this regard, the simulation expert at Momo said: "At present, this kind of data is mainly used for the development and testing of cognitive modules, so what we need is as realistic traffic dynamic behavior as possible. The data itself is discretized for the continuous world. , as long as the collection frequency meets the needs of cognitive algorithm calculations, we don’t need to compare this data with the true value (and there is no way to obtain the absolute truth value). Simply put, what we are pursuing is the rationality of the action and variety, not precision."
9. Consistency between simulation test and real vehicle test
A simulation engineer from a commercial vehicle unmanned driving company said that they often found that the results of the SIL test were the opposite of those of the real road test—there were no problems in the real road test, but there were problems in the SIL; and there were problems in the real road test. Problem scenario, but no problem in SIL.
A person in charge of the autonomous driving simulation of an OEM said that when they were doing HIL tests, they found that the performance of the vehicle in the simulation scene was more or less different from its performance on the real road. The reasons for this difference may be: (1) the virtual sensor, EPS, etc. are not completely consistent with the real vehicle; (2) the virtual scene is not completely consistent with the real scene; (3) the vehicle dynamics standard Can't do it right.
10. The role of simulation in the company's R&D system
The penetration rate of simulation in actual business, that is, the proportion of simulation data in the entire business usage data in the R&D process, and whether simulation is used as a basic tool for R&D and testing. (Millimeter Simulation Expert)
1 1 . Whether to form a commercial closed loop
A simulation expert of an autonomous driving company said: "For a simulation company, it is more important to take the lead in building a commercial closed loop than the advantages of the technology itself."
Bao Shiqiang, head of vehicle simulation at 51 World, said that the main points that customers pay attention to when choosing a simulation supplier are: A. Is the simulation module complete enough? B. What kind of toolchain can you provide him. C. The openness of the simulation platform.
Speaking of openness, Bao Shiqiang said: "The overall trend is that users do not want to directly buy a software to solve a specific problem, but want to build their own platform. Therefore, they prefer to simulate the supplier's technical modules. It can empower them to build their own simulation platforms. Therefore, simulation suppliers need to consider how to design API interfaces, how to integrate with customers’ existing modules, and even open part of the code to customers.”
Attachment: How to improve the reproducibility of the scene
"Whether a problem found on the road can be reproduced in the simulation environment" is regarded by companies such as Qingzhou Zhihang as one of the most critical indicators to measure the strength of a company's simulation capabilities. So, what factors will affect the reproducibility of the scene?
With this question in mind, the author repeatedly questioned many experts and got the following answers:
1. The vehicle model, sensor model, road model, and weather model may differ from the real situation.
2. The evaluation criteria of the cloud and the car may not be the same.
3. The communication timing and scheduling timing in the simulation system are inconsistent with the timing on the real vehicle. For example, when receiving a message, if you accidentally receive a frame early or a frame late, and finally under the butterfly effect, the difference will be very large.
4. The vehicle control parameters in the simulation system are inconsistent with the real vehicle. In the actual vehicle test, the accelerator, brake, steering wheel, and tires all exist in physical form, but there are no such physical components in the simulation system, so simulation methods can only be used. If the problem of vehicle dynamics is not handled well, The realism of the simulation will be compromised.
5. The scene data in the simulation system is incomplete. When doing simulation, we may only capture a certain segment of the scene, such as the data of a few seconds before and after the traffic light is not available.
6. The problem may be covered by the logical language describing the environment, and the level and coverage of the language definition may not be perfect.
7. The adaptability of the simulation software itself to various scenarios is not good enough, the switching between languages is not smooth, and it is difficult to support large-scale, multi-node operation.
8. The data in the real road has many variables. When doing simulation, in order to find problems as soon as possible, engineers need to "assume" certain parameters remain unchanged to reduce the interference on a key variable.
9. The calculation sequence between the perception, prediction, positioning and other modules of autonomous driving may be different in the cloud and on the car side; or the car side may not record certain information strictly—as long as there is one frame Differences can lead to problems with one result.
10. If it is a problem at the perception level, the scene reproduction needs to achieve a better reverse generation of the 3D scene, and then augment the data through generalization and perspective transformation. Every step here is a bit difficult. If it is a problem at the regulatory level, in order to accurately reproduce the scene, it is necessary to identify the interaction behavior and key parameters of the scene, so as to accurately generate and trigger the scene. (I firmly believe in Yang Zijiang, the founder of Kechuang)
The triggering scenario refers to whether the content that this scenario wants to test is realized. For example, if a pedestrian suddenly crosses the road in front of the main car, if the main car passes by the pedestrian before leaving, then the test effect will not be achieved, that is, the scene will not be triggered. For example, if a pedestrian crosses the road and then turns back, the speed of walking, the timing of turning back, and the speed of the main vehicle are all critical. This is a single person on a bicycle. Multi-traffic participants are much more complicated, and the relationship between them is coupled. Even if a parameter is slightly deviated, the effect of the simulation will be greatly reduced.
In the writing of this article, a lot of dry goods knowledge from the WeChat public account "Car Road Slowly" was cited. The author of the official account, Li Slowly, is a simulation engineer. This account focuses on sorting out simulation expertise, and recommends friends who are interested in this track to follow.
References:
A Super-Comprehensive Review of Autonomous Driving Simulation: From Simulation Scenarios, Systems to Evaluation
https://zhuanlan.zhihu.com/p/321771761
Recommended reading:
write at the end
About Contribution
If you are interested in contributing to "Nine Chapters Smart Driving" ("knowledge accumulation and sorting" type articles), please scan the QR code on the right and add staff WeChat.
Note: Be sure to note your real name, company, and current position when adding WeChat
And the information about the position of interest, thank you!
Quality requirements for "knowledge accumulation" manuscripts:
A: The information density is higher than most reports of most brokerages, and not lower than the average level of "Nine Chapters Smart Driving";
B: Information needs to be highly scarce, and more than 80% of the information needs to be invisible on other media. If it is based on public information, it needs to have a particularly powerful and exclusive point of view. Thank you for your understanding and support.
Recommended reading:
◆ Nine chapters - a collection of articles in 2022
◆Dedicated to "the first city of autonomous driving" - a "moving the capital"
◆ The 4D millimeter-wave radar is clearly explained in the long text of 4D
◆ Application of deep learning algorithm in automatic driving regulation and control
◆ Challenges and dawn of wire control shifting to mass production and commercial use