Communication group | To enter the "sensor group/skateboard chassis group" please add WeChat ID: xsh041388
Communication group | To join the "Auto Basic Software Group", please add WeChat ID: Faye_chloe
Remarks: Group name + real name, company, position
In October, Jiuzhang Zhijia published an article titled "Understanding the Application of Data Desensitization Technology in Smart Cars in One Article" . The article mentioned that in the current smart car industry, the car terminal needs to desensitize sensitive data. The scope is limited to the face and license plate information in the videos and images collected by the car. In addition, in the "Technical Requirements and Methods for Automotive Transmission Video and Image Desensitization" (hereinafter referred to as the "Method") issued by the China Association of Automobile Manufacturers, there are also requirements for functional requirements, desensitization methods, result evaluation and other related content.
At present, the "Method" has been officially released, and various OEMs have also responded to this requirement.
A company in the industry called Jiangsu Yuancai Technology Co., Ltd. has explored and researched the application of image data desensitization in the field of smart cars in advance, and has taken the lead in implementing the solution.
The author recently had an in-depth exchange and interview with Yuancai Technology’s CEO Zhou Xiang and deputy general manager Li Peng, and compiled the following for readers’ reference.
1. Why is image data desensitization so important?
With the continuous development of L2 level autonomous driving technology, human-machine co-driving has become a common phenomenon. Cameras, as the main sensor for outside-vehicle sensing and in-car driver monitoring, generate more and more sensitive image data, leading to data security issues. increasingly prominent in the field of smart cars.
On the one hand, the privacy leakage of images inside and outside the car is a serious problem. In the past two years, data security incidents caused by car camera functions have occurred frequently. For example, it was revealed that Gaohe Automobile’s “car-to-car interconnection” function allowed car owners to see the driving recorder footage of strangers; another example is Tesla The video image data captured by the camera in the car was stolen by hackers. These image data are collected and stored at will without the consent of the car owner, which poses a serious risk of privacy leakage.
On the other hand, OEMs have irregularities in the transmission and processing of image data. Before the relevant regulations were promulgated, OEMs believed that the data belonged to them and could collect it at will. Therefore, they did not take any measures to protect sensitive data throughout the entire data life cycle.
The above problems have prompted the continuous introduction of policies related to the smart car industry.
In July 2021, the State Information Office, the National Development and Reform Commission, the Ministry of Industry and Information Technology, the Ministry of Public Security, and the Ministry of Transport jointly issued the "Several Regulations on Automobile Data Security Management (Trial Implementation)" (hereinafter referred to as the "Several Regulations"), which clarified 6 types of important sensitive data, of which The fourth category is "out-of-car video and image data such as face information, license plate information, etc."
In August 2022, the China Automobile Association issued the "Technical Requirements and Methods for Automotive Transmission Video and Image Desensitization", which clarified the technical requirements and method standards for desensitization of relevant image data. This is also based on the "Several Regulations". Desensitization of image data has made detailed technical standards.
So, the development of smart cars is still in its early stages, and will strong industry supervision contradict this?
Zhou Xiang said: “The current tightening of supervision has increased the costs of OEMs and Tier 1, but data supervision is beneficial to the development of the industry, especially for the current L2 and L2+ levels of autonomous driving technology. Without data supervision, The data will be in a state of streaking, which will create a lot of risks."
After the introduction of relevant policies, OEMs have increased their emphasis on data desensitization.
Li Peng mentioned that it is expected that in the next six months, image data desensitization will be in the SOR stage (demand specification), and the OEM will put forward relevant requirements for data desensitization to suppliers, which in addition to the incremental market, also includes some existing stocks. market. For example, the 360 Sentinel function/surround view/driving recorder, which was forced to be shut down in the early stage, also urgently needs suppliers to discuss the need for desensitization in related functions again.
2. Application of image data desensitization technology in smart cars
After we learned above about the importance and urgency of image data desensitization in the field of smart cars, let's take a look at how image data desensitization technology is specifically applied in the field of smart cars? The author will elaborate on this issue in detail from 5 dimensions, including applicable data range, applicable data requirements, image desensitization methods, image desensitization execution, and data requirements after desensitization.
2.1 Applicable data range
Article 3 of the "Several Regulations" mentions: "Important sensitive data includes off-car audio and video data such as faces, voices, license plates, etc. This regulation only targets image data outside the car." However, the "Method" standard was subsequently introduced However, it is mentioned that it is “suitable for desensitizing face and license plate data in videos and images collected by the vehicle.” This standard is not limited to image data outside the vehicle, that is to say, in-car (face) and vehicle-side image data . All external image data (face + license plate) need to be desensitized .
2.2 Applicable data requirements
After determining the applicable data range, the next question is what kind of data needs to be desensitized. In this regard, the "Method" has made clear provisions on the data format and image quality requirements.
First of all, the "Method" stipulates that on-board data processing equipment should support desensitization of raw binary data, and the data uploaded to the cloud must at least meet the following format requirements.
Image file format: any one of JPEG, JPEG2000, BMP, and PNG;
Video codec format: any of H.264, H.265, and MPEG-4;
Video file format: any of mp4, avi, mov, wmv, 3gp.
Secondly, the "Method" also stipulates the image quality requirements for faces and license plates, including not only the resolution, posture, integrity, clarity, and intensity of the image RGB that need to be met by the face image, but also the image resolution of the license plate outside the vehicle. , Recognizability under lowest illumination, geometric distortion, and motion blur .
Figure: Image quality requirements for faces and license plates
In addition, if the image data cannot meet the above requirements, no desensitization processing is required.
Li Peng said: "Image data desensitization technology is different from autonomous driving technology. It mainly depends on whether the images of license plates and faces are clear . If the images are no longer clear, then there is no need to do desensitization. .For example, in high-speed scenes, the image in front has already appeared smearing, which reduces the requirement for desensitization, or even does not require desensitization.”
Therefore, the application of image data desensitization technology has nothing to do with specific application scenarios, such as high-speed or urban areas. Its technical performance mainly depends on differences in camera performance, such as sensitivity, dynamics, etc.
2.3 Methods of image desensitization
After determining the applicable data scope and data requirements, what specific technical methods are needed for image desensitization?
The "Method" points out that the main methods for desensitizing image data include erasing, applying uniform color blocks, etc., but do not include low-pixel processing and mosaic . It can be interpreted that:
Specific methods include erasing, applying uniform color blocks, etc.;
The desensitized image needs to be irreversible - the image cannot be restored.
Although an information security engineer from an OEM mentioned that the technical difficulty of image desensitization is not high in nature and has been used in other fields, the image data collected by the vehicle-side camera is often in a dynamic state. It is necessary to introduce some related technologies to locate the sensitive areas in each frame of image.
Zhou Xiang said: "Previous image desensitization technology mainly used target tracking, but now that the SoC's computing power has been enhanced, the desensitization technology can directly detect sensitive areas."
Li Peng also said: "The application of desensitization algorithms in smart cars is essentially similar to the sensing function of active safety, but the desensitization work will be much less difficult than the original active safety."
It seems that the threshold for desensitization technology is not high, but is this really the case?
Regarding the technical difficulties of desensitizing vehicle-side image data, Li Peng said: "The current image desensitization technology is mainly based on deep learning, and the strength of the technology depends on the adequacy of the data sample size used to train the model and the diversity of application scenarios. "
In general, the key to desensitizing image data is to rely on the accumulation of experience in early autonomous driving technology, including the accumulation of data and algorithm training experience.
2.4 Execution of image data desensitization
The "Method" points out the execution process of image data desensitization, including image data input, preprocessing, positioning of sensitive areas, desensitization processing, post-processing, and output of image data.
Figure: Process of image data desensitization
(Data source: "Technical Requirements and Methods for Automobile Transmission Video and Image Desensitization")
This execution process is mainly the process of image data desensitization, but how does image data desensitization operate specifically when combined with autonomous driving technology?
In response to this problem, Li Peng said: " Image data desensitization will not be carried out at any level of autonomous driving (such as perception, positioning, decision-making, control). Since the desensitization algorithm may cause a delay in the autonomous driving algorithm, so Data only needs to be desensitized when it is transmitted to the outside .
"For example, in Sentinel mode, when users view background data, the data needs to be desensitized before transmission, so users will not see sensitive information. Locally stored data does not need to be desensitized, but when Desensitization precedes any form of outward transmission.
"For another example, when the environmental data detected by the vehicle's multiple visual sensors has a large gap, all the data on the vehicle end needs to be uploaded to the cloud for retraining. At this time, the image data also needs to be desensitized."
2.5 Data requirements after desensitization
For desensitized image data, the "Method" also points out corresponding requirements.
First, the desensitized image should be unrecognizable by the human eye, and sensitive areas cannot be restored by related technologies, such as image super-resolution reconstruction based on deep learning or image restoration based on generative confrontation networks.
Second, the intersection-over-union ratio (IoU) of the face or license plate. Note: The intersection-over-union ratio is the overlap rate between the generated candidate frame and the original marked frame, that is, the ratio of their intersection and union. When there is complete overlap, the ratio is 1 ) should meet 50%-75% .
Third, after desensitization, each frame of the image in the original video needs to be converted into a new video according to the encoding and frame rate information of the original video, and the converted video format must be consistent with the original format.
3. Challenges of image data desensitization in smart cars
3.1 Technical challenges
At present, cameras have become an indispensable sensor in autonomous driving technology solutions. As the performance of vehicle cameras continues to improve, there are more and more types. On the one hand, the performance of cameras ranges from megapixels to 4K and 8K. etc., resulting in increasingly higher definition of image data; on the other hand, the types of cameras are very diverse and complex, including single-eye, binocular, multi-eye, surround view, etc. outside the car, as well as DMS, OMS, etc. inside the car .
Based on these factors, image data desensitization will face some technical challenges.
First of all, the higher the performance of various cameras, which means more high-quality image data will be produced, and the clarity will be easier to meet the requirements for image data desensitization, which will bring a large amount of sensitive data.
Secondly, cameras used in various scenes will have different technical solutions for image data desensitization due to different corresponding functional requirements, which requires deeper scene understanding capabilities and corresponding deep learning capabilities. For example, the horizontal FOV of some fisheye cameras is very large, up to 270° (such as when used in sentinel mode). While it acquires more sensitive data, it also requires real-time desensitization of image data.
3.2 Engineering challenges
In terms of engineering, the biggest challenge for image data desensitization lies in the difficulty of migrating the desensitization algorithm , that is, how to deploy the image data desensitization scheme on different SoC platforms.
Regarding this issue, taking the experience of actual project implementation as an example, Li Peng said: "First of all, chips with large platform and large computing power are easy to desensitize images, while some SoC chips with insufficient performance have very limited computing power. It is difficult to ensure the original system On the basis of normal business operation, if the image desensitization algorithm is deployed, the computing power allocated to image desensitization is quite limited, which will affect the efficiency of data desensitization.
"Secondly, there are differences in the architectures of different SoC chips nowadays, resulting in relatively large differences in performance occupancy. For example, some SoC chips are ARM-based processors, which may be equipped with A53 or A55 cores, and each processing Server performance will also vary, with some configurations having a GPU and some not.
"Furthermore, different OEMs have different requirements for image data desensitization—some OEMs want to deploy data desensitization on the GPU, which will lead to a large GPU occupation, while others hope that the GPU will not be occupied. , thus hoping to deploy data desensitization on the CPU."
4. How to deal with corresponding challenges
Faced with the above challenges, how should the industry respond? As image data desensitization becomes increasingly urgent, various data vendors will also have corresponding coping strategies. Among them, an image data desensitization vendor introduced its own coping strategies by combining its own characteristics and the status quo of industry development.
4.1 Who is this company - Yuancai Technology
At this point, readers may be curious: What kind of company is Yuancai Technology?
According to Yuancai Technology, the company is jointly funded and established by Huashe Design Group Co., Ltd. (hereinafter referred to as "Huashe Group") and Shenzhen Youjia Innovation Technology Co., Ltd. (hereinafter referred to as "MINIEYE"), aiming to promote domestic vehicle roads. The development of the collaborative autonomous driving industry is committed to becoming a digital transportation service provider with full-stack R&D capabilities in software and hardware as its core.
So, as a company positioned as a digital transportation service provider, why did Yuancai Technology choose to deploy image data desensitization business?
Zhou Xiang said: "In general, first of all, this is an industry and regulatory trend. The company has responded to the relevant calls of the government. Now the public is becoming more and more aware of personal privacy protection, and regulatory authorities and enterprises are also aware of this. Our R&D team has made a lot of relevant technical reserves in the field of image data desensitization, which can empower other companies in the industry.
"Secondly, the current OEMs require Tier 1 products to have image data desensitization technology when bidding externally. Although some OEMs are capable of self-developed image desensitization technology, the time and money cost of self-development is relatively high. It does not necessarily have an advantage due to the cost of directly using off-the-shelf solutions. At the same time, the regulations are not only mandatory to apply to all new models, but also cover the models that are currently on the market, that is, the stock market. Yuandrive Technology also seizes created such a market opportunity.
"Third, the company relies on its shareholder MINIEYE's accumulated advantages in algorithms in the vehicle industry, and has relatively complete technical support to complete the image data desensitization business."
4.2 The company’s corresponding countermeasures in facing challenges
4.2.1 Technical countermeasures
As mentioned above, the improvement of camera performance has brought about an increase in the clarity of image data, which in turn has brought about an increase in the amount of sensitive information, and has also increased the potential demand for image desensitization, which will eventually reduce the efficiency of desensitization. For example, when the original camera performance is not good, some image data is not clear enough, so there is no need for desensitization, but after the camera performance is enhanced, the highly clear image may force desensitization of each frame of the image. Sensitive .
Faced with this challenge, Li Peng made an analysis using the user side (car owner) as an example. He said: "When the user connects to the vehicle through a mobile phone or other mobile devices, the vehicle end will push video streams and image data to the user in real time, and these may be the image data generated by the 8MP camera, but in the actual data transmission process, the data It will be compressed or cropped to a certain extent, and the company will deploy a desensitization algorithm on the cropped video stream. This not only meets users' needs for privacy and security, but also meets users' needs for real-time access to image data."
4.2.2 Engineering Countermeasures
How does Yuancai Technology respond to the challenge of porting desensitization algorithms between different SoC platforms?
Li Peng said: “First of all, the company will have a certain understanding of the performance of each SoC, evaluate the needs of different customers, and make customized solutions. For example, when deploying data desensitization algorithms for the 360 Surround View function, customers will require desensitization. The desensitization algorithm does not occupy the performance of the GPU, so the desensitization algorithm will be deployed directly on the CPU to run. Secondly, without affecting the original business, the company will make reasonable resource utilization based on different SoC characteristics. For example, a certain Some SoC platforms have NPU acceleration units and can run desensitization algorithms smoothly, so there is no need to occupy GPU or other hardware performance."
4.3 Advantages of the company
To be able to cope with corresponding challenges freely, one's own basic skills must also be polished.
First of all, what kind of technical system does Yuancai Technology have?
Zhou Xiang mentioned that the entire system of Yuancai Technology comes from MINIEYE’s technical architecture, and both algorithms and software and hardware follow MINIEYE’s successful experience.
From the perspective of algorithms, MINIEYE self-developed neural networks such as ThiNet (neural network compression architecture), FastNet (embedded neural network acceleration library), HardNet (neural network architecture IP), etc., allow the company to quickly accumulate know-how related experience .
From a software and hardware perspective, the company has absorbed MINIEYE’s mass production experience in ADAS and in-cabin sensing.
In terms of hardware, it mainly includes the mass production experience of dual-warning ADAS products for commercial vehicles and intelligent driving domain controllers for passenger vehicles in Dongfeng, Liuqi, Shaanxi Automobile, Chery, BYD and other customers; Mass production experience from customers such as SF Express.
Secondly, with the support of the above-mentioned technical system, Yuancai Technology has formed its own unique advantages. The author summarizes the advantages in three aspects.
4.3.1 Technology and engineering advantages
(1) Algorithm transplantation ability
Algorithm transplantation capabilities can help companies better apply data desensitization technology to SoC chips on different platforms and solve engineering problems caused by hardware differences.
Regarding the company’s algorithm transplant capabilities, Zhou Xiang said: “First of all, if the algorithm is to be better applied, it requires a sufficient amount of sample data . Based on the accumulation of image data experience, Yuancai Technology has inherited the data sets accumulated by MINIEYE. Containing various types of information, these data will be imported into the technical system of Yuancai Technology to help the company improve its data processing capabilities.
"Furthermore, algorithm transplantation capabilities also require a certain understanding of the hardware performance of mainstream SoC chips . MINIEYE's accumulated development experience on mainstream platforms such as Qualcomm, Xilinx, and TI, as well as the strategic partnership with Horizon, will provide It provides strong support for the algorithm transplantation and development of Driving Technology."
(2) Image desensitization speed
One of the key technical indicators for measuring image data desensitization is the speed of image data desensitization , that is, how many frames of image data the desensitization algorithm can process per second, which mainly depends on the image recognition algorithm capability and the utilization of hardware performance.
Li Peng said: “Based on the characteristics of different SoC chips , the company can basically ensure that 25 frames per second images can be achieved on an A53 core by utilizing hardware performance resources, especially on some SoC chip platforms that do not include NPU and GPU. The data desensitization speed can meet the real-time desensitization needs of driving recorders and other devices.”
So, what is the specific industry level of image data desensitization speed of 25 frames per second?
Li Peng continued: "The desensitization speed of 25 frames per second is a relatively advanced level in the industry, which can basically guarantee that customers will not experience frame drops or freezes in image data when viewing mobile video streams. Although the company's current desensitization The speed can be greater than this value, but there will be no difference in user experience. However, if it is lower than this value, users will experience some unsmooth reading."
4.3.2 Business resources
For a new business, strong sales support is needed to promote it into the market in the early stage. Since Yuancai Technology mainly focuses on research and development, but does not have many marketing and sales personnel, will this affect the company's market development?
Zhou Xiang said: "At the beginning of the establishment of Yuancai Technology, the two major shareholders behind it have already brought in some business resources. Among them, MINIEYE will bring in the resources of the OEMs it has cooperated with and some long-term supplier resources; and Huashe Group will introduce smart parking and vehicle-road collaboration related businesses to Yuancai Technology. These two businesses require cameras to be deployed at the road end, and the image data at the road end may also have desensitization requirements in the future."
4.3.3 Policy understanding ability
Finally, during the communication process with the two managers of Yuancai Technology, the author deeply felt that the company pays close attention to policy trends and has a very deep interpretation of policies. This will also help the company form certain foresights in its future product strategic layout. This will help the company open up the market faster.
On the one hand, Yuancai Technology mentioned that the company participated in the compilation of the "Method"; on the other hand, the company's CEO Mr. Zhou Xiang is a distinguished expert of the big data center of the China Automobile Association and has participated in many domestic data security-related projects. discussion on specification formulation.
5. Unfinished words
Desensitizing images of faces and license plates is just a small step. In the future, in the field of smart cars, the definition of sensitive image data may continue to expand.
Zhou Xiang said: "Supervision on data security may become more stringent. The collection of image data may not only be limited to faces and license plates, but also some environmental data, such as desensitization of roadside camera data."
Referring to the GDPR "General Data Protection Regulation" promulgated by the European Union in 2016, Li Peng said: "This is a regulation covering a variety of data. The regulation points out that information that can be used to generate user profiles, even signage, stores and trademarks There will be a certain protection mechanism for other information, and the regulations are more stringent than the current domestic regulations."
"Furthermore, personal data needs to have rights to use, destroy, save, etc., but the current situation in China is not ideal. OEMs still need to provide users with an option in a similar written form, such as whether to authorize the OEM Use certain information, and the agreement needs to clearly state key information such as the collection and use of the data, the specific use of the data, and the method of terminating the agreement. The industry and regulatory authorities need to find a balance between data security and the development of autonomous driving. Only in this way can we steadily promote the progress of science and technology." Zhou Xiang mentioned.
In the future, in what direction will the policies related to image data desensitization be further refined? Perhaps we can wait and see in the near future.
write at the end
Communicate with the author
If you want to communicate directly with the author of the article, you can directly scan the QR code on the right and add the author's WeChat account.
Note: Be sure to note your real name, company, and current position when adding WeChat
As well as information such as intended positions, thank you!
About submission
If you are interested in contributing to "Nine Chapters of Intelligent Driving" ("knowledge accumulation and compilation" type articles), please scan the QR code on the right and add the staff WeChat.
Note: Be sure to note your real name, company, and current position when adding WeChat
As well as information such as intended positions, thank you!
Quality requirements for "knowledge accumulation" manuscripts:
A: The information density is higher than most reports of most brokerage firms, and not lower than the average level of "Nine Chapters Smart Driving";
B: Information must be highly scarce, and more than 80% of the information must not be seen in other media. If it is based on public information, it must have a particularly powerful and exclusive perspective. Thank you for your understanding and support.
Recommended reading:
◆ Nine Chapters - A collection of articles in 2021
◆Detailed explanation of Inceptio’s self-driving truck mass production technology methodology
◆Understand the application of data desensitization technology in smart cars in one article