Source: Seven Traffic Network
Zhang Yi | Author
Ann | Editor
Baotu.com | source of header image
Editor's Note: On July 25, at the "Data Brain" Technology Symposium on Transportation hosted by the Academy of Sciences of the Ministry of Transport and Saiwen Transportation Network, Zhang Yi, Director of the Shanghai Transportation Information Center of the Shanghai Urban and Rural Construction and Transportation Development Research Institute Made a report on "The trinity of "quantity, quality and use" promotes the development of Shanghai's traffic data brain".
The report introduces the traffic data processing and application practice in Shanghai from the three aspects of traffic data quantity, quality and application. In terms of "quantity" of data, the integration of traffic data resources is realized by gathering data from various parties in the industry; in terms of "quality" of data, four reasons for the current low quality of traffic data are analyzed, and a proposal based on system monitoring, data monitoring and Methods of standardizing data such as facility monitoring; in terms of "use" of data, it focuses on analyzing the system architecture of Shanghai's comprehensive traffic information platform and its corresponding application practices.
1
The "quantity" of data is the basis
The guiding ideology that Shanghai's smart transportation development has always followed is "grasp the status quo, find out the rules, scientifically induce and effectively command", emphasizing that system construction must pay attention to the effectiveness of data applications.
In a nutshell, the smart transportation in Shanghai has gone through four stages in the past 20 years: the first stage is before 2003, which is counted as the initial stage; the second stage is the large-scale and rapid development stage from 2003 to 2010; the third stage From the 2010 World Expo to 2015, it belongs to the stage of big data application in the transportation field; the fourth stage is from 2015 to the present, which belongs to the integration development stage of traffic data "brain" and smart city.
Practice has proved that adhering to the guiding ideology, "carrying out data construction and focusing on application effectiveness" is the main line to promote the development of Shanghai's intelligent transportation.
Before 2003, Shanghai's intelligent transportation system mainly built relatively small-scale business systems. At that time, there was still a lack of large-scale data volume, so it was impossible to talk about data collection and sharing.
From 2003 to 2010, in order to welcome the Shanghai World Expo, Shanghai began a large-scale transportation information system construction period. A large amount of data collection, mining analysis and business applications have strongly supported the transportation security of the World Expo. Some documents write: Shanghai The intelligent transportation system has played an irreplaceable role in the traffic guarantee of the 2010 Shanghai World Expo.
Benefiting from the comprehensive application experience of the World Expo, since 2011, we still pay attention to the continuous collection of data resources, especially the continuous collection of mobile Internet data, operator data and cross-field data, and the data volume on the Shanghai Transportation Comprehensive Information Platform has skyrocketed.
It is precisely because of the large amount of data that the Traffic Information Center undertook the first big data research project approved by the Chinese government in 2012, built a traffic big data platform, launched new services such as urban expressway traffic forecasting, and improved traffic big data. Empower traffic command.
At the same time, the real-time calculation and migration analysis of urban population travel based on big data provides data guarantee for the traffic planning and traffic policy research of the Institute of Transportation after the merger of subsequent institutions.
In 2015, Shanghai Traffic Information Center took the lead in implementing the comprehensive traffic information application and service project. The project not only empowers municipal-level traffic management departments to empower district-level traffic management through comprehensive analysis of data resources; it is also deeply integrated into the city's "urban transportation system" and the district's "urban transportation system" to empower one-network management. Driven by new technologies such as mobile Internet, mobile payment, cloud computing, and artificial intelligence, the four new economies have flourished in Shanghai.
In the field of transportation, there have been many new management requirements such as the supervision of online car-hailing, the supervision of shared bicycles, and the supervision of courier brothers. Shanghai must take the lead in proposing scientific solutions to these challenges.
To this end, in 2019, we built the Shanghai public parking information platform system and the Internet bicycle rental supervision platform. These platforms have become benchmarks for other provinces and cities to learn from.
Shanghai Traffic Information Center is deeply involved in almost all of the above systems. Summing up the experience, we can see that the reason why we can achieve practical results is because we always regard data as the core element and basic requirement, and always regard data quality as the lifeline of the system, and we will never relax.
For example, Shanghai’s public parking platform is based on “data” and evaluates district management departments, operating yard companies, equipment suppliers, and individual parking behaviors, and deeply integrates business management with the system, effectively reflecting the value of information system data resources.
For shared bicycles, the "last mile" problem of travel is solved at the transportation level, and a large amount of travel data is also generated; but it also brings urban management problems. How to solve the problems of excessive investment and random parking of shared bicycles, etc., the data must be integrated into the "urban transportation system" and incorporated into urban management.
In 2022, the Shanghai Urban Management Refined Platform Project launched in Shanghai will fully integrate various data to empower refined and intelligent urban governance.
2
The "quality" of the data
Traffic data naturally has the characteristics of big data, and the data scale is huge, but the feedback often heard is that the data quality is not high. The reasons for the low data quality can be roughly summarized into four reasons: First, the quality of the data generated at the source itself has defects. Data quality is difficult.
Second, it is difficult to find and repair data quality problems. For example, due to outage of power supply in the field, communication failure, etc., it is extremely difficult to repair data with a wide range and long time.
The third is that it is difficult to evaluate or evaluate data quality. It is difficult to judge data quality with a unified standard. Different judgment standards need to be proposed according to different application requirements. For example, research work on the surface can be supported by statistical data. Even if there is a certain lack of data, it will not affect the research results; but for "problem" research, it is necessary to pay attention to "extreme value data". reveal the problem. At present, the refinement of urban management must focus on research based on extreme values.
Fourth, the business data generated by the underlying business system is difficult to meet the data quality requirements of the superior department for comprehensive supervision and decision-making. Because the business system is a relatively single-function system, the comprehensive supervision of the upper layer, especially the comprehensive decision-making needs comprehensive data support. When making comprehensive decision-making, the data content and data elements of the underlying business system are incomplete. Support lacks effective support.
During system construction and daily operation management, attention must be paid to the monitoring of system operation and data quality evaluation. Our project feasibility study report will have two major supports for the expression of the overall framework of the system: one is "standards and specifications" and the other is "institutional mechanisms".
But in reality, system construction often ignores this aspect of work. How to achieve stable system operation and reliable data quality? A lot of meticulous work needs to be done, and it is necessary to start from data quality monitoring and conduct comprehensive monitoring of the system.
Taking the Shanghai Transportation Comprehensive Information Platform as an example, we monitor the health status of each device in the computer room in real time, the progress of the application software, each data update and its quality change, and even evaluate the reliability of each data record. If data interruption or data quality problems are found, the data-associated application, data source system and its data source management unit will be traced.
For example, the Shanghai Public Parking Information System assesses parking lot equipment suppliers through data quality, prompting manufacturers to improve product quality, thereby improving the data quality of the industry as a whole.
3
"use" of data
After assessing data quality, a post-assessment of data quality through data application is also required. For example, Shanghai Pudong New Area has a large geographical area and many smart traffic management systems, in order to better guide the follow-up development.
This year, the transportation department of Pudong New Area commissioned Shanghai Transportation Information Center and Tongji Design Institute to conduct an overall assessment of its smart transportation system. This work is still closely related to data quality and its supporting application effect, and carries out data evaluation, data application and external facility health evaluation.
Of course, all applications must have a reliable and stable system architecture. According to the needs of our own business, our team designed and built a system architecture that meets our own conditions, which can well support our system applications.
The design of the system architecture is also based on different data processing requirements and adopts different technical solutions. The architecture of the Shanghai Traffic Comprehensive Information Platform is shown in the figure, which includes the acquisition layer, communication layer, data layer and application layer from bottom to top.
In the data layer, the core problem is to build different storage processes according to data quality and data sources. It is hoped that the equipment can be used in large quantities, and the system is stable. It is also hoped that the related requirements will be higher. It can be summarized as "small core, large peripheral".
"Small core" refers to the top-level core database. Under the premise of satisfying all business functions, data processing logic is minimized, especially for heavy analytical business. By adopting dual-node Real Application Cluster ORACLE Database (real-time application cluster ORACLE database), it has Capabilities of fast online transaction processing type business access, high availability and load balancing.
The "big periphery" is a star branch, including a distributed processing architecture, an in-memory database, and a lightweight database/processing server farm.
HADOOP+SPARK is used to process full-scan analytical services, in which HADOOP's own MAP/REDUCE is replaced with the SPARK memory computing engine to obtain higher performance gains; TIMESTEN memory database is used to process streaming services with a large amount of data, and every Each cycle needs to process a large amount of real-time data, and only the data of the most recent cycle or several cycles are always kept in the memory for recycling; the lightweight database Mysql is used for data services with relatively simple business processing logic but relatively large IO throughput processing, such as data verification and cleaning.
In addition, by using VMware vSphere to build a virtualization platform, the in-memory database, lightweight database, and processing server are virtualized to provide services, and the virtualization platform is used to provide high availability, load balancing and horizontal expansion capabilities. Using KAFKA CLUSTER as the data communication bus simplifies communication interface management and improves horizontal expansion capability and high availability.
At present, the architecture has been used for more than 10 years, and the system has been stable and reliable, which well meets the data collection and data analysis needs of Shanghai Traffic Information Center. At present, the traffic data of Shanghai Traffic Information Center mainly has the following applications:
(1) Real-time monitoring: At the traffic level, it includes real-time monitoring of road traffic, public transport and external traffic. While monitoring, it automatically calculates traffic anomalies; pre-implementation and post-evaluation of traffic policies.
For example, we used data analysis to evaluate Shanghai's policy on foreign-brand vehicles, Shanghai's Wuning Road Expressway and other congestion mitigation measures. The data shows that the time limit for the congestion mitigation effect is about 2 months, that is, after the implementation of the congestion mitigation policy for 2 months, the congestion will return to the original congestion state.
(2) Public transportation evaluation: The second application is the evaluation of public transportation. The Academy of Transportation Sciences promotes the priority development of public transportation from the perspective of the Ministry of Communications. The Shanghai Transportation Information Center has also carried out assessments on public transport infrastructure supply, network design, operational efficiency, corporate economy, safety and environmental protection, etc. These assessments are of great help to bus line optimization, scheduling and service improvement.
(3) Signal evaluation: The intersection signal light control system is a powerful tool to ensure smooth and orderly ground roads. Using multi-source data such as GPS floating vehicles and SCATS, the Shanghai Traffic Information Center has carried out the analysis and evaluation of the traffic signal control system in Shanghai. The system can evaluate the pros and cons of the signal system at each intersection in real time and possible optimization solutions.
There are three main technologies involved in the signal evaluation system: data application system and service collaboration technology, intersection congestion holographic image technology, and intersection control intelligent evaluation technology. These technologies must have a higher data quality guarantee.
(4) Social services: Shanghai is the first city in China to develop traffic information broadcasting station services. Timely and accurate information is the basis for developing traffic information services.
To this end, we have developed the information broadcasting system of Shanghai Traffic Radio Station, which provides real-time and rich traffic travel information for the traffic station's travel service. When the system is designed, according to the differences in the needs of travelers for different levels of information, the travel-related information is classified and set, and different broadcast frequencies are used to ensure the quality of radio traffic information services.
(5) Industry management: In order to allow data to empower district-level traffic management, the Shanghai Traffic Information Center has developed a traffic management platform integrated into the district urban transport center for Xuhui District, Huangpu District, Changning District, Jing'an District and Yangpu District, which not only reduces The investment in the field collection facilities of the district-level government can meet the municipal-level digital transformation requirements such as "one map" and "one code" in the city. At the same time, the data quality of the municipal-level information system can be tested through district-level applications.
As shown in the figure, the Shanghai Xuhui District Traffic Police Command and Judgment Platform and the Xuhui District Construction and Communications Commission Traffic Comprehensive Analysis Platform supported by municipal-level data have realized the sharing and exchange of district-level data in application.
(6) Comprehensive application: cross-departmental comprehensive application is the touchstone for testing data. In 2022, the epidemic control in Shanghai will be effectively controlled. From which direction should the resumption of public transportation resume?
Entrusted by the Municipal Transportation Commission, we carried out an analysis of the public transportation allocation for the resumption of work and production based on the infected persons and population data in the community. Shanghai's transportation department and environmental protection department actively cooperated to carry out motor vehicle mobile source pollution monitoring and "double carbon" policy research, and achieved good results.
