The source of this information is open to the Internet and is for personal study only. Please do not use it for commercial use. If there is any infringement, please contact to delete it.
Development background of big data management center
In order to build an excellent global city and realize the goal of modernizing government governance capabilities, the Municipal Big Data Center has taken the lead in a series of practical work in government public data management and Internet government services. According to the requirements of the "Municipal Public Data and One-Network Office Management Measures", the informatization project construction has been carried out in the field of government affairs services of "One-Network Office" in the early stage. Public data to explore the needs of applications in the field of government services.
As the main management unit of the city's government data, the Municipal Big Data Center is responsible for the implementation of policy localization, government data exchange, big data application research, informatization construction operation and maintenance, and other data-related responsibilities. It has been researching since the center was established. At the end of last year, we explored the practice of government data exchange and sharing as the core of government data management and application methods. However, with the deepening understanding of the functions of the center, we believe that the center should not only serve as a "hub transfer center" for the city's government data, but should also become a The "collection management center" of each government service line field provides the greatest support at the data level for the city's economic activities, public utilities, social relations, personnel density and other urban management fields.
Construction and use of big data sharing and exchange platforms
Since last year, the Big Data Center has launched the construction of the city's data sharing and exchange platform project according to the overall plan, and officially started the city-wide trial operation in January 2019.
Platform construction includes:
1. General integration and partial application development: platform integration portal and basic function menu-level integration of the overall platform;
2. Data governance subsystem: including data exchange module and service management module, realizing data resource catalog and three lists 3. Data sharing and exchange subsystem
: realize data exchange engine, unified scheduling engine, task management, data bridging and other functions, as well as storage management, sharing and exchange of data lake data;
4. Data quality supervision and support subsystem: implement data quality management functions, including data quality rule formulation, data quality audit, and closed-loop management of data quality issues; 5. Big data support management subsystem: include municipal data lakes and municipal
databases , bringing together the original data after initial governance of the "four basic databases", "municipal unified construction system", "municipal government-run systems" and "administrative district systems"; and store them after a series of cleaning, conversion, loading, High-quality government data resources after the governance step provide data sources for urban management and public services;
Analysis of the target of platform construction in this period
Gathering: Promote the integration of population, legal person, and spatial geographic database data sources, and promote the docking and utilization of electronic certificate databases. Gradually promote the collection and aggregation of social data and Internet data.
Management: build a one-stop visual management of data assets, and realize the whole process management of data and the safety supervision of each link through the unified management of the blood lineage of data assets and the catalog of information resources.
Communication: realize the authorization management mechanism of data resources based on application scenarios, and promote the full sharing of public data on demand. Realize the open management of data classification and classification, and expand the functions of the shared exchange platform.
Purpose: To carry out thematic database construction, promote the integration of data resources and the application of data analysis. Carry out research on data application models and algorithms, and explore innovations in data service models.
Security: Create an overall security management and control system to ensure the security protection capabilities of government data on the platform in the entire life cycle of data aggregation, data management, data sharing, and data opening.
Content analysis of platform construction in this issue
Overall Architecture·Functional Architecture Planning of Big Data Resource Platform
Overall Architecture·Functional Architecture Design of Big Data Resource Platform
Overall Architecture - Big Data Resource Platform Deployment Architecture
Overall Architecture Big Data Resource Platform Data Architecture
The big data resource platform supports the three core data services of data sharing and exchange (production function), data analysis, and data openness, including the three major data areas of big data resource area, internal sharing and exchange area and external opening area.
Big data resource area: includes two major data resources, the municipal data lake and the municipal database.
The data lake: mainly stores the business database of municipal government departments and unprocessed original business database, and supports storage of various types such as structured and unstructured The data.
Municipal database: mainly used to store high-quality government data resources after a series of cleaning, conversion, loading, and governance steps.
Internal sharing and exchange area: mainly used to store data in the collection and sharing and exchange of government affairs data resources of various departments and districts in the city, including directly shared and exchanged data and processed data from municipal databases.
Open to the outside world: mainly used to store data that is open to the outside world, and can provide a higher level of data security protection through the isolation of a separate external development zone from the big data resource area.
Overall Architecture Big Data Resource Platform Permission Architecture
Overall Correspondence 17
1. Data management subsystem · batch data access
Batch data processing uses the task scheduling method of the data collection platform to regularly detect the changes in the front-end database data and structured files of the committee, office and bureau according to the time stamp, and extract incremental data to the data lake through the analysis of the data collection platform.
1. Data management subsystem · streaming data access
Flow data processing independently deploys the Kafka message system in the data lake, and connects the flow data to the big data support platform by pushing or automatically pulling from the committee.
1. Data management subsystem and data support platform
1. Data management subsystem and data support platform
Batch data processing:
All kinds of batch data enter the data support platform through the data collection function, and after storage, cleaning, summary and association summary, etc., generate application data, and realize data sharing or opening.
Streaming data processing:
After streaming data enters the data support platform through the data collection function, according to different needs, it can realize the opening after real-time data calculation, and also realize the aggregation and generation of application data after real-time data analysis, and then realize data sharing or opening.
1. Data management subsystem · data asset management
1. According to the provider's data supply responsibility, catalog the access data resources, maintain their business attributes and management attributes, and make them initially meet the sharing conditions.
2. Design data model structure and business code with reference to data standards, and realize data model management and control by establishing change management and version control.
3. Define data asset specifications (such as database tables, files, folders, interfaces, etc.) through basic metadata, and realize resource access with the help of a unified platform.
1. Data management subsystem · data development scheduling management
Data development scheduling management can support the development scheduling function of various mashup database environments, and realize the full life cycle management of data from "collection→processing→external application services". Provide end-to-end data development quality control capabilities based on metadata and log data.
Supports data architecture management, data relationship management, data exception management, data analysis management, data process management, stream data processing, knowledge graph, data hierarchical management, data operation and maintenance management, and data asset management functions.
1. Data management subsystem · data development scheduling management
1. Data management subsystem · data sharing service
The data sharing and exchange service is based on the data directory of the sharing layer of the big data area, and carries out data sharing and exchange between various committees, offices and bureaus and their application systems.
3. Platform integration portal, sharing and operation portal
Sharing and operation portal
The portal integrates business systems such as data governance subsystem, data management subsystem, event center, and data development subsystem. Provide unified to-do work order circulation and personal message reminder functions, and realize collaborative management of business through the portal entrance. The portal provides unified rights management and adapts to each management subsystem. Provide mutually exclusive configuration of roles to realize the separation of management, use, and review of permissions. Each user unit can independently manage users and authorization. The portal implements CA authentication, unified identity authentication and single sign-on. The interface style of each system integrated in the portal is unified, including list style, page layout and page color matching, font size, button style, etc.
4. Data Opening Subsystem · Data Opening Portal
The theme library/theme library responsibility committee is responsible for: theme library requirements, construction plan (business plan), development and theme application. The big data center is mainly responsible for: formulating standards (technical solutions), providing platforms, data and tools, and being responsible for service opening, operation and maintenance after completion.
Municipal database and basic database operation support
Municipal database · theme library operation support
The space is limited and cannot be fully displayed. If you like the information, you can forward + comment, and private message for more information.