The source of this information is open to the Internet, for personal study only, please do not commercialize it, if there is any infringement, please contact to delete
1.1 Project construction content
For the construction of this district-level big data resource platform, in accordance with the principle of "one-step overall planning, step-by-step implementation of platform construction, and gradual upgrading of data empowerment", the requirements for data exchange, sharing and collaborative applications will be implemented, and the district-level big data resources will be gradually improved. Resource center energy level. In the initial stage, the construction of the data resource platform will be completed first, and then relying on the construction results of the basic platform, the construction of big data resources, operation platform, management mechanism, operation and maintenance support, organizational security system and typical smart applications in XX District will be gradually improved and enriched.
The main goal of the construction of the district-level big data resource platform project is to build a unified big data resource management platform in XX District and realize the basic functional structure, introduce relevant management methods, and support the data sharing needs of various government departments in the district; establish a big data resource center , Complete the construction of a standardized data warehouse, build a basic and integrated database, and realize data empowerment for the region's government affairs processing, economic development, social people's livelihood, urban management, and rural revitalization.
l Construct a unified big data resource management and application platform: build big data with core functions including data support base, data exchange sub-platform, data sorting and management sub-platform, data service sub-platform, data analysis support sub-platform and data resource portal The resource platform realizes the comprehensive management and service functions of big data;
l Start the construction of district-level big data resource center: complete the design and planning construction of the standardized data warehouse, and complete the docking with the business and management systems of the district committees and offices, the big data resource platform of S City and the Internet of Things related platform data, and gather all Class business data, build a district-level big data resource center;
l Based on the data lake of the XX District Resource Center, build a basic database, including a comprehensive population database, a comprehensive legal person database, an electronic certificate database, and a geographic space database, to provide information related to population, legal persons, certificates, and geographic space for all government departments in the district data service;
l On the basis of the data lake and basic database of the district-level resource center, continue to deepen the construction and application exploration of big data resources, and basically complete the collection of resources supporting themes such as the one-network office, one-network unified management, urban planning, and emergency command in the whole district , and form related thematic libraries and theme libraries;
l Establish a unified catalog of government data information resources, establish a dynamic update mechanism, clarify the public data that all departments in the region can share with other government agencies, establish a unified data exchange and sharing platform and data center in the region, and realize vertical integration from top to bottom Sharing, horizontally open up data exchange and sharing among various departments, and realize inter-departmental business coordination and command linkage;
l Enrich the existing management functions and form a shared work system and system.
1.1.1 Construction of data exchange sub-platform
Build a unified data exchange sub-platform in the whole region to form a core hub for collection, exchange and sharing of public data; open up data resource sharing channels among units in the region to realize two-way data exchange. Specifically, in this district-level big data resource platform construction project, the construction content of the data exchange sub-platform includes:
Ø Data exchange configuration: Provide data exchange configuration function, can provide exchange policy configuration for exchange period and time, interface configuration exchange task configuration function;
Ø Data service management: Provide data service management function, register the cleaned data and the data after scene analysis as data service, and conduct unified management and external sharing;
Ø Data sharing interface: register the cleaned data and the data after scenario analysis into the service management system, share them externally, and conduct unified management and audit on the sharing behavior;
Ø Data exchange monitoring: Provide data exchange monitoring function, real-time monitoring of core service operation status during the exchange process, including the overall operation status of the exchange platform, exchange domain operation status, exchange node operation status, exchange process operation status, etc., to ensure the stability of data exchange.
1.1.2 Build a data governance sub-platform
Build a big data governance sub-platform to provide data standard management, metadata management, and data quality management capabilities to achieve standardized governance and management of data; provide data factory capabilities to clean and process collected data and support business data Application requirements. Specifically, in this district-level big data resource platform construction project, the construction content of the data governance sub-platform includes:
Ø Resource catalog management: Provide resource catalog management functions, including: resource catalog, catalog submission, catalog review, catalog release, catalog summary, catalog query, etc.;
Ø Data standard management: provide data standard management functions, including: dictionary standard management, naming standard management, dimension standard management and data item standard management;
Ø Metadata management: provide metadata management functions, including: metadata forward collection, metadata maintenance, metadata query, metadata import/export, and metadata blood relationship analysis;
Ø Data quality management: Provide data quality management functions, including: quality model configuration, quality rule management, scheme configuration scheduling, quality inspection result viewing, quality inspection analysis report;
Ø Data development platform: Provide data development platform capabilities, support two development modes of visual development and original ecological development, and realize mutual conversion between original ecological development and visual arrangement;
Ø Unified scheduling management: Provide unified scheduling management functions, provide process design and management, scheduling strategy control, task scheduling control, and other functional modules.
1.1.3 Build data resource portal
Build a unified data resource portal, effectively integrate and integrate the application function modules of each sub-platform, data resource directory integration, unified user account, organizational structure, role authority, business process, information display, and unified operation and maintenance management, etc. An integrated and unified data management and application portal for unit staff.
Use the data resource portal to promote the interaction and open use of public data and social data in the whole region, strengthen the interaction of government and enterprise data, accelerate the unified collection of public data, strengthen data management and governance, explore data applications, and promote data sharing, collaboration and openness, in order to improve the overall The governance capacity of the district government will be improved to provide support for the transition from management to service. Specifically, in this district-level big data resource platform construction project, the construction content of the data resource portal includes:
Ø Portal homepage: Provide the portal homepage, where you can see the number of shared directories, the number of shared resources, recommended directories and interfaces, and information resources;
Ø Unified user management: provide unified user management functions, realize unified organizational structure and personnel management, and divide and set user attributes and permissions;
Ø Unified authentication management: provide unified management of user identities and unified system user identity authentication services;
Ø Integrated resource management: provide unified data resource integration services, including: resource retrieval, resource query, resource display, resource application, etc.;
Ø Unified approval management: Provide unified process approval management functions, including: process initiation, process transfer, process approval, etc.;
Ø Unified message management: Provide unified message management functions, including: message type management, message channel management, general rule management, custom rule management, message expiration, message sending;
Ø Unified log management: provide a variety of log recording methods, and support log reminders, log analysis, and log audit management;
Ø Data asset management: Provide data asset management functions, including: data asset classification management, asset catalog, asset map, asset version management, asset statistics management;
Ø Resource directory service: Provide resource directory service function, realize external sharing of resource directory, and provide directory query, directory statistics and other services;
Ø Unified monitoring and operation and maintenance: Provide unified monitoring and operation and maintenance functions to ensure the stable and efficient operation of the entire data resource platform, and improve the guarantee execution and operation and maintenance efficiency of the entire life cycle of data resources. Specific features include:
l Provide monitoring resource management functions, support basic maintenance operations of resource model addition, deletion and modification, create resource objects that need to be managed, and centrally manage all supervised objects;
l Provide the basic resource monitoring function to monitor the basic implementation of the server, storage, operating system, middleware, etc. involved in the entire platform;
l Provide the application system monitoring function to monitor the operation of each application service on the platform, detect faults and risks in time, and give an alarm;
l Provide monitoring and alarm management functions, provide alarm rule management functions, and realize custom management and maintenance of alarm rules;
l Provide platform operation and maintenance management functions, responsible for centralized recording of faults or files submitted by users and automatically discovered by the system, and can assign faults to different personnel for processing and track and manage the fault handling process and results.
1.2 Overall construction plan
1.2.1 The overall positioning of the center
The core of the construction of the district-level big data resource platform is to gather all kinds of public data in the whole district, and establish a data resource library and data services for government consultants, economic development, urban management, social public services and rural revitalization. Providing comprehensive, efficient, safe, and intelligent data management and application support for smart cities is an important part of the overall strategy and implementation plan for smart city construction.
1.2.2 Overall solution
1.2.2.1 The overall blueprint of the platform
For the construction of a district-level big data resource platform, we will build a big data management and application platform for XX District based on the current mainstream big data technology framework. Using the data collection function of the data platform, the government affairs data of government units at all levels in the XX district, the surveillance video related to urban management and other Internet of Things data, the data of enterprises and institutions, and the Internet and other social data are gathered to form a unified XX district-level big data resource pool.
After that, using the data governance and modeling development capabilities of the platform, the collected data can be effectively classified and integrated to build comprehensive population databases, industrial economic theme databases, tourism databases, vegetable garden flower and bird databases, and rural industrial economic revitalization databases. and other data application services, and use the unified portal to open relevant data applications to the outside world, and provide data services to users at all levels, enterprises and institutions in XX District, thereby building a data resource center in XX District, and realizing the access to government affairs and other data in the whole district Centralized collection, unified management and shared utilization provide data support for the region's government affairs processing and economic development. Specifically, the solution blueprint for this project is as follows:
1.2.2.2 Overall architecture of the platform
The district-level big data resource platform platform is based on infrastructure (network & technology & storage), technical standard system specification and information security system construction data support base, data exchange, data resource center (basic and fusion database), data sorting and governance, data Service, data analysis, data sharing application and data resource portal module, and realize the sharing and exchange with the big data resource platform of S city.
1.2.2.3 Platform architecture relationship
Architectural relationship description:
ü Build a data governance production line based on the data governance platform: various committees, bureaus sort out resources, form a catalog list and formulate data governance-related specifications, and build a data governance production line based on the catalog and specifications;
ü Data sharing and exchange platform: the data governance/service/analysis platform sets data collection and scheduling tasks, and collects data through data sources to the data lake (data resource center);
ü Data source: each commissioner sets the front-end processor to place the source data;
ü Data resource center: store the data collected through the data sharing and exchange platform;
ü Data basic support platform: data governance/service/analysis platform reads, processes and stores data through the data basic support platform;
ü Data resource portal: After governance, the resources are displayed on the portal, and each committee and office are provided with browsing and application portals. At the same time, if the applied data needs to be cleaned/converted, the data governance platform will carry out the process according to governance specifications and production lines Data processing, and synchronizing the processed valid data to the portal;
ü Data sharing and exchange platform: The data that users apply for through the portal will be shared through the data sharing and exchange platform.
ü The unified maintenance platform and data security management will escort the platform construction of the entire digital district-level big data resource platform.
1.2.3 Project implementation plan
In this project, it is first necessary to build a set of big data resource management and application platform, and based on the big data resource management and application platform, use various functions in the platform to gather data, data governance, data development and data services and other means , and gradually build various district-level data resources (including: data lakes, basic databases, theme databases, theme databases, etc.).
Specifically, the overall construction implementation plan of the project is as follows:
2 Resource platform construction plan
2.1 Overall architecture of the platform
The overall structure of the district-level big data resource platform consists of support system (standard specification support system, management operation and maintenance support system, security support system), network system, information resource platform software and hardware system environment, platform application system (data support base, data exchange Sub-platform, data collation and management sub-platform, data resource portal), and data resource center (data lake, basic library, theme library, thematic library). The specific platform architecture is as follows:
2.2 Construction of data exchange sub-platform
Build a unified data exchange sub-platform in the whole region to form a core hub for collection, exchange and sharing of public data; open up data resource sharing channels among units in the region to realize two-way data exchange.
2.2.1 Overview of construction plan
Through the construction of the data sharing and exchange platform of this project, a data sharing and exchange system in XX District will be built to promote the sharing and utilization of government affairs data across levels, regions, systems, departments, and businesses, and provide the underlying infrastructure for public basic libraries and business theme libraries. data support.
The data exchange system in XX District is based on the e-government cloud, which provides a unified platform for cloud data sharing and exchange services for the data management departments of the whole district, and can realize stable and reliable two-way information between the front-end nodes of the department and the central node of the government affairs cloud, and between departments transfer.
2.2.1.1 Architecture Design
In view of the complexity of heterogeneous platforms for data source applications and access to heterogeneous systems, the construction of data sharing and exchange platforms should unify the technical architecture and access standards (including technical specifications, access specifications, management specifications, and operation specifications, etc.), At the same time, it is necessary to fully consider not only the security and scalability of the system, but also the real-time, reliability and efficiency of data exchange, so as to realize the business function goals of data collection, synchronization, screening and comparison of each system.
Therefore, the construction of this data sharing and exchange platform will make full use of informatization means, adopt current mainstream technical standards, mainly including database technology, data integration technology, message middleware, SOA architecture and other technologies, and adopt componentized and loosely coupled design Ideas to achieve a unified platform architecture, unified technical standards, and unified data interfaces. The construction of the data exchange sub-platform is carried out around the collection and exchange of data resources, data transmission, and exchange monitoring processes to ensure that the platform functions and services are closer to actual business application needs.
Specifically, the overall business and logical architecture design of the data exchange sub-platform in this district-level big data resource platform is as follows:
Use the data exchange platform to interconnect with various government departments, and collect the data resources of various committees and offices through the data transmission system into the data center; conduct unified processing and processing of the data collected in the data center; The data in various basic information databases are exchanged into various basic information databases, and the service management system performs service packaging and service publishing on the data in various basic information databases, and provides them to various committees and decision-making analysis systems.
In short, through the construction and implementation of the data exchange platform, a complete data sharing and exchange system will be built for XX District, and data sharing and exchange service support will be provided for the overall system construction of this project.
2.2.2 Data exchange configuration
Obtain the update frequency of information resources defined in information resources, automatically realize the collection, can configure full or incremental collection methods, and can configure the storage location of the data provider department in the front-end warehouse of the data center department, etc.
It can provide collection policy configuration, interface configuration and collection task configuration functions for the collection period and time.
For application data collection, the key business points that need to be managed include collection points and data transmission. The main performance indicators include: the number of files successfully obtained within the sampling period, the data collection interval, the size of the collected files, the process status, the continuity of the files, the collection status, the transmission rate, etc.
Obtain the update frequency of information resources defined in information resources, automatically realize the collection, can configure full or incremental collection methods, and can configure the storage location of the data provider department in the front-end warehouse of the data center department, etc.
2.2.3 Data exchange front
(1) Flexible data exchange requirements
The system needs to meet the data resource exchange needs of various departments and provide independent and controllable data exchange services:
In the matching mode of supply and demand relationship, one-to-one, one-to-many, many-to-one, many-to-many data exchange matching modes;
In the exchange direction, it provides support for one-way and two-way data exchange methods;
In terms of database data extraction, it supports full extraction and incremental data extraction, and incremental extraction supports timestamps, triggers, and data extraction methods based on log analysis;
In the operation mode, it supports automatic operation and manual control operation mode;
In the interactive mode of the application system connecting the exchange system and the department, it supports active exchange and passive exchange;
In terms of exchange timeliness, it supports modes such as real-time exchange, timing exchange, and periodic exchange;
(2) Heterogeneous Platform Compatibility Requirements
As a basic software facility, there are many departments connected to the data exchange system, and the system operating environment of each relevant unit is different. Therefore, the exchange system needs to support various heterogeneous platform environments, including: system platform, database environment, etc. The switching system should support running on various mainstream operating system platforms, including: Linux, Windows, Unix and other operating systems, and support physical machine deployment and virtual machine deployment. For different data storage systems, it should support various common database management system, and can provide good compatibility with domestic database systems.
2.2.4 Data sharing service
Only when data resources are effectively integrated, developed, and applied can they exert their corresponding value. Through platform construction and service-oriented processing of data resources, it is possible to better carry out relevant business applications in the fields of leadership quantitative decision-making, social and people's livelihood services, and provide more timely and effective data resource sharing application services.
Therefore, in this project, a data service sub-platform will be built. Through a unified data service platform, various data resources in the platform will be packaged into standard interfaces and shared externally in the form of services, so as to realize unified access to data resources, use applications, Shared scheduling, management and monitoring. Through a unified data service platform, unified access, scheduling, management and monitoring of service resources are realized.
The data service management platform is supported by the service bus to build an efficient and stable unified application integration and data service system. On the one hand, the existing business system functions can be connected to the unified platform to provide external services as shared services, and the shared data resources can also be shared It is shared externally through standard services, and at the same time provides shared security protection strategies.
The space is limited and cannot be fully displayed. If you like the information, you can forward + comment, and private message for more information.