In the process of data analysis and data application development, data engineers and business personnel are often in a situation where there are hundreds of millions of massive data, but they cannot find the data they actually need, it is difficult to understand the true meaning of the data, and they cannot accept that the data is not credible.
How to break this situation, do you know?
Xiao Huang is an employee of the equipment maintenance department of the enterprise. He plans to provide maintenance services for the delivered equipment, and needs to find multiple data such as all equipment information, maintenance records, and out-of-warranty equipment information.
But where should the data be obtained from? What data is obtained? Is the data complete and correct? Faced with these inexplicable data and problems, Xiao Huang was at a loss, let alone what to do!
Scenarios like this are commonplace in all aspects of the company's daily operations, which seriously hinder the efficient development of business.
The reason is nothing more than the lack of effective management of metadata at the data governance level in the process of enterprise digital transformation .
1
What exactly is metadata?
What is metadata? It is the data that describes the data , which can be commonly understood as the "instructions" of the data . Through the metadata, the data source, data structure, context and other information can be more accurately understood, and then the data can be classified and cataloged, and finally the data can be quickly retrieved and retrieved. application.
❖Metadata example:
Various types of metadata
❖Metadata attribute:
The systematic summary of data indicators enables the systematization of horizontal and vertical indicators in a certain business domain, summarizes and clarifies information such as the caliber, dimension, and index access logic of indicators, so as to quickly and accurately obtain relevant information about indicators.
For example, the indicator system of the sales theme is as follows, which usually includes three attributes: business, technology and management:
① Business metadata: Business metadata defines the business meaning and business rules of data; eliminates data ambiguity, allows users to have a consistent business understanding of data, and provides strong support for data analysis and application;
②Technical metadata: Technical metadata clarifies the storage and structure of data, laying the foundation for application development and system integration; clarifies data relationships through technical metadata, and supports data lineage tracing and impact analysis;
③Management metadata: Management metadata defines the operational attributes of data, including management departments, management responsible persons, etc.; it is conducive to the implementation of data management responsibilities to departments and individuals, and is the basis of data security management.
❖Metadata management:
As mentioned above, we have realized the importance of metadata management for enterprises, but currently most enterprises usually face the following difficulties in metadata management:
①The metadata management of many enterprises is based on partial data governance, lack of enterprise-level metadata management, resulting in incomplete metadata information;
②Many enterprises pay more attention to the management of technical metadata in the process of metadata management, and tend to ignore business metadata and management metadata; resulting in the inability to accurately understand the business meaning of data;
③Many enterprises lack a unified metadata management platform, resulting in the sorting, definition, collection, management and maintenance of metadata, which are usually handled manually, with heavy workload and error-prone, unable to ensure the timeliness and accuracy of metadata management ,reliability.
Problems facing metadata management
2
Metadata Management Objectives
For example, compare library books to data assets: the job of metadata management is to create book catalogs, indexes, and user guides to assist readers in quickly finding books and reading data.
At present, many enterprises do not have a complete data governance plan, resulting in the lack of key information and difficulty in obtaining metadata. Therefore, an overall plan for metadata management is required to better achieve metadata acquisition and management.
Enterprise metadata management needs to target business needs, mainly including:
❖Metadata Governance: Realize the unified governance of enterprise metadata, and provide reusable data models and metadata standards for enterprise data application development.
❖Enhancement of data governance: Unified metadata management lays a good foundation for data inspection and data quality management, and realizes the improvement of data governance capabilities and quality and efficiency.
❖Data asset cataloging: Based on metadata management, data asset distribution and data relationship can be sorted out, and enterprise data asset cataloging can be quickly formed; data asset management efficiency and application capabilities can be improved.
Goals of Metadata Management
3
Main content of metadata management
The management of metadata is involved in the process of enterprise data governance and data asset transfer, including unified management of metadata corresponding to data sources, data lakes, data warehouses, data assets, application layers, and BI display layers.
After the enterprise establishes the goal of metadata management, it carries out metadata planning and formulation of metadata management strategy. It mainly includes the following contents:
Main content of metadata management
❖Basic strategy of metadata management:
① Adhere to the basic principles of ease of use, practicality, comprehension, and accuracy;
②Support the unification of metadata standards between heterogeneous systems and support interoperability;
③Facing the complex and changeable data environment, it supports the scalability of metadata management.
❖Collating metadata:
There are usually two ways to organize metadata according to different classification frameworks. In the actual metadata management process, enterprises usually need to combine the two sorting methods according to business needs to build an enterprise-level metadata map.
①Sort from the perspective of business: based on the business domain or management domain of the enterprise, progressively decompose and sort from business topics, entities, data models, etc., to form an enterprise data catalog. The metadata formed in this way is easy for business personnel to understand and understand. use;
②Sorting from a technical perspective: By analyzing and sorting out the IT systems, data tables, and data structures corresponding to the data source, an enterprise data catalog is formed; the metadata formed in this way is easy for IT technicians to understand and use.
❖Definition of metadata: standard definition of business attributes, technical attributes, and management attributes of metadata, mainly to describe the content of each attribute of metadata: such as name, purpose, storage location, historical data, update time, etc.
❖Metadata collection: Through the automatic collection capability of the metadata management platform, metadata is identified and acquired, including old systems that lack original metadata information, and unified management is realized after manually supplementing metadata.
❖Metadata management: After completing the collection and sorting of metadata scattered in various business systems, establish the mapping of technical metadata, business metadata, and management metadata to form an enterprise-level metadata map and support multi-version management.
❖Metadata application: Support metadata query, metadata report and metadata analysis, and assign corresponding metadata usage rights to relevant users through the metadata management platform.
❖Change of metadata: Information such as data sources of various business systems will change at any time. Enterprises can adopt two methods of automatic identification and proactive application for metadata changes to implement metadata changes and release new versions.
4
Metadata Management System
To establish a metadata management system based on the overall framework of data governance, enterprises need to ensure the landing and persistent operation of metadata from the aspects of organizational guarantee, operating system, business process, and management platform, so as to help enterprises manage and use metadata well.
❖Organizational Guarantee: Organizational Guarantee: Establish a three-level professional data governance organization with high-level support, middle-level management, and basic implementation to provide organizational guarantee for metadata management.
❖Operational system: Metadata management is the foundation of enterprise data governance. It is necessary to formulate supporting management systems and reward and punishment measures for daily operation management. This is one of the driving forces for the continuous advancement of metadata management.
❖Business process: Establish the whole-process management process of metadata from generation, definition, release, change maintenance, etc., to ensure the continuous and efficient promotion of metadata management.
❖Management platform: build a unified metadata management platform, realize centralized management and control of enterprise-level metadata, support metadata collection, metadata management, metadata sharing, etc., and provide technical support for metadata management.
5
metadata management platform
OceanMind Hai Ruisi metadata management platform provides the whole process management capability from metadata identification, collection, management to application.
Through metadata management, enterprises can easily grasp the overall picture of metadata, quickly grasp the source of data, understand the flow of data, and analyze the lineage, influence, and quality of metadata.
The platform provides the following core capabilities:
❖Metadata collection: Supports the collection of required metadata information from different data sources, and monitors the execution of collection tasks. The collection method supports manual collection, and also supports setting a timing strategy for automatic collection.
Creating metadata collection tasks
❖Metadata Management: Display the metadata list according to the data source name. Supports importing local metadata files and exporting metadata. Support version management of metadata, record version changes and details.
Metadata Management Details
❖Metamodel management: Supports the definition of storage formats for different metadata types. Metadata types can support systems, tables, stored procedures, scripts, interfaces, etc.
Metamodel Management
6
The management value of metadata
The great value of metadata management to the digital transformation of enterprises is mainly reflected in:
❖Establish a complete data interpretation system: solve the user's needs for business and data understanding, and tell us what data the company currently has through data resource catalogs and business metadata. The business meaning corresponding to the data and the responsible department and other information. Business personnel can quickly obtain the required data through metadata management and improve the efficiency of data application.
❖Create a whole-process data traceability foundation: Based on metadata management, we can clearly see the ins and outs of data, processing process, results, etc., and realize full-link analysis and traceability of data through blood relationship analysis and impact analysis of data. Lay a good foundation for realizing business life cycle management.
❖Improve global data governance capabilities: Through metadata management, realize the classification and unified management of data scattered in various business systems; and formulate corresponding quality management rules and business audit rules to efficiently realize data integrity and accuracy verification; improve Enterprise data governance capabilities and efficiency. It provides high-quality data guarantee for data analysis and mining, data application and business development of enterprises.
OceanMind Hai Ruisi has long been committed to helping enterprises build one-stop data governance capabilities, accelerating the process of digital transformation of enterprises, improving the quality and efficiency of healthy operations of enterprises, and being a digital transformation expert by your side!