Why implement metadata management in big data centers

In the field of data management, we have been committed to letting data provide us with value, for which we put a lot of effort and investment.
In the past two decades, major companies have been building data warehouses. When we worked hard to build a data warehouse and successfully used it for online operation, we found that we would spend more time preparing data for data scientists and analysts. Analyze their data needs and provide sparkling data reports. This will take up 80% of the time investment of data maintenance personnel. This investment is relatively high, including a large amount of repetitive and unnecessary investment such as communication, reverse inspection, and correction.

How to improve data management capabilities?
We need to return to the 80% investment and do everything possible to compress them so that we can provide data services faster. At this time, there is a tool that shows its advantage, and that is "metadata."
Metadata is usually defined as "data describing data". To be more precise: Metadata is data describing processes, information, and objects. These descriptions involve technical attributes (such as structure and behavior), business definitions (including dictionaries and taxonomies), and operational characteristics (such as activity indicators and usage history).
We use "metadata" to *** this "80%". Centralized management of metadata, sorting out the metadata tree, translating, labeling, and supplementing metadata content. It is convenient for users to find data, understand data, trace the source and standardize professional knowledge. Reduce a large amount of repetitive and unnecessary investment in communication, back-checking, and correction during data preparation, so that we can leave more time for data analysis, which not only saves a lot of capital investment, but also earns more profits.

Take telecom operators as an example. Through IT construction in various periods, the company has billing systems, network systems, OA systems, accounting systems, and customer service systems. With the completion of the company's data warehouse project, the core business data of each system has been aggregated into the big data center. I thought it would greatly increase the "intelligence" of the IT system, but I didn't expect the response from the grassroots to be useless at all. Many problems stem from: lack of guidance on data, inaccurate control of business logic, and inconsistent indicators of various departments, leading to high data preparation costs.
Continuing to take telecom operators as an example, the definitions given by various departments are different for the operator’s "number of users of the day":

At the weekly meeting of the sales department, various departments are madly arguing about the "number of users of the day". The business staff who are under business pressure may not be able to persuade the other party to accept their own numbers, and have to ask the data maintenance staff to put in extra workload. To analyze the root cause of the difference, calibrate the report data.
According to the terminology of metadata technology, this problem is that in terms of business metadata, everyone's understanding of the business is not uniform. This kind of problem will cause a lot of waste of time cost:

The construction of the metadata management platform is to avoid such problems. The construction of a metadata management platform can:
realize the extraction, collection, and sorting of technical metadata, and annotate related database tables and column information. Support to view the complete data link and associated map.
Sort out business metadata, establish relevant indicators and processes in the platform, solidify and disseminate corporate expertise.
Link business metadata with technical metadata to connect business and technology, and provide more detailed guidance to business managers and technical maintenance personnel.
Taking the above example, it is possible to maintain confusing indicators such as "number of users of the day" on the metadata management platform, standardize the data source, associate it with the metadata of the technical library table, and make detailed comments. The report construction is based on the business knowledge maintained by the metadata management platform, so that the report data is rule-based, reasonable and well-founded, and unnecessary disputes are eliminated. Each department concentrates on analyzing the report data and makes full use of the value of the data.
Obviously, the metadata management platform sorts out corporate assets and regulates professional knowledge. Promoting metadata management can significantly reduce data preparation costs:

Metadata management is the basic
metadata management can sort out corporate assets, standardize professional knowledge, reduce a large amount of repetitive and unnecessary investment during data preparation, such as communication, reverse inspection, and correction, and assist in improving the efficiency of data analysis. It is the foundation of data management.
If your data processing is more complex, the supporting data needs to be better. If you take these two things into consideration at the same time, more people can directly use the data and serve themselves. In many cases, we tend to "put the cart before the horse". When we encounter problems, we only focus on how to solve them and ignore basic preparations. This not only increases complexity and repetitive work, but also spends a lot of communication and understanding costs.

Metadata management is
an additional foundation. As the underlying foundation, metadata management enables us to perform subsequent integrated operations, such as processing the cloud; do data integration in a hybrid environment; and obtain huge advantages when doing big data in a repeated environment. In fact, delivering data to enterprises faster is the focus.
In summary, for faster data analysis and business support, the ability of the IT center is essential because it is the source of "80%" investment. So what makes a successful IT center more remarkable.
Metadata management
Neusoft SaCa MetadataManagement metadata management platform software, dedicated to centralized metadata management solutions. Widely adapt to mainstream databases, ETL tools, BI tools, etc. Supports intelligent semantic retrieval, and provides functions such as data analysis, omni-directional association analysis, blood relationship/impact analysis with adjustable metrics, and metadata quality automatic analysis.

Guess you like

Origin blog.51cto.com/14925794/2536640