Hi, Minasang, I am Zhao Zhuangshi, everyone's strange girl!
It's another wonderful Saturday morning, today we will talk about-metadata!
Metadata is also a product position and concept that has only become popular in the past two years. Because it is biased towards the back, compared to cool terms such as data analysis, data BI, user stratification, and attribution, metadata is like a white flower. lotus.
01 What is metadata?
Well, why can metadata watch everything? Because it is "meta" data.
What is "metadata"-metadata?
Zhuang Shi shed tears of reunion after seeing this word for the first time, and even wanted to laugh a little. Could it be that metadata is the name of a big man who is familiar with modernism?
Okay, Zhuangshi first explained what is called "yuan" from a humanistic perspective.
Yuan still refers to the original and original meaning. Our meta-narrative, meta-data, and meta-fiction will tell you what “meta” is.
1. “Former” + “Storytelling” = Former storytelling
A complete explanation of the narrative, that is, the narration of historical meaning, experience and knowledge
2. "Meta" + "Data" = Metadata
Data about data
3. "Meta" + "Fiction" = Metafiction
In meta-fiction, the writer consciously exposes the fictional process of the novel, and produces a separation effect, so that the recipient understands that a novel is a fiction and cannot be regarded as a reality. In this way, fiction has acquired the meaning of ontology in the novel.
Well, to put it simply, Yuan is the original framework and elements of this thing. In the context of modernist large-scale industry, all mankind has entered the "order mode", so "yuan" is needed. So there is the "metamorphism" of everything.
Yes, will everyone hear a word called "metacognition" recently. There is nothing magical. To paraphrase the concept, meta-cognition is the cognition about cognition. Maybe the “first principles” of Shanyou teacher and the “friend of time” called the beast may be “meta-cognition”?
That metadata is easy to understand, which is "data about data". With metadata, it can make our data production and use more orderly.
Data production, some people will call it "background metadata": to guide data cleaning and loading.
Data usage, some people will divide it into "front-end metadata": descriptive, help us use reports and query tools more smoothly.
For the classification of metadata, we can divide it into the following three categories:
Business metadata describes metadata from the business level.
Technical metadata Various statistical information on the technical level, including data type, length, blood lineage, data analysis results, etc.
Process metadata ETL itself execution result statistics, such as how many rows are loaded, how many rows of data are discarded, and data loading time.
02 The core concept and key elements of metadata
Not much to say, let's go to the official Alibaba Cloud ppt:
Okay, if you don’t understand, Zhuangshi also drew a picture:
The original data production-use process
Related metadata involved
03 Metadata Products
There are currently three types of metadata management tools in China.
One is the specialized tools provided by companies such as IBM and CA, such as MetaStage acquired by IBM's acquisition of Ascential, and CA's DecisionBase;
The second is like DAG's MetaCenter, open source product Pentaho Metadata, they are not dependent on a certain BI product, but a third-party metadata management tool;
Open source product Pentaho Metadata diagram
Third, integrators like Puyuan and Dianthus also have their own metadata management tools: Puyuan MetaCube, Xinju Network Metadata Management System, Dianthus MetaOne, etc.
Universal metadata-driven microservice architecture:
https://cloud.tencent.com/developer/article/1080067
https://cloud.tencent.com/developer/article/1080078
Xinju Network Metadata Management System:
http://www.shsnc.com/index.php?m=content&c=index&a=lists&catid=188
Dianthus MetaOne product picture:
Specialized metadata management tools are better compatible with their own products. Once cross-system management is involved, it is not satisfactory.
04 The function and value of metadata
If you ask me, where is the function and value of metadata? At present, the industry has carried out some practices:
1、血缘分析：向上、向下表级、字段级别的追溯数据。血缘分析可以让您轻松知道：“我正在查看的报告数据来源是什么？”、“数据经过哪些转换处理？”、“销售额”从包含税费更改为不包括税费，哪些下游字段受到了影响。血缘分析可以满足许多行业(包括医疗、金融、银行和制造业等)对所呈现数据的特殊监管及合规性要求。
2、指标一致性分析：定期分析指标定义是否和实际情况一致。大佬会上对不齐数据是何等的尴尬。。。。
3、实体关联查询：事实表与维度表的代理键自动关联。
05 Postscript
Break everything: from humanistic postmodernism to data postmodernism.
In the 1960s, anti-Western modern system philosophical trends emerged in Germany, France, and the United States, which was academically called "postmodernism". You may not know what postmodernism is. Take a picture and feel it:
Correct! It is Dali's "Eternal Memory".
Postmodernism is a spirit that uses disorder to oppose order, personal rants to oppose grand narratives, and deconstruction to oppose structure.
Griffin, one of the active postmodernists in contemporary America, said: “If the term postmodernism can be used in different ways to find common ground, it means that it refers to a wide range of emotions. , Rather than a common dogma—that is, an emotion that believes that humans can and must transcend modern times."
Behind all theories is the mainstream cognition + emotion of this era. Data is no exception. Let's take a look at the dispute between the two big bosses of the data warehouse in "Zhuangshixue Data Technology 01", we know that people are repeatedly jumping in order and speed.
Therefore, we have to consider today, what is the rationality of metadata today?
After the messy data is managed, is it necessary to do subtraction in the complexity and programming?
More data ≠ more information. How to turn data into information is something that every data person needs to think about continuously.
The private place of a data person is a big family that helps the data person grow, helping partners who are interested in data to clarify the learning direction and accurately improve their skills. Follow me and take you to explore the magical mysteries of data
1. Go back to "Data Products" and get <Interview Questions for Data Products from Big Factory>
2. Go back to "Data Center" and get <Dachang Data Center Information>
3. Go back to "Business Analysis" and get <Dachang Business Analysis Interview Questions>;
4. Go back to "make friends", join the exchange group, and get to know more data partners.