Data governance is an essential part of enterprise data construction.
A good data governance system can revitalize the entire data link and maximize the protection of enterprise data.
collection
、
storage
、
calculate
and
use
Controllable and traceable process.
How to build an enterprise data governance system? What issues should be paid attention to in the enterprise data governance process? Generally speaking, one fat man cannot be eaten at a time, and the road must be walked step by step.
Below I will combine
Enterprise Data Governance
experience, detail
from 0 to 1
Build the whole process of the data governance system to help you sort out the main content of data governance and what pitfalls you will encounter in the process.
If there is anything missing, please leave a message in the comment area to discuss.
1 What exactly is data governance doing?
1.1 A short story
Before the main text, let me introduce a short story.
At the end of the year, Xiao Zhang, the financial administrator of the company, needs to make statistics on the company's financial situation. After a busy year, the boss of the company urgently needs to know the current operating status of the company.
What points does Xiao Zhang need to consider:
“
- What assets does the company currently own?
- Where do these properties come from? Where are they used?
- Are all properties used in accordance with codes and regulations?
”
Fortunately, Xiao Zhang had established a set of management standards at the beginning of the year. The entry and exit of each property is recorded and the usage is strictly controlled, and the process can be traced and reviewed.
In the end, Xiao Zhang won unanimous praise from the leaders.
1.2 What Data Governance Does
Xiao Zhang in the story supervises all financial property activities of the company to ensure the orderly and efficient use of property. This is also a similar function of the Data Governance role.
“
The core work of data governance: In the process of enterprise data construction, ensure that the enterprise's data assets are managed correctly andeffectively.
”
Generally speaking, after data is generated from the outside or inside, it is processed by big data means and then transferred to different business terminals to provide data empowerment for the upper-level applications of the enterprise.
The whole process is shown in the figure.
- Let's do some work like data synchronization to put the data into the big data system
- After the data comes in, it needs to be managed and stored, that is, to build a data warehouse with reference to modeling theory and actual scenarios
- After the steps of topic planning, dimension determination, label calculation output, etc.
- Data output to reports, application side use
The overall process data governance system will be supervised throughout. To confirm the entry and exit of the system
data quality
How about it? Is it convertible
data assets
？
data lineage
Is it traceable,
Data Security
And other issues.
“
Dirty and messy data is unusable, or even seriously buried.
”
2 Why do data governance
Some companies have a vague concept of this issue, thinking that the current data scale is small and can be controlled by humans, and there is no need for data governance for the time being.
But in actual use, there are still many problems:
- Insufficient data supervision and dirty data
- The scale of the data system is gradually increasing, and the management is chaotic
- The lineage of the data is lost, and the old and old data cannot be traced back
Regardless of the data size of the enterprise, I think it is still necessary to plan for data governance. Taking into account the issue of cost, it can be carried out in stages.
“
Why Data Governance:
- Is your data really available, what about missing and outliers?
- Where does the data come from and where does it go, and whether blood relationship information is lost
- Is data access secure, clearly identified or encrypted?
- What specifications should be referred to for new data processing, and are there standards for dimension and label management?
”
Having a sword in hand and not using it and having no sword at hand are two different things
. Doing a good job of data governance planning in advance will save subsequent transformation costs and avoid redundant process reconstruction or overthrow and start over.
Data governance can effectively ensure that the data construction process is carried out under a reasonable and efficient supervision system, and ultimately provide
high quality
、
Safety
、
Process traceability
business data.
3 Data Governance System
Enterprise data governance system includes
Data Quality Management
、
metadata management
、
master data management
、
Data Asset Management
、
Data Security
and
data standard
and so on.
1 ) Data Quality
Generally, the commonly used standards in the industry are used to measure the quality of data:
integrity
、
accuracy
、
consistency
and
timeliness
。
- Integrity: Whether the records and information of the data are complete or missing
- Accuracy: Whether the information and data recorded in the data summary are accurate, whether there are abnormalities or errors
- Consistency: Common data between multiple business data warehouses must be consistent in each data warehouse
- Timeliness: data can be output and early warning in time
2 ) Metadata management
Metadata is information about the organization of data, data domains and their relationships. Generally speaking, metadata is the data that describes data.
Metadata contains
technical metadata
and
business metadata
. It can help data analysts to clearly understand what data the enterprise has, where they are stored, and how to extract, clean, and maintain such data, that is,
data lineage
。
- Help build a business knowledge system and establish the interpretability of data business meaning
- Improve data integration and traceability capabilities, blood relationship can be maintained
- Establish a data quality audit system, classify management and monitor
3 ) Master data management
Enterprise master data refers to the consistent and shared business entities within the enterprise. In plain English, it refers to the data shared between professional companies and business systems.
Common master data such as the company's
staff
、
customer data
、
Institution Information
、
supplier information
wait. These data are authoritative and global, and can be attributed to the company's enterprise assets.
General master data management needs to follow the following points:
- Manage and supervise the access of various organizations, subsidiaries, and departments to master data, and formulate access specifications and management principles
- Regularly conduct master data evaluation to judge the completeness of the established goals
- Organize relevant personnel and institutions to unify and improve the construction of master data
- Provide technical and business process support, centralized coordination of the whole group
4 ) Data asset management
Generally, enterprises will consider data asset sorting during digital transformation. Is your data being used properly? How to generate maximum value? This is the core work that data asset management cares about.
When building enterprise assets, different angles are generally considered, namely, business angles and technical angles, and finally merged to output a unified
Data Asset Analysis
, and provide a unified data asset query service to the outside world.
How to revitalize data, form data assets, and provide a complete panoramic view of data assets can facilitate operators to control the dynamics of enterprise assets globally and macroscopically.
5 ) Data security
Data security is an essential part of enterprise data construction. Our data is stored in large and small disks, and we provide different levels of query and computing services to the outside world.
Data needs to be periodically
verification
、
Sensitive field encryption
、
access permission
control to ensure that data can be used safely.
6 ) Data Standards
To understand in plain language, we need to define a set of data specifications within the organization so that we can all understand the meaning of these data.
Today Zhang San said that this customer number is a customer who has applied for a bank card, and tomorrow Li Si said that this customer number is a customer who has borrowed money. By comparison, the field types and lengths of the two are the same. Which opinion should we adopt?
Data standards are normative constraints that ensure the consistency and accuracy of internal and external use and exchange of data.
Uniform specification
,eliminate
Ambiguity
。
4 Implementation Process of Enterprise Data Governance
4.1 Data Governance Implementation Framework
The data governance system is an organization, process, and tool established to standardize various management tasks and activities in business data specifications, data standards, data quality, and data security.
Through a normalized data governance organization, establish data
centralized management
The long-term mechanism standardizes the data management and control process, improves data quality, promotes consistent data standards, and ensures the safety of data sharing and use, thereby improving the operational efficiency and management level of enterprises.
4.2 Organizational Structure of Data Governance
In addition to the technical aspects of the enterprise data governance system
Implementation Architecture
, also requires management
Organization
support.
Generally, in the initial stage of data governance construction, the group will first establish a data governance management committee. from top to bottom by
decision-making
、
management
、
executive layer
constitute. The decision-making level makes decisions, the management level formulates plans, and the executive level implements them. Hierarchical management and unified coordination.
4.2.1 Organizational Structure
1 ) Decision-making level
Provides the decision-making function of data standard management, and the popular understanding is to make a final decision.
2 ) Management
- Review data standard management related systems
- Discuss and make decisions on cross-departmental difficult data standard management disputes
- Manage major data standard matters and submit them to the Information Technology Management Committee for review
3 ) Execution layer
- Business department: Responsible for the formulation, modification, review of business line data standards, promotion and implementation of data standards, etc.
- Technology development: undertake the implementation of governance platform, data standards, data quality, etc.; follow data standards in system design and development
- Technology operation: Responsible for the formulation of technical standards and technology promotion
4.2.2 Management Responsibilities
1 ) Project Manager
- Determine project goals, scope, and plans
- Develop project milestones
- Manage cross-project collaboration
2 ) Expert review team
Review the project plan and determine the rationality of the plan
3）PMO
- Ensure projects are executed according to plan
- Manage project material risks
- Execute cross-project collaboration and communication
- Organize Project Critical Reviews
3 ) Data Governance Task Force
Execute the implementation and operation promotion of various projects, and promote the implementation of data governance technology implementation and project progress at the executive level.
4.2.3 Responsibilities of the executive layer
Data architects, data governance experts, and business specialists form an " iron triangle " of data governance , and work closely together to promote the implementation of data governance and data architecture.
1 ) Business Specialist
As the interface person of business department data governance, business specialists
standard
、
quality
、
application
Organize business personnel in other fields to carry out work
- Define data rules
- Guarantee data quality
- Raise data requirements
2 ) Data governance experts
Data governance experts, as members of the data governance team, are responsible for designing data architecture and operating data assets; leading the organization of business and IT to achieve data governance goals.
- Build a data logic model
- Monitor data quality
- operational data assets
3 ) Data Architect
As an expert in the IT development department, the data architect undertakes the important task of implementing data standards and models, and assists in solving data quality problems.
- Data standard landing
- Logical Model Landing
- Physical model landing
4.3 Data Governance Platform
After determining the technical implementation plan and organizational management structure, the next step is to implement the data governance system.
In large enterprises, a complete
Data Governance Platform
, including all data governance functions, and providing platform services externally.
1 ) Core functions
As a product system of data governance, the data governance platform aims to ensure that the data on the data platform is safe, reliable, standard, and valuable.
- Data Asset Management
: Provide user-oriented scene search, provide panoramic data asset map, and facilitate quick asset search and asset analysis
- Data Standards Management
: Unify and customize data standards, improve management including fields, code values, and data dictionaries, and ensure unified standards for business data and middle-end data
- Data Quality Monitoring
: Provide data quality system before, during and after the event, and support data quality monitoring rule configuration, alarm management and other functions
- Data Security
: Provide data security desensitization, security classification and monitoring
- Data Modeling Center
: Unified modeling, providing business system modeling and model management
2 ) Metadata management
As the front-end display portal of the data governance platform, the metadata management system helps realize the management of data assets.
quick search
ability to improve the effectiveness and efficiency of data use.
By establishing a complete and consistent metadata management strategy, it provides centralized, unified, and standardized metadata information access, query, and call functions.
3 ) Data Quality
- Data quality monitoring: support all users to configure data quality monitoring rules
- Rule blocking: Configure data quality monitoring and blocking rules. If there is a difference in data quality, the operation of downstream jobs can be blocked in real time, and the spread of error results can be blocked.
- Warning: If there is a preset deviation in data quality, an early warning notification will be issued in time to repair it in time
4 ) Data standard
Support customization of a unified data standard platform, including field standard management, code value standard management and dictionary management, business source data and middle-end data unified standards.
5 ) Data security
Realize hierarchical data security management based on group data assets, automatically identify security information; provide data access security behavior monitoring, and identify access risks in a timely manner.
4.4 Data Governance Assessment
After the data governance platform is developed and put into operation, it is necessary to verify and evaluate the effect of the overall data governance system.
“
1 ) Whether the data can eliminate the phenomenon of " dirty, messy, and poor " 2 ) Whether the value of data assets is maximized 3 ) Whether the blood relationship of all data is complete and traceable. . .
”
1 ) Data assets
By building a data asset management system, full coverage of assets is achieved, and global search and precise positioning of target assets are supported.
- Realize global search and provide scene-based retrieval services for users
- Support multiple retrieval dimensions such as labels, data maps, table names, and field names
- Support data map and result screening of source business data dictionary
- For example, support PV/UV user search and asset display, and clarify service goals
2 ) Data standard
The accumulation of new and old data standards has opened up data modeling tools, data standard libraries and root standard libraries, and implemented data standards and roots.
- Realize 100% pull-through of data standard library
- Intelligently identify data standards and references
- The client synchronously updates data standards and stems
3 ) Data security
Keep
Prior system construction
、
In-process technology management and control
、
Post-event monitoring and auditing
The principle of establishing a full-process data security management and control system.
Based on the above data security management and control system, it supports data security grading and builds a flexible data security sharing process.
4 ) Data Quality
Through the data quality radar chart, the data and task quality are scored regularly, and the data quality effect is comprehensively inspected.
- Data Integrity: Check whether the data item information is comprehensive and complete without missing
- Alert response levels: day-to-day management, emergency response, impact mitigation; avoiding data corruption and loss
- Monitor coverage: ensure data adheres to uniform data standards and specification requirements
- Job stability: monitor job stability, whether there are problems such as job exceptions
- Job timeliness: check whether the acquisition of data item information corresponding to the task meets the expected requirements
5 Misunderstandings of Data Governance
1 ) Does data governance need to be large and comprehensive?
“
This is a classic problem. Generally, for enterprises of different stages and sizes, the degree of implementation of data governance will vary. It is generally recommended to proceed in stages according to your own data status, to avoid blindly spreading the scale, and it can be adjusted during the process.
”
2 ) Data governance is only a technical consideration
“
As mentioned in the article, data governance is not just a matter for the technical team, but for the entire group to work together. These include various business lines and other management organizations. Without a good implementation plan and coordination mechanism, it is often half the effort.
”
3 ) Data governance can be effective in the short term
“
Data governance is a long-term process that will be adjusted simultaneously with changes in the scale of enterprise data and data warehouse planning. Some functions may be effective in the short term, but it is difficult to achieve a complete system in the short term.
”
4 ) There must be a tool platform to carry out data governance
“
As the saying goes, if a worker wants to do a good job, he must first sharpen his tools. Of course, it is better to have good tools. The premise is that there is already a mature data governance system plan and strategy. The tools and technical means are very mature in the market at present, and the theory should be paved first.
”
5 ) Data governance feels vague? I don't know the final result
“
Data governance is a long-term job that requires relevant practitioners to construct and adjust according to the current data status and management model of the enterprise. It is recommended to summarize while doing practice. Small steps and jogging are a good way.
”