Hello everyone, I am Dugufeng, a former port coal worker, currently working as the person in charge of big data in a state-owned enterprise, and the manager of the big data flow of the official account.
In the last two years, because of the needs of the company and the development trend of big data, I began to learn about data governance.
Data governance requires systematic learning to truly master, and professional examination certification is also required to prove one's learning ability and knowledge mastery in data governance. If you have any questions about data governance and data governance certification CDMP, you can refer to my previous article for a detailed introduction.
Total text: 2994 words 11 pictures
Estimated reading time: 8 minutes
This document is based on the collation of learning materials related to data governance, and is collated for the study notes (mind map and knowledge points) of the data governance professional certification CDMP .
The article is long, it is recommended to read after bookmarking.
For subsequent documents, please pay attention to the big data flow of the official account , and will continue to update~
This document is part of the data architecture notes, mind maps and knowledge points. It is divided into 6 parts. Due to the display of the page, some levels cannot be fully expanded. The structure is shown in the figure below.
1. Architecture and data architecture
This part mainly learns the basic concepts of architecture and some knowledge of enterprise data architecture.
The architecture sounds lofty, but in essence it is still a high-level perspective to control the overall design and play a fundamental role.
What is the structure?
The organized design of component elements aimed at optimizing the function, performance, feasibility, cost, and user experience of an entire structure or system.
The basic structure of a system, embodied in its components, their interrelationships, and the principles governing their design and evolution.
Carry out at different scopes and levels of the organization. Responsible for defining what is difficult to understand clearly.
Data architecture is a type of architecture.
Data Architecture : Identifying the data needs of the enterprise (regardless of the data structure), designing and maintaining a master blueprint to meet those needs. Use an overall blueprint to guide data integration, control data assets, and align data investments with business strategy.
Data architecture is the foundation of data management and needs to be described at different levels to better understand and help decision-making.
Construction of data architecture : description of the current state, definition of data requirements, guidelines for data integration, and data asset management specifications.
The goal of data architecture : to establish a smooth bridge between business strategy and technology implementation , and data architecture is a part of enterprise architecture.
The main results of the data architecture : 1. Data storage and processing requirements. 2. Design structures and plans that meet current and long-term data needs.
The main work of the data architect : 1 Define the current state of the data. 2 Provides a standard business vocabulary for data and components. 3 Ensure that the data architecture is consistent with the corporate strategy and business architecture. 4 Describe the data strategy needs. 5 High-level data integration outline design. 6 Integrate the enterprise data architecture blueprint.
Overall data architecture implementation : 1 Use data architecture components (master blueprint) to define data requirements, guide data integration, manage and control data assets, and ensure that data project investment is consistent with corporate strategy. 2 Work with, learn from, and influence stakeholders involved in improving business or IT system development. 3 Build an enterprise data language through data architecture and common data vocabulary.
The context diagram of the data architecture is as follows:
Business Drivers : 1) Strategically help organizations rapidly change products, services, and data by leveraging the business advantages afforded by emerging technologies. 2) Convert business requirements into data and application requirements to ensure that effective data can be provided for business process processing. 3) Manage complex data and information and deliver it to the entire enterprise. 4) Ensure business and IT technology alignment. 5) Provide support for enterprise reform, transformation and improvement of adaptability.
For the convenience of understanding, organize the mind map of this part as follows:
2. Enterprise data architecture
In order to have a deeper understanding of enterprise architecture knowledge, you need to learn the types and frameworks of architecture.
1. Enterprise architecture type
Enterprise architecture includes business architecture, data architecture, application architecture, and technical architecture . A good architecture can help the organization understand the status of the system, speed up the improvement, and achieve the goal of compliance and efficiency improvement.
2. Framework
Architectural framework: the architecture of the architecture. Ways of thinking and understanding architecture.
The more famous one is the Zachman frame, 6X6 matrix. Inquiry, communication and redefine transformation are two dimensions. Inquiry communication:
1 What: directory column, the entity that builds the schema.
2 How HOW: Process column, indicating the activities performed.
3 where WHERE, distribution column, business location and technical location.
4 Who WHO: Responsibility columns, roles and organizations.
5 Time WHEN: Time column, representing interval, event, cycle and schedule.
6 Why WHY: motivation column, table goals, strategies and means.
3. Enterprise data architecture
Enterprise data architecture consists of two parts: 1. Enterprise data model (data structure, data specification). 2. Data flow design.
The enterprise data model includes: overall/enterprise-level/independent implementation/concept or logic/common/consistent/simplified abstraction. Data entities (such as business concepts), relationships between data entities, key business rules, and some key attributes.
Data Flow Design: Define requirements and master blueprints between databases, applications, platforms, and networks (components).
Data flow: 1 Application in business process. 2 A data store or database in an environment. 3 network segments. 4 Business roles. 5 Where local differences occur.
Data flow can be used to describe the mapping relationship of different hierarchical models: subject domain, business entity, and even the mapping relationship at the attribute level. Presented as a 2D matrix or data flow diagram.
For the convenience of understanding, organize the mind map of this part as follows:
3. Data Architecture Activities
Data Architecture Activities:
1 Establish an enterprise data architecture.
The work to be done to establish an enterprise data architecture: 1. Strategy. 2. Communication and culture. 3. Organization. 4. Working methods. 5. Results.
Data Architecture affects: 1 Defining project data requirements. 2 Review project data design. 3 Determine the impact of data traceability. 4 Data copy control. 5 Implement data architecture standards. 6 Guide data technology and update decisions.
Specific activities include:
A Assess existing data architecture specifications.
B Develop a roadmap.
Enterprise Architecture Development Roadmap : 1 High-level milestones. 2 Resources required. 3 Cost assessment. 4 Business capability workflow division. The roadmap should be guided by a data management maturity assessment.
C Manage enterprise requirements in the project.
Activities related to an enterprise data architecture project: Define the scope. Understand business needs. design. implement.
The way architecture activities are embedded into the project process: the waterfall approach. iteration method. Agile way (DevOps).
2 Integrate with other enterprise architectures.
From the subject domain to the level of refinement, connections with other structures need to be established.
For the convenience of understanding, organize the mind map of this part as follows:
4. Data Architecture Tools and Methods
Data Architecture Tools : Data modeling tools. Asset management software. Graphic design application.
A Data Architecture Approach : Lifecycle Prediction. Icon usage guidelines.
Lifetime Prediction: 1 Current. 2 of the deployment cycle. 3 of the strategy cycle. 4 retired. 5 priority. 6 Restricted. 7 Emerging. 8 audited.
Icon Usage Guidelines: 1 Clear and consistent description. 2 All chart objects match the description. 3 Clear and consistent line direction. 4 Consistent crosshatch display method. 5 consistent object properties. 6 Linear symmetry.
For the convenience of understanding, organize the mind map of this part as follows:
5. Data Architecture Implementation Guide
Data architecture includes components, activities, and behaviors. Data architecture implementation work content:
1) Establish an enterprise data architecture team and conduct problem workshops.
2) Generate a data schema version.
3) In the development project, form and establish the working method of data architecture.
4) Improve the organization's awareness of the value of data architecture work.
When implementing it, two things should be clarified.
1. Readiness assessment and risk assessment
Readiness Assessment and Risk Assessment: 1 Lack of management support. 2 Lack of evidence of success. 3 Lack of trust in managers. 4 Incorrect decisions by management. 5 Culture shock. 6 Lack of experienced project managers. 7 Single-dimensional perspective.
2. Organizational and cultural dependencies
Organizational and cultural dependencies (acceptance to dataframes depends):
1 Acceptance of the architectural approach.
2 Confirm that data is a business asset of the organization, not just an IT task.
3. The ability to abandon the partial data perspective and accept the enterprise-level data perspective.
4 Ability to integrate architectural deliverables into project implementation.
5 Regulate data governance acceptance.
6 Based on the enterprise layout, not limited to the ability of project deliverables and IT solutions.
For the convenience of understanding, organize the mind map of this part as follows:
6. Data Architecture Governance
Likewise, the data architecture also needs to be governed.
Data Architecture Governance Activities:
1) Project supervision.
2) Manage architecture design, life cycle and tooling.
3) Define the criteria.
4) Create data-related components.
Data Architecture Metrics:
1. Architecture standard acceptance rate.
2. Realize the trend.
3. Business value metrics.
Implementation trends: use/reuse/replace/discard measurements. Project execution efficiency measurement.
Business Value Metrics: 1 Business Agility Improvement. 2 Quality of Service. 3 Quality of business operations. 4 Improvement of the business environment.
For the convenience of understanding, organize the mind map of this part as follows:
To be continued~
I also organized a CDMP self-study exchange group here, only for students who want to learn data governance and students who intend to take the CDMP certification exam .
(Because more than 200 people cannot enter directly, if you need to enter, please add my WeChat invitation to enter and note CDMP )
My own self-control is too weak, so I chose to enroll in a DAMA official training class. The training class will provide a complete video explanation course, as well as teaching materials and handouts, online Q&A, exam registration and other services. Students who are interested in studying with the class can also contact me, and fans of the big data official account can contact me to apply for discounts.
Recommendation of Popular Articles on Big Data Flow
Big Data Data Governance | WeChat Exchange Group~
What exactly is CDMP - a super-comprehensive introduction to the international certification of data governance
Open Source Data Quality Solutions - Apache Griffin Getting Started
One-stop Metadata Governance Platform - Datahub Getting Started
Pre-research on data quality management tools - Griffin VS Deequ VS Great expectations VS Qualitis
Thousand-character long text - Datahub offline installation manual
Metadata Management Platform Datahub2022 Annual Review
Big data flow: big data, real-time computing, data governance, and data visualization practice self-media. Regularly publish data governance and metadata management implementation technology practice articles, and share relevant technologies and materials for data governance implementation implementation.
Provide learning exchange groups such as big data introduction, data governance, Superset, Atlas, Datahub, etc.
Big data flows, and the learning of big data technology will never stop.
Long press, identify the QR code, follow us!