Author: Zen and the Art of Computer Programming
1 Introduction
Data middle platform, as the name suggests, is a platform for data integration and a tool for sharing and integrating data within the enterprise. The main role of the data center is to enhance the data value between the data warehouse, data lake and data applications, establish a unified business domain data model, as well as a unified data development process, data service interface and standardization. The data middle platform can not only avoid the duplication of data construction between multiple systems, but also provide more extensive support for data analysis, artificial intelligence, etc.
The positioning of the data center product is very clear. It is a platform that integrates data collection, processing, distribution, analysis, storage and application. It serves as a bridge and transfer station to connect data from different systems and channels. Its core capabilities include data Collection, data transmission, data storage, data analysis, data visualization, data reporting and data subscription, etc. Such products have high complexity, ease of use, flexibility, elasticity, agility, real-time, security, reliability, etc. Therefore, the design, development, and operation of data center products require sufficient professional knowledge, accumulated experience, and capabilities, as well as long-term tracking and research of market demand changes, and adjustments to product functions, architecture, interfaces, and protocols based on these needs.
When designing data middle-end products, factors such as business data security, data quality, data consistency, data diversity, data availability, data feedback speed, data reliability, data development efficiency, and data query efficiency need to be considered, and strive to be To be accurate and seamless, authentic and trustworthy, continuous innovation, efficient and reliable. In order to realize the smooth implementation of data middle platform products, enterprises not only need to have a deep understanding of the characteristics of the data middle platform, but also need to be good at using new technologies such as cloud computing and big data to promote the automation and intelligence of the data middle platform and build a new Data community in finance, e-commerce, government affairs and other fields.
This book will elaborate on the product strategy and planning of the data middle platform, hoping to help readers understand how to design, deploy and operate the data middle platform, and gradually create a complete, high-performance data integration solution. By reading this book, readers can master the theory and methods of data middle platform, use actual cases to guide enterprises in planning and implementing data middle platform product strategies, and build their own "data middle platform".
2. Explanation of basic concepts and terms
2.1 Overview of data center
Data Hub is a data collection that serves all systems or applications of an enterprise. It aims to reduce the coupling between various data links, reduce repeated development, improve overall data efficiency, optimize data accuracy, and enhance data. transparency and data security. The data center consists of five levels: data collection, data transmission, data processing, data analysis, data management, and data services.
- Data collection layer: The collection end uploads the data to the middle server, and then performs preprocessing (cleaning, conversion, auditing), and then performs data quality assurance work such as data verification, data deduplication, and data verification.
- Data transmission layer: Use distributed file systems, message queues, stream computing, heterogeneous data sources, etc. to achieve centralized storage, distributed circulation and rapid access of data.
- Data processing layer: Analyze, filter, transform, complete, and reshape data to make the data more refined, orderly, and understandable.
- Data analysis layer: supports a wealth of data mining and analysis technologies such as data statistics, machine learning, and image recognition, and provides pattern mining functions such as multidimensional analysis, correlation analysis, factor analysis, cluster analysis, anomaly detection, and decision trees.
- Data management layer: The data center provides functions such as data lake, database, data precipitation, and data metadata management, which are used to store data metainformation and metadata, and provide unified data views, control permissions, and usage rules.
- Data service layer: The data center provides a unified data service interface for each system or application, including API, SDK, microservices and other forms. It also supports multiple data interaction protocols, such as HTTP, HTTPS, TCP/IP, UDP, etc. , in order to facilitate data access and data sharing in heterogeneous systems.
2.2 Data center product types
Data center products can generally be divided into two categories: central products and edge products. Central products usually bear responsibilities such as data collection, data transmission, and data processing. They usually design corresponding data models, data structures, data processes, data scheduling, data quality assurance, data backup, and operational strategies based on the company's specific business needs. Central products often have relatively complete technical frameworks. For example, the data collection end usually chooses open source components or customized services provided by cloud service providers, while the data processing end can build its own analysis, reports, dashboards, etc. based on the data lake. And provide relevant APIs for upstream system calls. One advantage of central products is that they bring together data sources from various systems to achieve full data coverage. However, the disadvantages are fixed technical frameworks, long maintenance cycles, and high resource consumption. They are suitable for small companies, small and medium-sized companies, and medium-sized companies. companies and other larger companies.
Edge products are generally only responsible for the most basic functions such as data transmission and data processing. They usually use lightweight middleware or plug-ins to provide services to the outside world, and select appropriate transmission protocols and codecs based on the company's actual business scenarios. Such as FTP, SFTP, MQTT, AMQP, etc., but for complex business logic and data analysis requirements, some performance may be sacrificed to improve operating efficiency. Edge products often only undertake the functions of data transmission and data processing, but do not have too many data mining, calculation, and analysis capabilities. They are suitable for companies with a certain scale and do not want to introduce an overly complex technology stack. company.
2.3 Data center architecture model
The data center architecture model is divided into four types: data warehouse type, data lake type, data intelligence type, and data pipeline type. The corresponding architectures are as follows.
- Data warehouse-type data middle-end architecture
Data warehouse-type data middle-end architecture is designed for large, complex, and large-volume enterprise-level organizations. It is divided into various stages of data warehouse. Generally speaking, it is divided into the following stages:
- Raw data collection layer: Enterprise raw data collection. This stage is mainly to obtain raw data generated by various departments of the enterprise.
- Data conversion layer: This stage mainly converts, reorganizes, expands, standardizes, etc. the data generated by various departments of the enterprise to form a universal structured data model.
- Data integration layer: This stage mainly integrates data from different systems, different databases, different files, etc. according to a unified data model.
- Metadata management: This stage is mainly to assign unified metadata definitions and rules to the data to provide support for data analysis and decision-making.
- Data governance layer: This stage mainly manages the life cycle of data, ensures data quality, and improves data value.
- Data analysis layer: This stage is mainly based on the multi-dimensional analysis, correlation analysis, factor analysis, cluster analysis and other pattern mining technologies built into the data warehouse to conduct data analysis and generate visual reports, result notifications, etc.
Although the data warehouse-type data center architecture is very complex, its advantage is that it can achieve true data integration and analysis, comprehensively support the entire enterprise's data, thereby promoting the enterprise's core competitiveness. On the other hand, due to its inherent data scalability and fault tolerance, it can cope with complex business scenarios and achieve highly integrated and reusable goals. However, there are also some problems, such as complex architectural technology framework, high data threshold, and high resource consumption.
- Data Lake Data Middle Platform Architecture
The data lake data middle platform architecture was proposed at the 2016 International Data Lake Conference. It is a data middle platform architecture model that is more suitable for big data application scenarios. The characteristic of this architectural model is that based on the original technology stack, a data lake cluster is added, and through data sharing between the data lake cluster and the central cluster, data from different systems, different networks, and different protocols can be centrally managed and analyzed.
- Central cluster: The central cluster refers to a structured data set that stores original data and undergoes data conversion.
- Data lake cluster: Data lake cluster refers to the real storage cluster of data, which is the place where data analysis and mining analysis work are truly carried out. The data lake cluster is a place where massive data is stored, and real-time analysis can be performed in this area.
- Metadata management layer: The metadata management layer mainly provides unified metadata definitions and rules for the data in the data lake to provide support for data analysis and decision-making.
- Data governance layer: The data governance layer mainly manages and governs the original data and structured data stored in the data lake to ensure the validity, stability, availability, and security of the data.
Compared with the data lake-type data middle-end architecture and the data warehouse-type data middle-end architecture, its technical framework, architectural topology and data integration model are very different. The data lake-type data center architecture is mainly based on the storage, retrieval, analysis, display, and output of massive data. It uses cloud computing, big data and other technical means to centralize and distribute data to achieve fast, efficient, and efficient data management. Intelligent analysis. The advantage of the data lake-type data middle-end architecture is that it can truly realize the storage, analysis, processing and sharing of massive data, making up for the shortcomings of the central data middle-end architecture. However, there are also some problems, such as the technology stack limitations of the central cluster and data lake cluster, and the high cost of metadata management and governance.
- Data-intelligent data middle-end architecture
Data-intelligent data middle-end architecture refers to taking artificial intelligence technology as the core and combining data collection, transmission, storage, analysis and other technical means to achieve automated intelligence on data through the training and deployment of AI models. Analysis and prediction, thereby providing users with efficient and convenient decision support.
- AI model training and deployment: Enterprises build AI models and conduct model training and deployment through customized or open source components.
- Data collection layer: The data collection layer is responsible for external access to data, data cleaning, conversion, etc.
- Data transmission layer: The data transmission layer is responsible for transmitting data in a unified format.
- Data storage layer: The data storage layer stores different data into the data lake.
- Data analysis layer: The data analysis layer analyzes the raw data stored in the data lake.
- Data report layer: The data report layer generates reports from data analysis results.
- Data application layer: The data application layer provides data analysis results to end users.
The data-intelligent data middle-end architecture is still in its infancy. It lacks mature products and has limited technology accumulation. It is difficult to completely replace the central data middle-end architecture.
- Data pipeline type data middle platform architecture
Data pipeline type data middle platform architecture is a variant of edge data middle platform architecture, which mainly realizes the rapid transmission of data through technical means such as pipelines and agents.
- Data collection layer: The data collection layer is responsible for external access to data, data cleaning, conversion, etc.
- Data transmission layer: The data transmission layer is responsible for transmitting data in a unified format.
- Data processing layer: The data processing layer performs data processing on the received raw data.
- Data subscription layer: The data subscription layer subscribes to data from other systems.
- Data service layer: The data service layer provides data service interfaces for other systems.
The data pipeline-type data center architecture is just in its infancy, and we have not yet seen large-scale companies adopt this architecture model. But it also shows that there is still a lot of room for optimization in the edge data center architecture.
2.4 Data center product strategy
The product strategy of data middle platform is how to design and deploy data middle platform products, which involves three main aspects: data middle platform architecture, data collection, and data transmission.
- Data middle platform architecture design
The data middle platform architecture design includes not only the design of a series of links such as data collection, processing, and transmission in various aspects such as data warehouse, data lake, and data intelligence, but also includes data integration, data distribution, data services, etc. Overall system architecture design. Data center architecture design involves many aspects such as business understanding, technology selection, plan formulation, implementation and evaluation. - Data collection design
Data collection design includes the design of data collection, storage, processing, transmission and other processes. Collection design is carried out according to the company's different business scenarios and different data collection configuration parameters are designed, which is conducive to the accuracy, completeness and timeliness of data. . - Data transmission design
Data transmission design includes the design of data storage, query, analysis, display, output and other processes. It is designed according to the transmission protocol and network transmission rate of the data center, which is conducive to data efficiency and low latency. , safety and reliability.
2.5 Data center operation strategy
The data center operation strategy includes the deployment, operation and maintenance, management, and monitoring of data center products.
- Data middle platform product deployment
The deployment of data middle platform products mainly refers to the deployment of all aspects of data middle platform products, including data collection end, data transmission end, data processing end, data analysis end, data management end, data server end, etc. Different data center products can be installed on different hosts to achieve distributed deployment. - Data middle platform operation and maintenance
Data middle platform operation and maintenance mainly refers to all aspects of data middle platform operation and maintenance, including logging, monitoring and alarming, data disaster recovery, fault recovery, etc. The operation and maintenance personnel of the data center can formulate corresponding operation and maintenance plans based on the characteristics of the data center products to ensure the normal operation of the data center products. - Data middle platform management
Data middle platform management mainly refers to the management tools and methods of data middle platform, including data analysis, data quality, metadata management, data operations, etc. The management tools of the data center can be used to perform functions such as data analysis, data quality, metadata management, and data operations. - Data middle platform monitoring
Data middle platform monitoring mainly refers to the operating status monitoring of all aspects of data middle platform products, including real-time data collection, transmission, storage, processing, analysis, output, etc. The monitoring tools of the data center can conduct real-time data monitoring of various data center products.
3. Core algorithm principles, specific operating steps and mathematical formulas
One of the most important parts of a data center product is data analysis. Here, we use simple data analysis functions to explain the principles and specific operation steps of data center products.
3.1 Core algorithms for data analysis
There are four main types of data analysis algorithms:
- Classification algorithms: including Bayesian, decision tree, naive Bayes, K nearest neighbor, random forest, support vector machine, etc.
- Regression algorithms: including linear regression, polynomial regression, smooth regression, ridge regression, chi-square regression, etc.
- Clustering algorithms: including k-means, hierarchical clustering, DBSCAN, agglomerative clustering, etc.
- Probability density algorithm: including Gaussian kernel function, Laplacian pyramid, maximum entropy model, local density range model, etc.
3.2 Data analysis steps
The core function of data center products is data analysis. However, due to the characteristics of various fields and the ever-changing structure of data, we cannot generalize. We can only choose appropriate algorithms for analysis based on our own data conditions. Here are some common steps:
- Data preparation: loading data, cleaning data, transforming data, matching data, merging data, normalizing data, etc.
- Data filtering: Select data based on specific criteria.
- Data preprocessing: Processing of data, including normalization, standardization, outlier processing, missing value filling, etc.
- Data exploration: Visualize and explore data through statistical charts or numerical descriptions to find patterns and patterns in the data.
- Model fitting: Select an appropriate algorithm model and train the model through training data to obtain model parameters.
- Model evaluation: Evaluate the model, evaluate the quality of the model, and select a good model to continue using.
- Model prediction: In the test set or production environment, input the data to be predicted, and obtain the prediction results after model processing.
- Model publishing: Convert the model into an API interface for calls by other systems to provide model results.
4. Specific code examples and explanations
The details of some algorithms are more complicated, so you need to use codes to explain them. The following provides you with some code examples and explanations of algorithm implementations commonly used in data center products.
4.1 K-means clustering algorithm
The K-means algorithm is one of the simplest clustering algorithms. Its basic idea is to allocate each data point to the cluster where the nearest mean point is located through iterative solution until the center position of each cluster no longer moves.
import numpy as np
def k_means(data, K):
N = data.shape[0]
C = np.zeros((N, K)) # 初始化K个中心点
labels = np.zeros(N) # 初始化每个样本对应的簇标签
for i in range(K):
C[:,i] = np.mean(data[np.random.choice(N, size=1)], axis=0) # 随机选择K个中心点
while True:
dist = ((data[:, None]-C)**2).sum(-1) # 计算每个样本到每个中心点的距离
new_labels = dist.argmin(axis=-1)
if (new_labels==labels).all():
break;
for i in range(K):
C[:,i] = np.mean(data[new_labels==i], axis=0) # 更新中心点
labels = new_labels
return C, labels
4.1.1 Parameter analysis
data: Sample matrix, each row corresponds to a sample, and the number of columns indicates the number of features;
K: The number of clusters.
4.1.2 Return value analysis
The function returns two values, which are:
C: clustering center matrix, each row corresponds to a clustering center, and the number of columns is equal to the number of clusters;
labels: cluster label corresponding to each sample.
4.2 DBSCAN clustering algorithm
The DBSCAN algorithm is the abbreviation of Density-Based Spatial Clustering of Applications with Noise. This algorithm uses the idea of density connectivity to determine whether it belongs to the same cluster by scanning the density of surrounding neighborhood samples.
from collections import deque
def dbscan(data, eps, minPts):
visited = set()
result = []
n_cluster = 0
for idx in range(len(data)):
if idx not in visited:
queue = deque([idx])
cluster = []
while len(queue)!=0:
point = queue.popleft()
cluster.append(point)
visited.add(point)
neighbors = [neighbor for neighbor in get_neighbors(point, data)]
for neighbor in neighbors:
if neighbor not in visited and distance(data[point], data[neighbor])<=eps:
queue.append(neighbor)
if len(cluster)>minPts:
result.append(cluster)
n_cluster += 1
print("Number of clusters:", n_cluster)
return result
def get_neighbors(point, points):
distances = [(pidx, euclidean_distance(points[pidx], points[point])) for pidx in range(len(points))]
sorted_distances = sorted(distances, key=lambda x: x[-1])[:5] # 取最近的5个邻居
return [sorted_dist[0] for sorted_dist in sorted_distances]
def euclidean_distance(pointA, pointB):
return sum([(a-b)**2 for a, b in zip(pointA, pointB)])**0.5
4.2.1 Parameter analysis
data: Sample matrix, each row corresponds to a sample, and the number of columns indicates the number of features;
eps: Specify the radius size. When the distance between the sample point and any two sample points is less than or equal to the radius, they are considered to be neighbors; : Specify the
minPtscore point the minimum quantity.
4.2.2 Return value analysis
The function returns a value. The type of this value is list, where the elements are all lists, and each list represents a cluster.
The elements in each cluster are integers representing the index numbers of the corresponding points.
4.3 Probability density algorithm
Probability density algorithms include Gaussian kernel function, Laplacian pyramid, maximum entropy model, local density range model, etc. Among them, the Gaussian kernel function is the most commonly used method. It assumes that the data obeys the normal distribution and obtains the local density curve through fitting.
import numpy as np
import scipy.stats
def gaussian_kernel(X, Y, sigma):
return np.exp((-1/(2*sigma**2))*euclidean_distance(X,Y))
def laplacian_pyramid(X, Y, scales, truncate):
pyramids = []
for scale in scales:
dists = [[gaussian_kernel(x,y,scale*truncate), gaussian_kernel(x+scale*truncate, y, truncate),
gaussian_kernel(x-scale*truncate, y, truncate)] for x,y in zip(X, Y)]
marginalized = normalize(np.hstack(dists)).reshape(-1,1)
pyramids.append(marginalized)
return pyramids
def maxent_model(samples, n_components):
X = np.array(samples)
M = local_maxima(X)
phi = compute_phi(M, X)
w = softmax(-phi, axis=0)
mean = np.dot(w, X) / np.sum(w)
cov = np.cov(X.T, ddof=0, aweights=(w/np.linalg.norm(w)))
model = {
'phi': phi, 'w': w,'mean': mean, 'cov': cov}
return model['mean'], model['cov']
def local_maxima(X, kernel='gaussian', bandwidth=None):
"""Compute the indices of all local maxima"""
if kernel=='gaussian' and not isinstance(bandwidth, float):
raise ValueError('For Gaussian kernel, need to specify the bandwidth')
if kernel=='gaussian':
D = np.sqrt(laplacian_pyramid(X[:,0], X[:,1], [0.1, 0.5, 1], bandwidth))
L = np.prod(D, axis=0)*np.ones(X.shape[0])
elif kernel=='uniform':
pass
peaks = find_peaks(L)[0].tolist()
return peaks
def find_peaks(arr):
"""Find the local maxima of an array."""
diff = arr[1:] > arr[:-1]
rising_edges = np.where(diff & ~arr[1:-1])[0] + 1
falling_edges = np.where(~diff & arr[1:-1])[0] + 1
left_bases = rising_edges - np.arange(1, len(rising_edges)+1)
right_bases = falling_edges + np.arange(1, len(falling_edges)+1)
bases = list(zip(left_bases, right_bases))
heights = [max(arr[left:right+1]) for left, right in bases]
return np.asarray(heights), np.asarray(bases)
def normalize(v):
norm = np.linalg.norm(v, ord=1)
if norm == 0:
return v
return v / norm
def compute_phi(M, samples):
num_clusters = len(M)
dim = samples.shape[1]
phi = np.zeros((num_clusters, dim, dim))
for j in range(dim):
for l in range(j, dim):
marginals = [samples[(M[i])] for i in range(num_clusters)]
phi[:, j, l] = [scipy.stats.multivariate_normal.pdf(sample, mean=marginals[i][:,j],
cov=marginals[i][:,:l]+marginals[i][l+1:,j:]) for i in range(num_clusters)]
phi[:, l, j] = phi[:, j, l]
return phi
def softmax(z, axis=None):
s = np.exp(z - z.max())
return s / s.sum(axis=axis, keepdims=True)
4.3.1 Parameter analysis
X: Sample matrix, each row corresponds to a sample, and the number of columns indicates the number of features;
scales: Pyramid scale list, where the first scale is 1, the second scale is 0.5, and the third scale is 0.1; :
truncateLaplacian pyramid Truncation coefficient;
n_components: the number of hidden variables of the maximum entropy model.
4.3.2 Return value analysis
The function returns two values, which are:
mean: the mean of the maximum entropy model;
cov: the covariance matrix of the maximum entropy model.