With the rapid advancement of hardware technology, especially the rapid improvement of the performance of network and storage devices, and cloud computing vendors promoting cloud storage services with software and hardware synergistic acceleration, more and more enterprises have begun to build data storage services based on cloud storage, or For data lakes, it is necessary to build an independent computing layer to provide data analysis services, which is the Disaggregated Storage and Compute Architecture (Disaggregated Storage and Compute Architecture). This article introduces the storage-computing separation architecture.
— Background —
Apache Hadoop has started the wave of distributed storage. The architecture it adopts is a "storage-computing integration" architecture, that is, computing and storage functions are implemented in a cluster, and in order to minimize the performance loss caused by the horizontal network, the computing engine is designed It adopts the design of "computing close to storage", that is, each computing task will choose to run on the server where the corresponding data file is located, so that the performance of local IO can be exerted, and the network single-point bottleneck caused by a large number of point-to-point data transmission can be avoided. question. The diagram below depicts this design, with storage and compute services on each worker node.
From an abstract point of view, the storage-computing separation architecture is shown in the figure below. Its storage layer and computing layer are relatively independent. The storage layer uses HDFS or other Hadoop-compatible storage (HCFS) or even a relational database, while the computing layer generally adopts diversified Computing engines such as Spark, Presto, Flink, etc. The benefits of this architecture are mainly in the following three aspects:
- It is more convenient to provide data analysis services for different businesses, connect with different computing engines, and avoid the problem of repeated storage of popular data in different businesses.
- The computing layer and the storage layer can be independently expanded and contracted according to their respective business needs. Generally, the growth rate of computing resources should be significantly faster than that of storage. This method can reduce the cost of storage.
- Computing services and storage services are relatively resource isolated, which also improves business stability.
—Architectural goals and technical requirements—
In recent years, the storage-computing separation architecture has not only been widely implemented on public clouds, but also gradually become a hot topic in privatization scenarios. However, it needs to be emphasized that the storage-computing separation architecture is not equivalent to using object storage compatible with the S3 interface to build a data lake, nor is it using containerization to achieve resource isolation or elastic scaling. The storage-computing architecture better meets business needs. A root cause for the upgrade. Since the academic community has not had a rigorous discussion and definition of this architecture, this article attempts to make a more abstract definition of the storage-computing separation architecture:
| The storage-computing separation architecture is a new data architecture design paradigm. It is divided into data analysis layer, computing layer and storage layer from top to bottom. The computing layer and storage layer are decoupled and are independent distributed services. The goal of its design is to solve three requirements: data can be flexibly opened to different businesses for data analysis, independent expansion of computing and storage, and isolation of computing and storage resources, while also providing storage and computing performance equivalent to the integrated storage and computing architecture. |
The reference diagram of the storage-computing separation architecture is as follows:
- Data flexibility and openness
This is one of the most important business goals of the separation of storage and computing, which can open data to business analysis in a better and more flexible manner. Each business team may have its own technology stack and data architecture for data analysis. For example, teams doing interactive data analysis rely more on data marts built with databases and BI tools, while teams doing predictive analysis rely on machine learning and data Lake, etc. Therefore, the storage-computing separation architecture provides a relatively unified interface at the storage layer, while the computing layer can support multiple computing engines, and can perform data query and analysis based on the data interface of the storage layer. The advantage of this method is that most of the data is only stored in one copy, instead of each business user saving a copy of the data results they need. In addition, users can use the Serverless mode to apply for data and computing resources on demand for data analysis, reducing project startup. Cost, providing flexibility and convenience for various businesses to do data innovation.
- Independent scalability of computing and storage layers
This is a very direct technical requirement, that is, the services of the storage layer and the computing layer are relatively independent, especially computing services can be deployed on servers without data storage, and the computing resources or storage resources can be flexibly expanded according to the characteristics of the business. Shrinkage. Based on Hadoop's integrated storage and computing framework, if the computing resources are insufficient, the cluster needs to be expanded, and the storage capacity is also expanded as a whole, which may lead to low utilization of storage resources. With the storage-computing separation architecture, if the computing resources are insufficient, the server dedicated to computing will be expanded while the storage resources remain unchanged, or the storage pool will be expanded when the storage resources are insufficient, which can improve the overall resource utilization rate and better management of heterogeneous servers.
- Improve the resource isolation of storage and calculation
The isolation of storage and computing resources is another technical requirement that promotes the separation of storage and computing architecture. In the integrated storage and computing architecture, if the computing demand increases, computing services can only be added to the server. If resource isolation is insufficient, computing Services and storage services will compete for the same memory or computing resources, resulting in a decline in service stability. And if the resource isolation of storage and computing services is guaranteed through architecture upgrades, the stability and operability of the data platform itself can be greatly improved
- Performance equivalent to storage-computing integration
In addition to business goals, one thing to note is that the storage-computing integration ensures performance by "computing close to storage", while the storage-computing separation framework will inevitably lead to a larger amount of network and storage traffic in the data analysis process, which requires Do more technological innovations to ensure that the performance of data analysis can be at the same level as the integration of storage and computing. The ways that can be realized can include better network/storage hardware and supporting management strategies, or through better cloud scheduling algorithms, or through better cloud scheduling algorithms. Or a better data caching strategy to achieve. Many companies in the industry are already exploring the implementation of the storage-computing separation architecture. At present, this architecture is mostly implemented in the public cloud field, but in the privatization field, this technology is still developing rapidly. There are several schools of thought that promote the development of related technologies, including Big data platform vendors, cloud vendors, storage vendors, and database vendors. These different routes have many similarities in technical implementation, but also have their own uniqueness.
— Separation of storage and calculation of Transwarp big data platform —
Transwarp Technology began to explore the cloudification of big data in 2015, and chose Kubernetes and Docker technology to realize this path, and completed the commercialization of Transwarp Data Cloud in 2018 and completed the production landing. At that time, the storage-computing separation architecture had not been formally proposed, and the K8S-based data cloud architecture also realized storage-computing separation in terms of technical architecture, so we also elaborated on the relevant architecture design.
In terms of design, we abstract the big data platform and services into four layers: cloud operating system layer, data storage layer, computing layer, and data application service layer.
The cloud operating system layer is responsible for abstracting resources such as CPU, GPU, memory, SSD, and HDD into a unified resource pool, so that regardless of the heterogeneous configuration of each server, they can be effectively utilized in real time to improve the effective utilization of resources . The cloud operating system layer shields the underlying hardware details from the upper layer, and exposes resource interfaces such as storage volumes, CPUs, GPUs, and memory in a declarative manner. The upper layer data storage or computing engine can achieve cloud-based elastic expansion by declaratively adding or deleting resources. , without making code changes, which is also the design concept of the currently popular Infrastructure as Code.
The application service layer also uses container technology to support containerization of microservices, traditional applications, and .Net applications and run on cloud platforms. Applications can set scheduling optimization properties to match computing or storage to achieve optimal performance.
In the data storage layer, we further subdivide distributed storage such as HDFS and Search into various sub-services, and gradually containerize these sub-services. At the same time, in order to support high-performance data storage, we use local storage volumes to support Data storage without using distributed cloud storage. The advantage of doing this can make the deployment and operation and maintenance of storage services easier. Although expansion and contraction and cross-node migration are not as smooth as microservices, operations are relatively standardized due to containerization. In actual deployment, TDC allows each business department in the enterprise to use independent tenants to start internal distributed storage on demand, and also allows multiple tenants to globally share the same distributed storage instance. Due to the use of local storage, the same disk is limited in scheduling to allow only one high IO throughput storage service to use.
At the distributed computing layer, TDC containerizes database computing nodes, artificial intelligence frameworks Spark, TensorFlow and other related computing services to achieve online dynamic scheduling, elastic expansion and contraction, etc. In order to ensure the performance of data analysis, we continue the idea of integrating storage and computing, and try to make computing as close to storage as possible. The idea of this optimization is that the distributed storage layer directly provides metadata interfaces so that the computing engine can understand the distribution of data files and share relevant information Exposed to the cloud operating system scheduler, the scheduler will run the computing service as close to the data node as possible by tagging the service, so as to achieve optimal analysis efficiency.
The dynamic elasticity of computing services is a core goal of the storage-computing separation architecture. Since TDC has micro-serviced the internal computing engine and arranged it in the form of containers, the scheduling and arrangement capabilities based on K8S can flexibly add or delete instances according to business needs, ensuring that level of scaling efficiency. We designed two scheduling strategies: time-period-based computing unit auto-scaling and workload-based auto-scaling.
For Inceptor, a relational analysis engine that mainly provides batch processing services, or ArgoDB, which provides integrated lake and warehouse capabilities, due to the obvious tidal effect of night batch processing tasks and daytime data analysis, operation and maintenance managers can choose according to the characteristics of their respective businesses Appropriate scheduling strategy. Auto scaling based on time period is more suitable for scenarios where the business time is very certain, while auto scaling based on workload is theoretically more applicable, but the requirements for related performance indicators will be higher.
Judging from the effect of final enterprise user deployment, Transwarp TDC provides enterprises with a unified multi-tenant storage and computing separation architecture based on container technology. Different tenants are isolated at the network layer to achieve an effect similar to VPC on the public cloud. Data is isolated between different tenants, but data sharing can be done through the data storage deployed in the central data lake. A physical node can run multiple different stateful applications belonging to different tenants. The scheduler will balance resources according to resource conditions, but storage services and computing services are scheduled independently. The computing services of each tenant support default elasticity. A small amount of computing resources can be used when the load is low, and the operating system will automatically expand the capacity when the load is high.
— Storage and computing separation of Cloudera big data platform —
In solving the problem of resource isolation between the storage layer and the computing layer, Cloudera expects to use Kubernetes and Docker technologies to solve the isolation of services and meet the flexibility of data analysis. Since 2019, Cloudera and Redshift have cooperated to develop a container-based big data platform, and will deploy its machine learning product Cloudera Machine Learning (CML) to Kubernetes in 2021, so that users can use CML more conveniently and flexibly. For machine learning work, achieved partial results. However, the storage and computing products in CDP's Private Cloud Base (such as Hive, HDFS, Hbase, Kudu, Impala, etc.) have not yet achieved cloud delivery based on Kubernetes, so they cannot be flexibly opened to businesses, and resource isolation is not enough. not good. In actual deployment, if you need to be able to run a cloud-based machine learning or data engineering product, you also need to deploy a Private Cloud Base based on bare metal alone. Generally, data lakes are built on Private Cloud Base, which is multi-tenant on the upper layer. Computing services provide data interfaces. The CDP topology architecture is shown in the figure below. The lower Private Cloud Base is the main storage layer, and the upper Private Cloud Base is the main computing layer. The abstraction of separation of storage and computing has a relatively large granularity, and it is a series of services composed of multiple components.
In addition, in order to allow more flexible separation at the data lake layer, Cloudera has developed the Ozone storage project compatible with the S3 interface as a supplement to HDFS. The metadata management model adopted by HDFS causes the number of files it can handle to be closely related to the service memory of Namenode, so there is an upper limit on the number of files. Ozone has redesigned the metadata management algorithm so that the maximum number of managed files can reach billions, and the underlying data storage is based on Hadoop Distribution Data Store, which reuses many of the original HDFS designs for performance and uses the Raft protocol to achieve distributed consistency.
The architecture diagram of Ozone is shown above. Its interface layer supports multiple protocols, including Hadoop-compatible Ofs and O3fs, as well as the S3 protocol, and provides Java API and command line support. Since Ozone comes from the Hadoop community, the original applications built based on Hive, Spark and other Hadoop community components can be smoothly migrated to Ozone. In addition, new applications using the S3 protocol can also be supported, which is more ecological than Ceph-like technical solutions. There is a great advantage in compatibility. Ozone has just entered the GA stage, and it still needs to continue to accept the polishing of production cases to improve its maturity and security.