CNSBench, a storage benchmark based on Kubernetes and cloud native

basic knowledge

If readers want more first-hand raw material resources, please refer to CNSBench A Cloud Native Storage Benchmark
Before introducing the main text, taking into account the reasons for the masses, some explanations for the concepts involved in the follow-up will be helpful for understanding
cloud computing(cloud computing) is a kind of distributed computing, which refers to decomposing a huge data computing processing program into countless small programs through the network "cloud", and then processing and analyzing these small programs through a system composed of multiple servers Get the result and return it to the user. In the early days of cloud computing, simply put, it was simple distributed computing, solving task distribution, and merging calculation results. Therefore, cloud computing is also called grid computing. Through this technology, the processing of tens of thousands of data can be completed in a very short period of time (a few seconds), thus achieving powerful network services. Cloud computing refers to a system with extremely strong computing capabilities formed through a computer network (mostly the Internet), which can store and gather related resources and configure them on demand to provide users with personalized services.

The cloud service mentioned at this stage is not just a kind of distributed computing, but the result of the mixed evolution and leap of computer technologies such as distributed computing, utility computing, load balancing, parallel computing, network storage, hot backup redundancy, and virtualization. .

Private Cloud(Private Clouds) are built for single customer use, thus providing the most effective control over data, security and quality of service. The company owns the infrastructure and can control how applications are deployed on this infrastructure. Private clouds can be deployed inside the firewall of an enterprise data center, or they can be deployed in a secure hosting location, and the core attribute of a private cloud is dedicated resources.
public cloudIt usually refers to the available cloud provided by third-party providers for users. Public clouds are generally available through the Internet, and may be free or low-cost. The core attribute of public clouds is shared resource services. There are many instances of this cloud, providing services throughout today's open public web.
== Containerization == is an application or system distribution method that isolates the application or system and its dependencies from the underlying infrastructure. It is an operating system-level virtualization that allows users to deploy and run distributed applications or systems in containers without launching an entire virtual machine for each application.
Containers play a very important role: to ensure the consistency of the code running environment.
Containers provide a lightweight and immutable infrastructure for application packaging and deployment to solve the problem that applications cannot run normally when they are moved to other environments. A program or service, its dependencies, and its configuration are packaged as a container image. Container technology enables developers and IT professionals to deploy applications across environments with little or no modification.

As for why containers are better than traditional virtual machines:
(1) Containers are lighter than virtual machines
(2) Containers are virtualized at the operating system level, while virtual machines are virtualized at the hardware level
(3) Containers Shared operating system kernel takes up much less memory than virtual machines

cloud native(Cloud Native) is an application developed based on the principle of microservices and packaged in a container. At runtime, containers are scheduled by the platform running on top of the cloud infrastructure. Application development employs continuous delivery and DevOps practices. To sum up, cloud native is a technology based on cloud computing, which is the continuous refinement and refinement of cloud computing technology.
microservice: The traditional large monolithic application is split into smaller components or modules, which are called microservices. Split into vertical splits, from the underlying IT infrastructure to databases, to application middleware, and to software program deployment packages can be completely independent, and can be independently designed, developed, packaged and deployed to achieve each micro A complete loose coupling between services. At the same time, various microservices can interact and collaborate through the lightweight HTTP rest interface. To sum up, the core of microservices is two points: large monomers are disassembled into small ones, and small microservices interact and collaborate through interfaces.
KubernetesIt is a container cluster management system open sourced by Google in June 2014. It is developed in Go language, also called K8S. The goal of Kubernetes is to make deploying containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. The correct pronunciation is **[kubə'netis]**

Summary and introduction of CNSBench

CNSBench (github.com), the open source address of the article application is as before, and readers can find it in the GitHub warehouse.
Abstract :
Modern hybrid cloud infrastructures require software to be easily portable across heterogeneous clusters. Application containerization is a proven technology that provides this portability of application functionality. However, to ensure performance portability, reliable validation of cluster performance under realistic workloads is required. This validation is usually achieved by benchmarking the target environment and its storage, since I/O is often the slowest component of the application. However, existing storage benchmarks are not suitable for generating cloud-native workloads because they do not generate any storage control operations (e.g., volume or snapshot creation), cannot easily orchestrate a large number of different The ability to dynamically change workload characteristics in .
In this paper, we present the design and prototype of the first cloud-native storage benchmark - CNSBench. CNSBench combines control operations with traditional storage benchmark workloads with user-defined control operations workloads. Since CNSBench itself is a cloud-native application, it natively supports orchestrating different combinations of control and I/O workloads at scale. We built a CNSBench prototype for Kubernetes, leveraging several existing container storage benchmarks for data and metadata I/O generation. We demonstrate the usefulness of CNSBench to uncover and analyze previously unknown performance characteristics through case studies of Ceph and OpenEBS, two popular storage providers on Kubernetes.

Introduction :
The basic requirement of benchmarking is to be able to generate actual workloads, and the results can reflect the performance after it is actually used. Because of the following three shortcomings but the existing storage benchmarks are insufficient to generate workloads in modern cloud-native environments, we propose the Kubernetes-based CNSBench.
① Cloud-native storage workloads include a large number of control operations, such as volume creation, snapshots, etc.
② Typical workloads should host a large number of different workloads that run concurrently
③ Applications in cloud-native environments are highly dynamic to
start, stop, and expand , failover, update, rollback, and more. This will cause various changes in workload behavior over short periods of time as the arrangement of workloads running on each host changes. Although existing benchmarks allow configuring separate runs of a benchmark to generate workloads of different stages, such benchmarks do not provide a general way to represent dynamics within a single run

Cnsbench accomplishes the following three improvements for this :
(i) generating actual control operations;
(ii) coordinating various storage workloads; and
(iii) dynamically changing workloads as part of the benchmark run.
Therefore, CNSBench can be regarded as an application that can generate control operation benchmarks and make various parts coordinate when a large number of containers are running.

文章的主要贡献有
1. 我们将确定云本地存储基准测试的需求和独特要求。
2. 本文介绍了CNSBench的设计和实现，这是一个满足上述需求的基准测试，允许用户在实际工作负载的情况
   下方便地大规模地对云本地存储解决方案进行基准测试。
3.我们使用CNSBench来研究Kubernetes的两种存储解决方案(Ceph和OpenEBS)在之前没有研究过的工作负载下
的性能。

Before introducing CNSBench, it is necessary to introduce the environment it is based on. CNSBench is equivalent to a real-time and fast application for cloud native built in the Kubernetes environment, and it performs well and is innovative.

Kubernetes Background

It consists of control plane nodes, worker nodes, and storage providers (among other components). Worker nodes and control plane nodes run Pods, which are the smallest workload units in Kubernetes consisting of one or more containers. User workloads run on worker nodes, while core Kubernetes components run on control plane nodes. Core components include (1) the API server, which manages the state of the Kubernetes cluster and exposes HTTP endpoints for accessing the state; (2) the scheduler, which assigns Pods to nodes. Typically, a Kubernetes cluster has multiple worker nodes and possibly multiple control plane nodes for high availability.
Storage providers are responsible for providing persistent storage in the form of volumes as needed by individual pods.

Kubernetes workflow

Pod (Plain Old Documentation) is the smallest resource management component in kubernetes, and Pod is also a resource object that minimizes running containerized applications. A Pod represents a process running in the cluster.

Persistent storage is also an introduction to the workflow of Kubernetes: a Pod that needs storage generates a PVC (persistent storage statement, persistent volume claim), PVC indicates how much storage space it needs, storage control panel, working nodes and storage providers, and provides for Pod Operations and resources involved in storage. Then the persistent volume (PV, persistent volume) is provided by the provider. If there is an existing PV that satisfies the storage request, it is used. Otherwise, a new PV is provided from the storage provider specified in the PVC. Once a PV is formed, it is bound to a PVC and the volume is mounted on the Pod's filesystem.

这里的PVC就相当于是任务清单，告诉你要做什么，然后根据任务清单从存储库中实例化PV，PV就相当于是OS
中进程的PCB，具体对应到工作节点中的最小资源调度单位Pod，PV对其进行管理和说明。但这里资源调度的最
小单位Pod不是说大小最小，而实际上是把它绑定为了一个整体，每次运行都是整体运行调用。

Requirements analysis proposed by CNSBench

(i) The frequency of control operations increases; such as the creation and deletion of containers, the types of control operations are also more and more abundant. In addition to being rich, control operations that rely on the underlying storage technology can also be data-intensive. This makes them slower and increases their impact on the I/O path of the running application. For example, volume creation often requires (i) time-consuming file system formatting; (ii) creation or deletion of snapshots, which, depending on the design of the storage, may consume significant I/O traffic; volume resizing, which may require data migration and Updates to many metadata structures; and (iv) volume remounts, which cause cache flushes and warmups.
(ii) High diversity of individual workloads;
the variety of applications and workload specialization makes actual workload generation only by manually selecting, creating, and deploying a few appropriate containers (e.g., running multiple individual storage benchmarks, each of which simulates the characteristics of a single workload). However, as more and more diverse applications are containerized and broken down into specialized sets of microservices, the number of containers that must be selected to make up the actual workload continues to increase. Making this selection manually is not feasible in today's cloud-native environment.
(iii) The dynamic nature of these workloads.
Benchmarks lack the ability to easily assess application performance under these highly dynamic conditions. In some cases, benchmark users resort to manually creating these conditions in order to assess how applications respond—such as manually scaling the number of database instances [46]. However, the high dynamism and variety found in cloud-native environments makes it nearly impossible to manually recreate these conditions.

Therefore, the core requirements of the following five-point design are proposed

1. I/O workloads should be specified and created independently of control workloads in order to have an understanding of (i) the performance of I/O workloads under different control workloads and (ii) the performance of control workloads under different I/O workloads Benchmark.
2. It should be able to orchestrate I/O and control workloads to simulate the dynamic environments that represent today's cloud. Additionally, it should be possible to generate control workloads as micro-benchmarks for evaluating the performance of individual control operations.
3. The I/O workload should be generated by running an existing tool or application, either a synthetic workload generator like Filebench, or a real application like a web server with a traffic generator
4. Users should be able to quickly configure and run benchmarks without sacrificing the customizability offered to more advanced users.
5. Benchmarks should be able to aggregate unstructured output from different benchmarks into one convenient location for further analysis

CNSBench design and implementation

After introducing the implementation background and requirements, let’s start introducing the text

Schematic diagram of CNSBench

The blue part is the component of CNSBench.
The overall control flow is as follows: A benchmark controller monitors the API server to create new benchmark resources. B When a new baseline resource is created, the controller creates the resources described in the baseline's I/O workload: I/OW orkload P ods for running the workload and Persistent V volume for running the workload (pv) of pvc (Persistent V volume Claims). C To run the control operation workload, the benchmark includes a rate generator that triggers an action executor at user-specified intervals to invoke the desired control operation (action).

Under the original Kubernetes, CNSBench implements the benchmark definition controller, which includes separate control of actions and rates, I/O load control, etc., and completes the packaging in the container. When control is required, only Friendly use such as parameter input is required, which reduces a lot of code editing and multiple but non-repeated declaration work in multiple different working environments. CNSBench is compatible with many kinds of benchmark test situations, allowing users to concentrate more on the test without worrying about the tedious work of grammar, definition, code writing, etc.

There are two main cores of CNSBench: benchmark custom resources and benchmark controllers

Benchmark Custom Resources for CNSBench

Baseline Custom Resource Definition

The benchmark custom resource allows the user to specify three main benchmark attributes: (1) the control operation workload; (2) the I/O load of the run; and (3) where the output should be sent for collection.
(1) Control operation workload:
One of the main requirements of CNSBench is to be able to create a realistic control workload, using a combination of operation and rate to specify the control workload, the action and rate can be separated, so that the problem can be viewed more clearly.
(2) I/O workloads
Typically, the goal of benchmarking is to understand how a particular workload or set of workloads performs under various conditions. The role of the I/O workload component of CNSBench is to instantiate these workloads, or to instantiate synthetic workloads with the same I/O characteristics as real workloads. Specifying these I/O workloads requires defining all the different resources (e.g., Pods and pvcs) that must be created in order to run the I/O workload. This can be difficult and makes benchmark specification lengthy and complex.
To ease the burden on users and help them focus on the overall benchmark specification rather than the specifics of the I/O workload, CNSBench separates the I/O workload specification from the rest of the benchmark specification. I/O workload specifications are defined using configmap, a core Kubernetes resource used to store configuration files and other free-form text.
Users can therefore specify which I/O workload to run in a Benchmark custom resource using a create resource action that references (by name) the I/O workload to be created. To enable reuse across various use cases and benchmarks, fields in an I/O workload specification can be parameterized and specify values ​​when the workload is instantiated by a particular benchmark.
(3) Where should the output be sent for collection (benchmark output)
There are three challenges in collecting output: there is currently no clean and general way to extract files from Pods; the runtime to generate output can be very long; a lot of output is unstructured text form, which makes the results difficult to analyze.
To address these issues, we allow I/O workload authors to specify which files should be collected from workload Pods, and provide parser scripts to process the output. Parsing the output allows reducing large files to a more compact size and outputting the result in a standard way. Use a helper container to collect and parse output files. The user specifies where to send the final parsed results to the output section of the Benchmark custom resource. The results need not all be sent to the same output.

Baseline custom resource instance

6-13: Benchmark I/O workloads specified
This allows I/O workload specifications to be parameterized to support their reuse across different use cases and benchmarks.
14-18: Specify a snapshot volume operation

Corresponding to this is the I/O workload worksheet and the I/O load graph. The

above figure is a sample I/O workload specification.

The above figure is a subset of a Kubernetes cluster with a single worker node and a PV. Shows the CNSBench resources involved (I/OW workloads and benchmarks), as well as the core Kubernetes resources (snapshots, pvcs, PVs, and workload Pods) created by the CNSBench controller according to the benchmark specification.

Benchmark Controller

The benchmark controller monitors newly created benchmark objects and runs the actions specified by them. The controller has three main responsibilities: (1) trigger control operations; (2) synchronize each benchmark workload; (3) collect the output of each workload.

Performance Tests and Experiments

Test environment and content

Each worker node is a virtual machine with 4 vcpus, 8GB RAM and 384GB locally attached storage. The control plane node is a virtual machine with 4vcpu, 12GB RAM, and 100GB local storage. The virtual machine hosts are located in multiple racks, the racks are connected by a 10Gbps network, and the individual hosts are connected to the top of the rack switch by a 1Gbps link.

Performance was tested under Ceph and OpenEBS storage provider platforms with no data replicas, three data replicas, running only empty storage provider and erasure coding mode

Experimental results

Configure the cdf of the time it takes to create and mount volumes for different storage providers. N is the number of volumes created at the same time. For all storage configurations, an increase in the number of simultaneous volume operations increases the average time to create and mount a single volume

Volumes and attachments created per minute for varying numbers of simultaneous operations. The vertical bars at each point show the standard deviation of volume creation and attachment rates at that point.

Impact of snapshots on I/O workloads. R = 0 means 0-volume copy, R = 3 means 3-volume copy, R = ec means erasure coding

CDF of snapshot creation times for different storage provider configurations.
and operating data in combination

Performance change from baseline for three different proportions of I/O workloads on five different storage configurations.

future outlook

In some cases, control and I/O operations may be interleaved. For example, an increase in I/O operations may cause the workload to scale out to perform more control operations. Reproducing such events with CNSBench requires a feedback mechanism to pass information to CNSBench about the I/O operations performed by the I/O workload. The design and implementation of CNSBench does not preclude such a mechanism, but we leave its implementation to future work.

总的来说，文章就是在云原生的环境中，设计了符号各种平台负载，很好连接并可以进行性能评估的应用。
应用实    现了内部接口，使得生成负载参数化，不需要每次进行繁琐的代码设计编程，且也可以忽略内部
Kubernetes的设计规范，因为内部已经完成了封装，将其考虑进接口中的内部程序了。通过分析，应用更贴近于
现在流行的云原生，且在大环境下具有创造性地提出可广泛应用的性能评估测试应用，使得文章能够在FAST
中脱颖而出。

references

[9]Merenstein A, Tarasov V, Anwar A, et al. {CNSBench}: A Cloud Native Storage Benchmark[C]//19th USENIX Conference on File and Storage Technologies (FAST 21). 2021: 263-276.