1 preamble
Starting with file storage, we start defining the storage of information:
Most people's understanding of storage starts with files. Files hold information, can be read and modified randomly, files are organized by directory, and so on.
The upgrade and evolution process of the storage system:
With the beginning of the information world, the amount of high-value data is increasing rapidly, and the demand for data storage systems is also constantly changing, which promotes the continuous evolution of storage systems.
Distributed architecture storage systems in different scenarios:
After entering the era of the Internet and AI, the growth rate of data volume is much faster than the growth rate of hardware capabilities. It is inevitable for distributed architecture to replace centralized architecture. A large number of distributed architecture storage systems suitable for different scenarios have sprung up like mushrooms.
In general, the evolution of storage systems is determined by the following factors:
To measure the quality of a storage system, we should start from the following aspects:
2 Introduction to Object Storage
In short, object storage is key/value-based storage . Logically, object storage can be regarded as an infinite key/value table.
Many students are familiar with kv-based caches, such as redis and memcache. When kv can be fully persisted, it will have the prototype of object storage.
The kv-based database is also a kind of object storage, but the storage capacity is limited, and it is limited to storing metadata.
Object storage in a general sense is a storage system for storing data. It has good horizontal expansion capabilities in capacity and performance, and is suitable for storing large amounts of unstructured data.
The reason for object storage is to meet the storage requirements of a large amount of unstructured data - the amount of data is too large, the cost of using traditional storage is too high, and the complexity of management and use caused by the huge expansion of the scale is also greatly increased. Therefore, there is an urgent need for a low-cost storage system that is easy to deploy, manage, and use, easy to scale out, suitable for unstructured data, and this is object storage.
Correspondingly, the main purpose of object storage is simple, low-cost access and management of large amounts of unstructured data . When object storage and cloud storage requirements are combined, object storage is endowed with the mission of cloud data lifecycle management.
The above is the origin and development direction of object storage.
3 Features and advantages of object storage
Easy to use, wide range of applications
Standard Restful API (and API-based SDK) access, independent of OS/platform, access data anytime and anywhere (whether it is a virtual machine, container, or embedded system, as long as there is a network, it can be accessed)
The data organization is flat and simple, essentially accessing (value, attribute) through key. On this basis, it supports the ability to read and write value according to key , scan attribute according to key range, etc. The data management capability is simple and powerful. In the case of a large amount of data, the metadata management performance far exceeds that of the file system.
natural cloud storage
1) AWS S3 is the most widely compatible object storage interface protocol, and it is a practical common standard for object storage interface protocols. S3 is defined to support cloud storage and naturally supports cloud native.
2) Multi-tenant data life cycle management in the cloud (authorization; access control; compression; encryption; label; log audit; regulatory compliance; WORM; data quota; Qos; multi-version; layering; remote replication; ...).
3) The container instantly accesses the object storage through S3 without mounting.
4) Good scalability, flexible supply of storage space and performance on demand (declarative API), and a huge upper limit of supply.
5) Fault tolerance and self-healing.
6) Automated operation and maintenance.
low cost
The data is read-only and cannot be modified. Under this premise, the storage system can be optimized to reduce costs.
Simple deployment and management, low maintenance costs.
4 Limitations of object storage
Applicable data type range restrictions
The applicable data type is unstructured data, not structured data that needs to be modified randomly.
Access Interface Compatibility
It can only be accessed through the Restful interface and is not compatible with Posix/Nas. Most old applications must pass through the Posix/Nas gateway if they want to apply object storage.
performance limitations
High access latency (caused by cloud storage software stack, separation of protocol layer and storage layer, etc.).
In the previous section "The Ins and Outs of Object Storage" , we introduced the elements that drive the evolution of storage systems, the cause, development direction, and characteristics of object storage. In this section, we describe the application of object storage in combination with the usage scenarios of object storage.
Rule # 1 -- Object Storage and Unstructured Data --
We know that object storage was born to solve the pain points of large-scale unstructured data storage. Therefore, object storage is very suitable for scenarios where a large number of documents, pictures, and videos are stored and processed, including media, backup/archive, video surveillance, satellite/meteorological/geological data, Internet of Things, big data, data lakes, etc. The basic requirements of these scenarios are low-cost, high-performance access and processing of a large amount of document data. At the same time, the emphasis is different in terms of file size, quantity, and IOPS performance. For example, media, backup/archive, and video surveillance scenarios are generally large files, requiring high throughput ; satellite/meteorological/geological data, and Internet of Things scenarios may have a large number of small files, requiring high performance for storage and retrieval of a large number of small files ; The requirements for big data and data lake scenarios are more comprehensive, including the ability to store a large number of small files, comprehensive performance, and heterogeneous data access capabilities .
Before the large-scale application of object storage, most unstructured data was stored on cheap and simple storage systems represented by NAS. The object storage system must consider how to be compatible with the old storage system and application system. During a long product replacement period, the newly deployed object storage system must be able to adapt to both new and old data and applications. This requires storage gateways and storage management systems.
Take storage in the video surveillance industry as an example. Most of the older video surveillance systems use NAS to store data. Compared with NAS systems, object storage systems have many advantages such as high reliability, good scalability, low cost, global namespace, and easy data processing. However, if you want to replace NAS storage with object storage without modifying the application system (only supports NAS protocol), you need to convert NAS to object storage gateway. At the same time, the old NAS storage system and the new object storage system will coexist for a period of time, requiring unified management, data import (NAS data import object storage) and other functions.
To use object storage in the old version of the big data system, it is also necessary to deploy the HDFS to object gateway. New versions of big data systems can directly use S3-compatible object storage.
Rule 2 -- Object Storage and the Cloud --
Object storage not only solves the problem of massive unstructured storage, but also becomes the standard of cloud storage.
With the development of cloud storage to this day, the main goal is to provide agile, highly available, reliable, and secure storage for cloud applications based on container and microservice architectures.
From the perspective of data and service availability, cloud storage has the following characteristics:
extremely high reliability
Compared with traditional storage, cloud applications have greatly improved data reliability standards. For example, the reliability standard of public cloud storage can reach 99.99999999%
high availability
Compared with traditional storage, cloud applications have very high standards for storage service availability. For example, the availability standard of public cloud storage can reach 99.995%
excellent elasticity
On-Demand Storage - Instead of pre-allocating large amounts of resources, they are provisioned dynamically
Save as much as you want - the application can store as much data as you want, without modifying the program, expanding the capacity or interrupting the service
Performance elasticity - performance linear, agile horizontal expansion; during the peak period of business, more service resources are required to be started quickly, business requests fall back, and resources are recovered
multi-tenant
The resource allocation of tenants can be carried out according to the policy to achieve the security, availability and performance isolation of different tenant data
From the application level of data, cloud storage manages the entire life cycle of data.
data generation
Import from outside the cloud, or generate in the cloud
compression, deduplication
Quota, QoS
Data security, sharing, retrieval
Security compliance (multi-version; encryption; log audit, regulatory compliance; WORM...)
Authorization, Access Control
label search
data flow
life cycle
layered
Copy and Migrate
Cloud storage includes block storage, file storage, and object storage. Among them, only object storage is the most in line with cloud-native specifications, and can best achieve the reliability, availability, elasticity, multi-tenancy, and full lifecycle management of cloud storage data. Object storage is the cornerstone and future of cloud storage.
Rule # 3 -- Object Storage and Data Lakes --
The use of large amounts of data storage is mainly data analysis. Data analysis, machine learning, and intelligence based on big data are the direction of future digitalization. However, the current big data processing system has problems such as scattered data sources and isolated information systems, which lead to difficulties and inefficiencies in data processing. The data lake was born to solve the problem of isolated information islands and process data in a unified and efficient manner.
A data lake is an architectural system. Through the data lake, massive amounts of data can be stored, processed, and analyzed in a fast, safe and compliant manner; data-oriented, interactive integration with external computing resources through interfaces, to achieve any source, any speed, any scale, any type of data acquisition, Storage and full lifecycle management.
With the data lake, the data analysis system does not need to switch between different data warehouses and file storage frequently, nor does it need to repeat the logic of writing, extracting and loading, which greatly improves the efficiency.
The storage of data lakes relies on S3-compatible object storage. S3 is the most important part of data lakes due to its powerful features :
Amazon Cloud Computing Services (AWS) clearly uses S3 storage as the base of the data lake and the center of data flow. S3-compatible object storage is the best choice for data lake storage.
In the previous section " Usage Scenarios of Object Storage ", we introduced the use of object storage in scenarios such as large-scale unstructured data storage, cloud storage, big data, and data lakes.
So, what does a good object storage product look like? How to design and implement? In this section, we analyze and discuss these issues.
Through the study of the first two sections, we should understand the characteristics and elements of object storage, and then we can draw a more perfect outline of object storage. But in practice, many advantages are difficult to coexist, or cannot be achieved in one step, and trade-offs must be made. For example, high performance and low cost. Generally, an increase in performance will bring about an increase in cost. Another example is full-featured and streamlined functions. Full-featured is widely applicable, but it will bring the consequences of many software bugs and difficult maintenance. Therefore, object storage products with different characteristics have different audiences and scopes of application.
Take the open source object storage MinIO as an example
MinIO's design philosophy is to be minimalist in architecture and compatible with S3 in function. It is very simple to install, deploy, and use MinIO. At the same time, MinIO is functionally compatible with S3 and is very suitable for small-scale cloud-native application scenarios . With these two main features, the open source community of MinIO is very active, and MinIO is widely used as a lightweight and excellent object storage.
In fact, MinIO also has obvious shortcomings, but this does not prevent its use in small-scale cloud-native scenarios.
MinIO's short board is limited by scalability, scale and performance
At present, other object storage products also have trade-offs in terms of functions, elasticity, scale, and performance. Generally speaking, they are relatively large-scale (relative to distributed files, but not extremely large-scale), low-cost, and average-performance. .
The main purpose of object storage is large-scale data processing. In terms of requirements, it should not be limited to scenarios with large files and high latency. Object storage should also be competent for scenarios that require high IOPS and low latency, such as machine learning. Only a new generation of high-performance, all-flash object storage products can handle this scenario.
Next, let's discuss the requirements and capabilities of a new generation of high-performance object storage products.
1 Requirement: Expand capabilities in the original scenario and eliminate short-term expansion
new market
→ Unstructured data storage
Improve product software-defined flexibility
improve performance
→ Cloud storage (private cloud, hybrid cloud, data lake)
Reduce the cost of scaling while increasing the scalability and upper limit of scale
Improve performance to meet the requirements of high-performance data processing
S3 is fully compatible, with multi-tenant data full lifecycle management capabilities
2 abilities
→ High security, high reliability, and high availability of data in a distributed environment
Data security, reliability, and availability meet cloud storage standards
→ Horizontal linear expansion of data volume and performance
The elasticity of data volume and performance meets cloud storage and cloud native standards
→ high performance
Throughput, IOPS, latency and other indicators meet high-performance data processing (such as machine learning) standards
→ low cost
Reduce the total cost of ownership of large-scale data storage (high density, green energy saving, increase the service life of large-capacity storage media)
The core issue of the above requirements and capabilities is how to achieve top-level performance while supporting elastic and low-cost storage of ultra-large-scale data. Aiming at the core issues, we analyze the key points of the design step by step.
3 design points
→ To achieve top performance, an all-flash storage architecture must be used to reduce the data flow delay of the software stack
Software stack data flow (taking writing as an example), including protocol layer processing, link layer (RPC and RDMA), and storage layer processing. The storage layer processing includes data redundancy calculation, resource allocation, writing data, and writing metadata (resource metadata and object metadata) transactions. Most of the IOPS bottlenecks and delays occur at the storage layer.
To reduce the latency of the storage layer, the most critical point is the distributed metadata service.
Metadata services must be distributed and horizontally scalable in order to support the horizontal expansion of capacity and performance of storage products.
Metadata-free service architectures, such as consistent Hash, have two fatal shortcomings and are not suitable for large-scale distributed systems. First, without the range query capability of metadata, the range query will be enlarged to the entire cluster, resulting in very poor metadata management performance in a system with a large number of objects. Second, the rebalance state caused by cluster elasticity requirements is very complicated, which deviates from the original intention of stateless Hash.
Back to the metadata service. The data layer of the metadata service generally implements copy replication and horizontal sharding of the MVCC collection on the Raft protocol. The interface layer implements transactions. These matters make it very difficult to implement low-latency metadata services, and it is necessary to master and optimize these core technologies.
In addition to low-latency metadata services, there is another key point, which is the computing scheduling of the all-flash software stack.
The all-flash software stack adopts new technologies of RDMA and NVMe , and the storage layer delay is greatly reduced, which is no longer a performance bottleneck. On the contrary, calculation and scheduling in an unoptimized multi-core CPU environment will become a performance bottleneck under extremely high concurrency. The all-flash software stack uses a micro-computing pipeline architecture to eliminate the performance barriers of multi-core CPU NUMA architecture caching and scheduling, achieving extremely low computing latency and high concurrency.
→ To support flexible storage of ultra-large-scale data, a fully distributed architecture must be adopted to increase the capacity of storage media
Among them, the elastic scaling of stateful distributed service clusters is the core difficulty.
→ High security, high reliability, and high availability of data in a distributed environment
Currently, enterprise-level large-capacity QLC_SSD has been used in all-flash storage software systems. The all-flash storage software stack needs to solve the problem of low erasing times of QLC_SSD. If the GC, write amplification, and wear leveling problems of SSDs are solved at the storage cluster level, we can reduce the write amplification ratio and balance the number of erasures for all SSDs in the cluster. Not only can we use QLC_SSD with a low number of erasures, but also Greatly increase the service life of SSD.
ZNS_SSD is a standard for using large-capacity SSD, reducing cost and power consumption. The all-flash storage software stack must be adapted and support the use of ZNS_SSD.
To minimize the cost of large-capacity SSDs, you can use the customized Open_Channel_SSD solution to put the SSD's FTL layer and error correction layer on the host side, saving the DRAM controller and adapting to the use of cheap SSD particles, which greatly reduces SSD cost.
The all-flash storage software stack can also achieve energy saving in cluster resource scheduling and make full use of the energy-saving status of SSDs.
→ The contradiction between high performance and low cost is alleviated by data layering
The hot data layer adopts high-performance all-flash software stack and high-priced high-performance NVMe_SSD.
The warm and cold data layer adopts ordinary all-flash software stack and cheap large-capacity SSD.
Data layering is transparent, intelligent, and application-insensitive.
4 Design Discussion
Why propose a seemingly perfect distributed architecture instead of the trade-offs mentioned above?
When we introduced MinIO, we discussed the trade-offs, focusing on the simplicity of architecture and functions. But the demand is developing. The flexibility and high performance of the horizontal expansion of the object storage system have become the most important requirements at present and in the future, and can expand a large market, so we focus on the two most difficult aspects. Enterprise-level storage with a distributed all-flash architecture.
5 Avenue Cloud FOSS all-flash distributed object storage introduction
Avenue Cloud FOSS is a cloud storage system that adopts an advanced distributed all-flash architecture, aiming at long-term, reliable, green, low-carbon, and high-performance access to ultra-large-scale data.
→ All-flash architecture, ultra-high performance, horizontal linear expansion of data volume and performance
millisecond latency
Single cluster, 100P-level space, 100G-level throughput, million-level IOPS
Multi-cluster expansion in the same city and across regions
Cross-zone data replication and disaster recovery
→ Green and low carbon
Green and energy-saving (most SSDs are in low power consumption state most of the time, single-chip SSD<0.5w).
The SSD has a long service life and more actual available space (7% GC reservation is not required).
→ Long-term reliability
Regardless of whether it is stored, it ensures reliable availability of data for more than ten years (data silent error protection; intelligent media and data inspection, health scanning, erasure recovery, and migration of expired data).
We can see that two important processes are taking place and developing.
One, is the storage alternative
At present, most of the unstructured data in our country are stored on the old NAS storage system. NAS storage systems have many problems:
Cloud storage (private cloud, hybrid cloud) with object storage as the core surpasses NAS storage in all aspects, such as capacity, scalability, security and reliability, performance, unified management, multi-tenancy, and data lifecycle management. And the overall cost of use is equivalent. Therefore, it is inevitable for object storage to replace NAS and improve the productivity of application systems in various industries. Of course, this process is gradual, accompanied by the use of storage gateways and management systems.
Big Data System 2.0 proposes the concept of separation of computing and storage, and advocates the use of data lakes to manage all data processing in a unified manner. Big data storage is accelerating the shift from HDFS to (S3 or Swift-compatible) object storage.
Second, the booming cloud storage market
Cloud storage in a broad sense includes distributed storage of different scales, forming edge cloud, private cloud, hybrid cloud, and public cloud.
At present, the digital economy driven by data is becoming a new engine for my country's economic development, and data is value. However, the value of data needs to be discovered and reflected through an intelligent data analysis system. In the future, all walks of life will need intelligent data analysis systems in line with industry characteristics to improve productivity and competitiveness.
The ultimate goal of big data is intelligence, and intelligence needs to rely on AI technologies such as machine learning and deep learning. Therefore, the future data processing platform must support machine learning. Correspondingly, there are high requirements for high performance (high IOPS, low latency, and large number of files) of the storage system.
In the future, the intelligent data processing platform must be built on the cloud, and the high-performance, large-scale, and low-cost cloud-native storage system based on the all-flash architecture will always flourish along with the intelligent cloud and become the dominant Storage System.
Intelligent data processing is not only high-performance centralized processing, but also includes edge computing, cold data access, backup and archiving, remote replication and disaster recovery, etc. Storage systems have a huge market space here. But as the future progresses, the higher the degree of intelligence, the higher the requirements for cloud-native storage systems, and the higher the threshold.
How to do?
Although the prospect is bright, it is a gradual process to effectively increase productivity and replace old ones with new ones. In other words, there is a time window for market development.
We need to focus on the demand for intelligent data analysis in the industry, and create an intelligent data analysis platform that meets the characteristics of various industries, such as industry cloud, Internet of Things, and data lakes, to truly and effectively transform data intelligence into productivity and competitiveness. , our storage system has high-value data, which reflects the value of its excellent capabilities.
Specifically, it is to deeply participate in the solutions and construction of enterprises, industry data systems and Internet of Things/cloud/data lakes, and develop together with cloud (private cloud, edge cloud) in terms of intelligent data processing.
References:
[1] AWS. Data Lake Storage
https://aws.amazon.com/cn/products/storage/data-lake-storage/?nc=sn&loc=4
[2] Minio. WhitePaper
https://storage.oak-tree.tech/acorn/documents/minio.high-performance-s3.pdf
[3] Mikito Takada. Distributed Systems for Fun and Profit
[4] Martin Kleppmann. Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
[5] Diego Ongaro, John Ousterhout. Raft Consensus Algorithm
https://raft.github.io/raft.pdf
(Original by TaoCloud team)