Introduction : This article is organized from the series sharing of the "Cloud Intelligence Open Course" of the same name, and introduces in detail how Baidu Canghai·Storage conducts large-scale data transfer to the cloud, intelligent hierarchical storage, security management, and multi-service applications. .
The full text is 5657 words, and the estimated reading time is 20 minutes.
01 Four challenges facing storage in the ABC era
We call this current era the ABC era. A means that we are in the era of artificial intelligence; B means that we are in the era of big data; C means that we are in an era where everything can go to the cloud.
Storage systems have appeared many years ago. What new challenges will storage face today?
The first keyword of the challenge: massive. For enterprises, it may only store some application data on the Internet in the past, and at most some text data, backup data. But now, we see more video, audio, etc. as carriers, and the amount of data shows an explosive growth trend. In this context, cloud service providers are faced with how to solve the problem of cloud migration and storage of massive data. In other words, how better our physical capacity can handle the explosion of data.
The second key word of the challenge: cost performance. In this day and age, we view data as a valuable asset. Since the data is valuable, it requires the cloud service provider how to help customers spend a small amount of money to do big things on the premise of ensuring that the data is not deleted. This is what customers and we are more concerned about. For example, 10 years ago, the customer's data was 10 petabytes, and 10 years later, its data grew to 50 petabytes, which is 5 times the amount of data. Does that mean customers are also 5x the cost of storage? This is not necessarily the case, as we try to help our customers to minimize the cost of data storage.
The third keyword of the challenge: stability. When the distributed system carries tens of thousands of customer services, we are walking on thin ice, because this requires us to ensure the stability of the system. At the same time, our storage products also need to help customers achieve certain disaster recovery capabilities and certain backup capabilities.
The fourth keyword of the challenge: diversity. Diversity is actually reflected in many aspects. The most important one is that the business scenarios of customers are becoming more and more diverse. For example, many years ago, the customer's scenario was that after the data was stored, the data could be read out when needed. Today's data is not only storage, but also different scenarios, such as big data analysis, AI training, hybrid cloud platform construction, etc., need to use different storage products and combinations to meet business needs.
In response to these four challenges, today I mainly share with you five parts.
02 Overview of Baidu Canghai Storage Product System
First of all, let me introduce you to Baidu Canghai Storage. He has ensured the efficient and reliable operation of Baidu's core business. For example, we are familiar with Baidu Search, Baidu Netdisk, Baidu Tieba, Baijiahao, Baidu Map, Baidu's world-leading AI business and so on.
Canghai's product system is a matrix structure. Including object storage BOS, block storage CDS, file storage CFS, parallel file storage PFS and so on.
In addition, we will also have products for specific scenarios, such as data lake storage acceleration RapidFS, which aims to accelerate the access of object storage data in big data or AI scenarios. It also includes edge storage, as well as ABC Storage, a hybrid cloud storage for traditional customers.
In addition, we also have some tool-based products, such as the data transfer platform CloudFlow, which solves some problems of data transfer to the cloud and transfer. In addition, there are scenarios such as cloud migration for IDC enterprises. We have launched the product Moonlight Treasure Box, which can realize data copying and physical relocation. In addition, for hybrid cloud scenarios, we also provide storage gateway capabilities. For example, if a user's computing node is local, and he purchases object storage in the cloud, he can integrate the local and the cloud through the storage gateway, and expand the local resource space in the cloud.
The above is the product system of Canghai. Below the product system is our overall technology platform. We emphasize three points, one is the coordination of storage and computing, the other is the integration of software and hardware, and the other is the integration of cloud and edge.
The upper layer of the product system is our solution. We have served tens of thousands of customers, and in the process we have accumulated many solutions, such as cloud album solutions. Everyone knows that mobile phones now have a cloud photo album function, and integrate some capabilities in the photo album, such as classifying faces to form face photo albums, etc. Based on this, we also provide cloud photo album solutions for mobile phone manufacturers.
There are also storage distribution schemes like the Internet. For example, a TV series, a movie, and a short video need to be distributed to terminals all over the world, so we have also launched an Internet storage and distribution solution. In addition, for customers who store data in different cloud vendors, we also provide multi-cloud solutions.
In addition, we also provide archive backup solutions. Some data is not used for a long time, but is accessed occasionally, so we provide a low-cost archive backup solution. In addition, we will have different solutions for different industries or different scenarios, such as game storage, autonomous driving, compliance storage, and medical image storage.
03 How Baidu Canghai·Storage solves four major challenges
3.1 Panoramic solution of data flow, efficient cloud migration
For migrating to the cloud, we generally distinguish the data source first, including: the enterprise's own IDC, and other cloud service providers (such as AWS, Tencent Cloud or Alibaba Cloud, etc.). For customers such as enterprise-owned IDC, customers often want local data to be uploaded to our cloud object storage BOS.
We provide three methods, such as disk array hybrid cloud, as well as the moonlight treasure box mentioned just now. It is like a large U disk. After the data is copied locally, the large U disk is sent to the computer room of Baidu Smart Cloud by means of logistics, and our professional operators help customers to upload the data.
In addition, some customers may have a relatively large amount of data. At this time, they can also be migrated through our dedicated line service. For example, a dedicated line is drawn between the customer's IDC and Baidu's intelligent cloud computer room, so that the data can be efficiently transmitted to the BOS through the intranet.
For customers who have already migrated to other clouds, their migration involves cross-cloud migration. Users can use the data flow platform CloudFlow to initiate data migration and synchronization in one click visually. The user only needs to fill in the information of the source and destination, and also fill in the requirements for performance or storage path, etc., and click OK to start the migration task automatically.
In addition, for some special scenarios, such as the user wants to migrate the incremental data of other clouds to the BOS, the mirror back-to-source function can be enabled at this time. When the data is accessed, the data can be automatically synchronized to the BOS directly from other sources to help users achieve business continuity.
In addition to cross-cloud migration, we can also achieve cross-cloud synchronization.
Cross-cloud synchronization generally refers to the cross-cloud migration of incremental data. Users can configure an event-based notification function in CloudFlow to complete scheduled scan tasks. For example, every hour or a day to scan the source to see if some new data has been written, I can accurately migrate these incremental data to the BOS.
3.2 Intelligent life cycle management, optimal storage
Users are more concerned about the cost of storage. For object storage BOS, it has developed to EB-level physical space, tens of thousands of physical servers, and trillions of files. This scale is very large in China.
With the passage of time, for example, after half a year or a year or three years, the data may not be accessed by anyone, but the user must still store it.
Therefore, in response to such a demand, we have launched tiered storage, including standard storage with multiple AZs, standard storage, low-frequency storage with multiple AZs, low-frequency storage, or cold storage and archive storage.
With different storage types from left to right, the access frequency of the corresponding data decreases gradually. For frequently used hot data, standard storage is generally used. As it is accessed less frequently, low-frequency storage, cold storage, or archival storage can be gradually settled. Especially like archival storage, it is more aimed at scenarios that are accessed once every three years. For example, some data needs to be stored for a long time, such as genetic data, e-commerce live broadcast data, some compliance data that must be retained in response to inspections, and so on.
Object storage also provides a feature called "life cycle settlement" for optimizing costs.
For example, the data is initially hot data, that is, stored in standard storage. We can set a life cycle rule, for example, to settle from standard storage to low-frequency storage 30 days after upload, and further settle to archive storage after another 60 days. Users can set such a rule in advance, and when the settlement date arrives, the data will automatically settle. In terms of specific price, our coldest primary archive storage is only 18% of the unit price of standard storage, so the effect of reducing costs through settlement is very obvious.
In addition to subsidence, we also support life cycle uplift. For example, there may be a file now, which is a cold storage file. Generally speaking, the access frequency of cold storage files is relatively low, but there may be a situation where the access frequency of this file becomes very high within a period of time.
In this case, the user can set a life cycle floating rule, and through the automatic monitoring of BOS, when the cold data is frequently accessed, it will float to the upper storage type such as low-frequency storage and standard storage. Therefore, the use of life cycle management is very flexible, and users can choose the appropriate storage type according to their own needs, and set appropriate settlement rules at the same time.
A typical case, such as iQiyi's long videos, including movies, TV series, etc., are stored in the BOS. This data may start out as hot data, using standard storage. But when this data has not been accessed for a long time, it can be automatically deposited into cold storage. This rule helps iQIYI save a lot of usage costs.
At the same time, iQIYI distributes data through our CDN nodes to ensure that data can be distributed to terminals around the world.
3.3 Multi-level disaster recovery of data storage, safe and reliable
When customers use cloud storage, how to ensure the safety and reliability of data? Here we want to talk about two indicators of cloud storage.
The first one we call reliability. The reliability of the object storage BOS commitment is 12 9s, which is 99.9999999999%, which is a very high level, and the probability of data loss is one in 100 billion. How do we achieve high reliability? BOS has established a super-large-scale erasure code cluster to evenly distribute data to multiple AZs, which means that we can redundantly support the failure of N switches and the failure of a single AZ.
Another metric we call availability. For availability, the availability is 99.95% for the single-AZ storage type and 99.99% for the multi-AZ. But from the long-term experience, our real availability is 99.9995%, which is a very high level.
How do we guarantee this availability? BOS uses four-layer load balancing, and there is no single point in cluster mode. The data EC coding also guarantees multiple redundant reads. Moreover, the access layer can be extended horizontally, which further improves the usability of the product.
We provide multiple levels of disaster tolerance.
First of all, BOS has disaster recovery at the physical machine level. The bottom layer of BOS adopts a distributed storage architecture and adopts EC coding technology. If a physical machine is temporarily down due to network reasons or other reasons, the business can be automatically switched, and the user cannot perceive the downtime of the physical machine at all.
Second, we introduced the multi-AZ storage type**. **For example, like the standard storage multi-AZ mentioned just now, and the low-frequency storage multi-AZ, we store the data in multiple computer rooms at the same time. When a computer room is suddenly unavailable due to natural disasters and other reasons, BOS can implement disaster recovery switching at the computer room level. In addition, we can also achieve cross-regional backup and disaster recovery. We provide services in Beijing, Suzhou, Guangzhou, Baoding and other regions, and users can synchronize data to other regions in advance.
Finally, we provide the ability to mirror data back to the source. When the data does not exist in the primary origin site, it will automatically go to the secondary origin site to retrieve data.
3.4 Multi-product data flow linkage, easy to use
In the last part, I will introduce the application diversification mentioned earlier. A single product is increasingly unable to meet the needs of customers, and it is necessary to provide multiple products to form a complete set of solutions to help users solve problems.
Today, I will focus on introducing these three solutions to you. The first is the data lake acceleration solution in the big data scenario, the other is the solution in the hybrid cloud storage scenario, and the third is the HPC storage in the AI scenario.
The first one is the data lake acceleration solution in the big data scenario. Our data lake acceleration uses BOS as the base of the entire data lake. At the same time, we will have a data lake storage acceleration product called RapidFS, which will open up the data highway between computing and storage in big data scenarios.
Whether it is a big data scenario such as MapReduce or an AI scenario, the bottom layer can actually choose the object storage BOS to undertake the storage capacity of massive data. For big data scenarios, common scenarios include offline computing scenarios and online computing scenarios.
In offline computing scenarios, a typical example is website content recommendation. The browsing behavior of users on a website will form a lot of browsing data. For website manufacturers, these behaviors of users are often analyzed at night, so as to recommend content for users when they browse the website next time. We call this an offline training scenario.
There is also the online computing scene. Typically, when we are using some apps or web pages, we click on a search box to search for something, and the website/APP will perform online calculations on a series of user behaviors to optimize search results.
Offline scenarios often require low computing latency, so it is recommended to use the native-level Namespace architecture of BOS. Compared with S3 storage using flat Namespace, object storage using hierarchical Namespace has the atomicity of operations and is more friendly to frequent access to a large number of small files. At the same time, RapidFS can be used to cache hot data in nearby computing nodes to further accelerate data access.
For online computing scenarios, customers can install RapidFS components in the VPC. In addition to caching, you can also enable hierarchical namespaces within the VPC. Since the hierarchical Namespace is deployed in the VPC, the acceleration effect of the solution on the right will be better compared to the solution on the left in the figure below. For common operations such as rename, list, and delete in big data scenarios, the access performance will be greatly improved.
The other is our hybrid cloud storage. For example, this customer will have its own IDC. Because the local capacity is limited, customers hope to synchronize the old cold data to the cloud in some way. In this way, you can not only save some local space, but also use the hierarchical storage and life cycle capabilities of BOS on the cloud to reduce storage costs.
In this scenario, we provide a product such as storage gateway BSG. Users can deploy BSG to their local IDC, and connect local and cloud with one click. For example, after the BSG is deployed in the IDC, the user can mount a bucket of the BOS through the BSG, so that when the user writes data to the local IDC, what he sees may be a path written to his local IDC, but in fact This data has been written to the cloud. We can achieve compatibility of different protocols, and help users to build hybrid cloud storage without changing their usage habits.
The last scenario, we specifically target the AI scenario. In this scenario, we also recommend using the object storage BOS as the data base, and at the same time collocating the parallel file system PFS on the upper layer. In AI scenarios, more operations are mainly based on reading data. For example, during AI training, there will be many operations to read data sets.
Specifically, this scheme will have three characteristics. First, we include an acceleration layer compatible with the POSIX interface, PFS based on local disks and all-flash hardware; in addition, we can automate the preparation of resources and data sets, and deeply integrate with the scheduler to reduce the complexity of use; third, in the When training data, you can configure different data loading strategies, such as preloading, loading on first access, etc.
——————————END——————————
Recommended reading:
The Practice of Augmented Analytics in Baidu Statistics
Introduction of Baidu Secure Communication Protocol bdtls Based on TLS 1.3
Real-time data warehouse practice of Baidu user product flow and batch integration
How to manage waste of resources? Baidu Cloud Native Cost Optimization Best Practices
Data Lake Acceleration Solution for Big Data Storage and Computing Separation Scenarios
Baidu APP Android package size optimization practice (3) Resource optimization
Analysis of ffplay video playback principle