Distributed file storage

1 Common terminology

1.1 Backup technology

An extra copy of data created for data recovery purposes

Category: direct connection backup, network connection backup, off-LAN backup, off-server backup
online backup, offline backup, near-line backup

① cold backup

When the cold backup is performing operations, the server no longer accepts data updates from users or applications, which solves the problem of concurrent updates when backup replication is in progress. The cold backup time is too long, and the server no longer accepts any update operations

② Hot backup

When users and applications are updating data, the system can also perform backups, and there are integrity issues; every file or database entity needs to be backed up as a complete and consistent version

③File level backup

File-level backup: The advantage of using the server operating system and file system to back up files is that it is easy to restore one or some specific files. During backup, the operating system and applications can continue to access files, but it takes a long time. For example, backing up a large number of small document recommendations, the access overhead of the operating system and file and directory metadata is high, and the backup of "open files" cannot be solved

④Block level backup

Image-level backup, completely bypassing files and file systems, using data as raw disk blocks, reading data directly from the disk, regardless of how many files there are, and not going deep into the file, and reading in the order of the blocks on the disk, Rather than the order in the file, so there will be no performance degradation caused by seeking when reading file data with a large number of fragments. Block-level backup usually has built-in instant snapshot support, which can fundamentally solve the "open file" problem. By performing special operations, the running applications and their data in the server memory will enter a consistent state and flush to the disk. Then take a snapshot, the backup process reads the changed data blocks from the snapshot instead of the actual volume, and deletes the snapshot after the backup is complete

⑤ active-active

There are generally three backup methods: hot backup, cold backup, and active-active backup between the primary and backup data centers.

In the case of hot standby, only the main data center undertakes the business of users, and the main data center is backed up without stopping.

In the case of cold standby, only the main data center undertakes business, and the main data center is backed up in case of shutdown.

Active-Active thinks that it is too wasteful for the backup data center to only do backup, so both the main and backup data centers undertake the business of users at the same time. At this time, the main and backup data centers back up each other and perform real-time backup. Generally speaking, the load of the primary data center may be higher, for example, sharing 60% to 70% of the business, and the backup data center only shares 40% to 30% of the business.

⑥Three Centers in Two Places

Two places = local + remote

Three centers = local center + local disaster recovery center + remote backup center

1.2 Snapshot technology

Snapshot technology is a method of creating data backups in a storage system. It can record the data state at a specific point in time on the storage device, and can restore to the state at that point in time when needed. Snapshot technology enables efficient backup and recovery operations by recording changes to data rather than duplicating entire data sets. Snapshot technology can help users quickly restore to the previous data state in the event of data loss, data corruption or other failures.

1.3 Continuous Data Protection Technology

Continuous data protection technology is a method of data backup and recovery, which provides more accurate and timely data protection by recording data changes in real time. Unlike traditional periodic backups, continuous data protection technology can be used 每次数据变化时立即备份数据to reduce the risk of data loss. Continuous data protection technology usually uses incremental backup to record data changes, and can provide flexible recovery point selection, so that users can choose any point in time for data recovery.

1.4 RAID technology

RAID (Redundant Array of Independent Disks) technology is a storage technology that combines multiple independent hard disks to provide higher data reliability and performance. RAID technology protects data integrity by distributing data on multiple hard disks and using redundant checksum information. Different RAID levels provide different data protection and performance characteristics, such as RAID 0 provides high performance but no redundancy protection, RAID 1 provides data redundancy but no performance

1.5 Multipath Technology

Multipathing is a method of using multiple independent paths in a storage system to provide higher reliability and performance. Multipathing can be accomplished by using multiple network connections, multiple storage controllers, or multiple storage devices. 当一个路径发生故障时，多路径技术可以自动切换到另一个可用的路径，从而保证存储系统的连续性和可用性. Multipathing can also improve storage system performance by using multiple paths simultaneously, for example by transferring data in parallel or load balancing to provide higher throughput and responsiveness.

1.6 Virtualization Technology

Virtualization technology is a method of abstracting physical resources into virtual resources to provide higher resource utilization and flexibility. In the field of storage, 虚拟化技术可以将多个物理存储设备组合成一个逻辑存储池，并为用户提供统一的存储接口. Virtualization technology can hide the details of the underlying storage device, so that users can manage and use storage resources more conveniently. Virtualization technology can also provide advanced functions such as snapshot, clone, compression, and deduplication to improve storage efficiency and data management flexibility.

1.7 Data replication and disaster recovery

1.7.1 Recovery Time Objective (RTO, Maximum Tolerable Service Downtime)

Recovery Time Objective (RTO: Recovery Time Objective), which takes the application as the starting point, that is, the recovery time objective of the application, mainly refers to the longest time that the application can be tolerated out of service, that is, from the occurrence of a disaster to the recovery of the business system The minimum time period required by the function. RTO is an indicator reflecting the timeliness of business recovery, indicating the time required for business to return to normal from interruption. The smaller the value of RTO, the stronger the data recovery capability of the disaster recovery system;

1.7.2 Recovery Point Objective (RPO, Maximum Tolerable Data Loss)

Recovery Point Objective (RPO: Recovery Point Objective), RPO is an index that reflects the integrity of restored data. It takes data as the starting point and mainly refers to the amount of data loss that the business system can tolerate

1.7.3 Active/Passive and Active-Active

①Active/Passive:

A way of data backup, the data is mainly stored in the main data center, and the backup data center only plays a backup role when the main data center fails or shuts down. During the failure of the primary data center, the backup data center will not be used for real-time data access or application use.

②Hyperactive: Two data centers undertake business.

Active-Active thinks that it is too wasteful for the backup data center to only do backup, so both the main and backup data centers undertake the business of users at the same time. At this time, the main and backup data centers back up each other and perform real-time backup. Generally speaking, the load of the primary data center may be higher, for example, sharing 60% to _{70% of the business, and the backup data center only shares 40% to} 30% of the business.

[Transfer] A second understanding of stand-alone, cluster, hot standby, hard disk array
insert image description here

1.8 Bucket object query

Object: An object is the basic unit of a bucket, which can be any type of file or data. Each object has a unique key (key) for identification and retrieval.

①Lifecycle Management

Lifecycle management is a strategy for automatically managing objects. Objects can be automatically moved from buckets to other storage tiers by defining rules and conditions. Such as: Infrequent Access Storage, Archive Storage, etc., to optimize storage cost and performance.

②Fragmentation Management

When objects are modified or deleted, it may result in discontinuous space in the bucket. Fragmentation management can optimize storage space utilization by reorganizing or compacting buckets.

③Tags

Tags are metadata used to categorize and organize objects in a bucket. You can add custom tags to objects, such as project name, owner, expiration date, etc., to manage and query objects more conveniently.

1.9 Others

①workflow

In file storage, workflow (workflow) refers to a series of operations and steps used to manage the creation, storage, access, sharing and processing of files. It describes the tasks and activities involved in the entire process of a record from its creation to its final processing and archiving.

Create a file

select storage location

save file

Organization and Classification

access and share

Collaboration and Version Control

processing and modification

review and approval

Archiving and archiving

destroy or delete

② EOS(Enterprise Object Storage)、OSG、XMS、3DS

EOS is the abbreviation of Enterprise Object Storage, which means enterprise-level object storage. It is a storage solution for storing and managing large-scale data. Object storage is a storage model that differs from traditional file systems or block storage in that it stores data as independent objects, each with a unique identifier (usually a URL).

OSG (Open Storage Gateway): Open Storage Gateway is a software solution for connecting local storage and cloud storage. OSG provides a unified interface and protocol, so that applications can seamlessly access local storage and cloud storage, and achieve high availability and flexibility of data.

XMS (eXtensible Metadata System): Extensible metadata system is a framework for managing and storing metadata. XMS can help organizations efficiently manage and retrieve large amounts of metadata to support tasks such as data management, data analysis, and data governance. It provides a flexible architecture and interfaces that can be extended and customized as needed.

3 three-dimensional data management system

③ETag、Key、MetaData、Bucket、Grok等

ETag (Entity Tag): ETag is a mechanism used to identify resources in the HTTP protocol. It is a unique identifier generated by the server to determine whether a resource has changed. When a client requests a resource, the server returns the ETag value of the resource to the client. The client can use the ETag value in subsequent requests to verify whether the resource has changed, thereby avoiding unnecessary data transmission.

Key (key): In the storage system, Key is a string used to uniquely identify a storage object. It is similar to the primary key in the database and is used to locate and access specific objects in the storage system. Keys are usually specified by users and can be designed according to business requirements.

Bucket (storage bucket): Bucket is a container for organizing and managing storage objects. It is similar to a folder in a file system and can contain multiple storage objects (such as files). Buckets are usually created by storage system providers, where users can create, delete, and manage storage objects.

Metadata: Metadata is the data that describes the storage object. It contains attribute information about the object, such as size, creation time, modification time, etc. Metadata is usually stored together with storage objects, and can be retrieved and modified through the storage system's API.

ACL (Access Control List): ACL is an access control mechanism used to define the access rights of storage objects. Through ACL, users can set read and write permissions for storage objects to control who can access and modify objects.

CDN (Content Delivery Network): CDN is a distributed network architecture used to accelerate the transmission and distribution of static content. By caching content to nodes closer to users, CDN reduces the delay of data transmission and improves the user's access speed and experience.

Replication: Replication is a data protection and disaster recovery technology that improves data reliability and availability by replicating data to multiple storage nodes. When a node fails, data can be obtained from other nodes to ensure data persistence and recoverability.

Grok is an expression for parsing and matching text data. It is a powerful pattern matching engine developed by the Elasticsearch community and is often used in log analysis and data processing.

# 以下是一个使用 Grok 表达式解析 Apache 访问日志的示例：
%{
    
    IP:client} %{
    
    USER:ident} %{
    
    USER:auth} \[%{
    
    HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{
    
    NUMBER:response} %{
    
    NUMBER:bytes} "(?:%{URI:referrer}|-)" "%{DATA:agent}"

The difference between ETag (entity tag) and Version (version control):

ETag: entity tag

ETag is a unique server-generated identifier that identifies a specific version of a resource.

The ETag can be generated based on the hash algorithm of the resource content, or based on other rules.

When a client requests a resource, the server returns the ETag value of the resource to the client.

The client can use the ETag value in subsequent requests to verify whether the resource has changed, thereby avoiding unnecessary data transmission.

ETags are usually suitable for fine-grained control or verification of resources, such as the implementation of caching mechanisms.

Version: version control

Version control is used to identify different states or modification history of resources by maintaining different versions of resources.

Versioning can describe different versions of a resource based on timestamps, incremented version numbers, or other mechanisms.

Whenever a resource changes, a new version is created and the corresponding change information is recorded.

Clients can fetch or roll back to a specific version based on the resource's version number or history.
Version control is often suitable for complete history and management of resources, such as team collaboration or code version management systems.

The main difference between ETag and version control lies in the generation method and application scenarios:

ETag is generated based on resource content, suitable for fine-grained control and verification of resources, and can be used for caching, conditional requests, etc.; while version control is generated based on resource change history, suitable for managing and tracking different versions and changes of resources .
ETag is mainly used to control caching and optimize network transmission, while version control is mainly used for team collaboration, code management and other scenarios that require complete history and version rollback.

④HCI、SDS、RAID、SAN

HCI (Hyper-Converged Infrastructure): Hyper-converged infrastructure is an architecture that integrates computing, storage, and network functions into one server. HCI provides simplified management and scaling, and highly integrated computing and storage resources.

SDS (Software-Defined Storage): Software-defined storage is an architecture that implements storage functions through software. SDS can decouple storage functions from hardware, providing flexibility, scalability, and cost-effectiveness.

RAID

RAID (Redundant Array of Independent Disks): Redundant Array of Independent Disks is a technology that provides data redundancy and performance enhancement by combining multiple disks together. RAID can provide high reliability and high performance storage solutions.

SAN (Storage Area Network): Storage Area Network is a high-speed network architecture used to connect storage devices and servers. SAN provides high-bandwidth, low-latency storage connections, and supports multiple servers accessing shared storage resources at the same time.

2 Common distributed file systems

2.1 GFS（Google File System）

Google GFS file system, a scalable distributed file system for large-scale data-intensive applications. Although GFS runs on cheap common hardware devices, it still has the ability to provide disaster redundancy and provide high-performance services for a large number of clients. GFS is a large data distributed storage framework designed for Google internally. It was proposed by a paper published by Google in 2003. Unfortunately, Google did not open source it, but based on the design concept of GFS, many excellent open source GFS files were born. system.

2.2 HDFS（Hadoop Distributed File System）

HDFS is a dedicated file system for Hadoop (Apache open source project). Hadoop is a big data computing framework that allows distributed processing of large data sets across computer clusters using a simple programming model. It is the implementation of the MapReduce algorithm published by Google. HDFS is GFS open source implementation.

2.3 Ceph

Ceph is a new generation of free software distributed file system designed by Sage Weil of the University of California, Santa Cruz (co-founder of DreamHost) for his doctoral dissertation, and used Ceph to complete his thesis. Ceph's main goal is to design a distributed file system based on POSIX without a single point of failure, enabling data to be fault-tolerant and seamlessly replicated. Compared with other distributed file systems, ceph has the highest performance because it is written in C++. Because ceph uses the btrfs file system, and the btrfs file system requires Linux 2.6.34 or higher kernels to support it. After graduating in 2007, Sage began to devote himself full-time to the development of Ceph to make it suitable for the production environment. After more than ten years of development, Ceph has become mature.

2.4 Lustre

Luster is an open source, distributed and parallel file system software platform, featuring high scalability, high performance, and high availability. Luster's architectural goal is to provide a globally consistent POSIX-compliant namespace for large-scale computing systems, including the world's most powerful high-performance computing systems. It supports hundreds of PB of data storage space and hundreds of GB/s or even several TB/s of concurrent aggregated bandwidth.

2.5 GridFS

GridFS is MongDB's distributed storage system for storing and restoring files (such as pictures, audio, video, etc.) that exceed 16M (BSON file limit). GridFS is also a way of file storage, but it is stored in a collection of MonoDB. GridFS can better store files larger than 16M. GridFS will divide large file objects into multiple small chunks (file fragments), generally 256k each, and each chunk will be stored in the chunks collection as a document of MongoDB. GridFS stores a file in two collections: fs.files and fs.chunks. The actual content of each file is stored in chunks (binary data), and the meta data related to the file (filename, content_type, and user-defined attributes) will be stored in the files collection.

2.6 MogileFS

MogileFS is an open source distributed file storage system developed by Danga Interactive, a subsidiary of LiveJournal. At present, there are many companies using MogileFS, such as the top Internet companies in Japan and domestic Yupoo, digg, Douban, Dianping, Sogou, etc., which manage a large number of pictures for their organizations or companies. Taking Dianping as an example, all user pictures are stored in MogileFS, and the data volume has reached more than 500TB

2.7 FastDFS（Fast Distributed File System）

FastDFS is an open source distributed file system written in pure C with high performance. She manages files, and its functions include: file storage, file synchronization, file access (file upload, file download), etc., which solves the problems of large-capacity storage and load balancing. It is especially suitable for online services with files as the carrier, such as photo album websites, video websites and so on.

2.8 TFS（Taobao File System）

TFS is a highly scalable, high-availability, high-performance, Internet service-oriented distributed file system, mainly for massive unstructured data, it is built on an ordinary Linux machine cluster, and can provide high reliability and high concurrency for the outside world storage access. TFS provides Taobao with massive small file storage, usually the file size does not exceed 1M, which meets Taobao's demand for small file storage and is widely used in various applications of Taobao. It adopts HA architecture and smooth expansion to ensure the availability and scalability of the entire file system. At the same time, the flat data organization structure can map the file name to the physical address of the file, which simplifies the file access process and provides good read and write performance for TFS to a certain extent.

2.9 NFS（Network File System）

NFS is a distributed file system protocol for sharing files and directories over a network. It allows remote computers to access and manipulate shared files over the network as if they were local files. NFS is a client-server protocol in which the server maintains file systems that store files, and clients can mount and access these file systems over the network. NFS is widely supported on many operating systems and storage devices, and is often used in scenarios where files and data are shared.

3 Introduction to Common Storage Service Software

3.1 Object Storage

①S3（Simple Storage Service）

S3 (Simple Storage Service) is a highly scalable object storage service provided by Amazon Web Services (AWS). It is widely used to store and retrieve various types of data including text files, images, videos, database backups, application logs, etc.

S3 features:

object storage
high scalability
Data Persistence and Reliability
data security
buckets and keys
Data Access and Transfer
Data Lifecycle Management
version control
event

Reference article: https://blog.csdn.net/qq_37862148/article/details/113999206