GlusterFs the type of volume analysis and creation, use (in conjunction kubernetes cluster analysis)

introduction

Through analysis and comparison of volume types, to help readers choose to mount a production environment that best meet storage service, commands can be combined with " GlusterFS Detailed and kubernetes build-GlusterFS heketi " experiment, the following entry to the topic

Several types of roll

• Base volume: cloth reel (distribute), striped volumes (stripe), the replicated volume (replica), an error correction volume (Dispersed)
• Composite volume: volume distributed strip (distribute stripe), distributed replicated volume (distribute replica), stripe copy volume (stripe replica), distributed replicated volume stripe (distribute stripe)

First, the basic volume

The following create mount volumes can be viewed through the following command to start, stop, delete

#查看已创建挂载卷
gluster volume info
#启动挂载卷
gluster volume start gv0
#删除前，先停止挂载卷
gluster volume stop gv0
#删除挂载卷
gluster volume delete gv0 

1. cloth reel (distribute voulme)

Distributed mode, both DHT, is the default mode of GlusterFS, when you create a volume, the default option is to create a distributed volume. In this mode, not the file into blocks, but is distributed to all the brick server through the hash algorithm, but expanded disk space, similar to the window spanned

distribute voulme Features

• Files are distributed in different servers, does not have the redundancy.
• More easily and inexpensively expanded size of the volume.
• Single point of failure can cause loss of data, do not have fault tolerance.
• Rely on the underlying data protection.

1.1. Create a command

gluster volume create gv0 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0

2. a striped volume (stripe volume)

Banding patterns, either Striped, similar RADI 0, in this mode, the offset amount of the file divided into N blocks of (N nodes strip), is stored in the Server node polling each Brick. Each node of the data block into the local file system as a normal file, and the total number of blocks of each block number by the extended attribute record. When configuring the specified number of strips must be equal to the number of volumes included in the storage server Brick, when storing large files, especially performance, but does not have redundancy.

stripe volume Features

• More data is divided into blocks of different pieces distributed stripe server farm.
• Reducing the file load distribution and smaller accelerate the speed of access.
• No data redundancy, does not have redundancy.

• Single point of failure can cause loss of data, do not have fault tolerance.

  ⚠️RAID0 called striping, will each disk, the write data can be processed in parallel fragmentation, therefore N times the write rate of a single disk, there is no redundancy, no damage would result in a disk the data is not available.

  RAID 0 characteristics:

  • Distribution data strip formula
  • No redundancy, the best performance (no storage mirroring, parity information)
  • It can not be applied to data security for high occasions

2.1. Create a command

gluster volume create gv0 stripe 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0

3. Copy Volume (replica volme)

Copy mode, both the AFR, file-level RAID 1, that is, the same file saved one or more copies, saving the same content and directory structure on each node. Because you want to save a copy of the copy mode, so the lower disk utilization, if the storage space on multiple nodes are inconsistent, it will take a minimum capacity of a node in accordance with the total capacity of the volume barrel effect. Replication volume having redundancy, even if a node is damaged, it does not affect the normal use of the data.

replica volme Features

• All volume servers hold a complete copy.
• Created by the customer when the number of copies of the volume of decisions.
• Disk utilization is low.

• With redundancy

  RADI1 is mirrored storage, no data check, data is written to the same two or more disks.

  RAID 1 features:

  • Provide data redundancy block
  • Relatively slow write speed, but the faster read speed

3.1. Create a command

gluster volume create gv0 replica 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0

4. Volume Correction

Dispersed Volume is based on a new type of ErasureCodes (error correction code) of Volume (3.6 version released this feature), similar to RAID5 / 6. Redundancy configuration improves reliability by (redundant) level, while ensuring high reliability, can enhance the utilization of physical storage space.
For details, please see " GlusterFS Dispersed Volume (volume correction) summary "

4.1. Create a command

gluster volume create gv1 disperse 4 redundancy 1 10.8.4.92:/var/lib/heketi/mounts/gv1 10.8.4.93:/var/lib/heketi/mounts/gv1 10.8.4.131:/var/lib/heketi/mounts/gv1 10.8.4.132:/var/lib/heketi/mounts/gv1

Second, the composite volume

1. Distributed striped volume (distribute stripe volume)

It requires a minimum of four servers can be created. brickserver number is a multiple of the number of tape strips, characteristics of both stripe and distribute volumes, DHT is combined with a Striped type.

1.1. Create a command

gluster volume create gv0 stripe 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0 10.8.4.131:/var/lib/heketi/mounts/gv0 10.8.4.132:/var/lib/heketi/mounts/gv0

2. Distributed Replicated Volume (distribute replica volume)

It requires a minimum of four servers can be created. brickserver number multiple of the number of mirrors, and distribute characteristics of both replica volumes, can replicate data between two or more nodes, DHT is combined with AFR type.

2.1. Create a command

gluster volume create gv0  replica 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0 10.8.4.131:/var/lib/heketi/mounts/gv0 10.8.4.132:/var/lib/heketi/mounts/gv0

3. The tape volume copy (stripe replica volume)

It requires a minimum of four servers can be created. Similarly RAID 10, the AFR is Striped combination type.

First the composition of RAID1, then the RAID0

RAID 10 (also known as RAID 1 + 0) Features:

• Press RAID 0 divided into two groups, the two groups were then mirrored RAID 1 by way
• Both redundant (mirrored memory provided) and performance (data stripe-shaped distribution)

3.1. Create a command

gluster volume create gv0 stripe 2 replica 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0 10.8.4.131:/var/lib/heketi/mounts/gv0 10.8.4.132:/var/lib/heketi/mounts/gv0

4. Copy volume distributed strip (distribute stripe volume)

Requires at least 8 servers can be created. Mixed three models.

4.1. Create a command

gluster volume create gv0 stripe 2 replica 2 10.8.4.92:/var/lib/heketi/mounts/gv0 s10.8.4.93:/var/lib/heketi/mounts/gv0 10.8.4.131:/var/lib/heketi/mounts/gv0 10.8.4.132:/var/lib/heketi/mounts/gv0 exampleNode5:/var/lib/heketi/mounts/gv0 exampleNode6:/var/lib/heketi/mounts/gv0 exampleNode7:/var/lib/heketi/mounts/gv0 exampleNode8:/var/lib/heketi/mounts/gv0

Third, the combination of comparative advantages and disadvantages of cluster

1. binding " GlusterFS Detailed structures and kubernetes-GlusterFS heketi " article explains

1.1 replication volume:volumetype: replicate:3

Overview:

• A single node or a hard disk linked loss does not affect the use of stored data is not lost
• For kubernetes StatefulSet type of service, such as this inherently es service data synchronization, although the automatic construction volume reached a purpose, but a number Brick1 node data back-up copy x number of volumes, more harm than good.

1.2 Distributed volumes:volumetype: none

Overview:

• For high-availability service can synchronize data, recommended this volume.
• For general service (Deploy), this is not the type of fault tolerance and redundant data.
• Facilitate the expansion of disk space

1.3 volume correction:volumetype: disperse:4:2

Overview:

• For general service (Deploy), this type has the high availability, and improved utilization of storage space, is recommended.

2. With regard to manually create volume, PV, Overview

Gluster: Quick Start Guide
Follow the official website set up, very simple, is not repeated.

The main steps:

• Creating Luo disk
• disk partition
• Disk Mount
• glusterd peer probe server2New gluster pool node
• gluster create volume

 For critical data storage, three storage mode kubernetes provided are not met, you can create volume, PV manually. Author bands tend to copy volumes (stripe replica volume), usually with a copy volume (distribute stripe volume) with strips distributed more.

⚠️ consolidation analysis is not easy, if you have help, attention author, leave your love, as well as technical follow-up of dry material! ! !

Fourth, data distribution

The following quote from " the GlusterFS volume type and data distribution analysis " herein can be directly read or jump to read

1. Distributed volume

In this mode, the file does not block, file stored directly on a server node. "There is no reinventing the wheel," the phrase good summary of the design idea of ​​this GlusterFS. Because it uses existing local file system to store files, so a lot of linux common commands and tools can continue normal use. This makes GlusterFS can develop in a more stable basis, more easily accepted by people. Because of the need to use extended file attributes, which currently supports the underlying file system: ext3, ext4, ZFS, XFS and so on.

Due to the use of the local file system, on the one hand, and there is no access efficiency did not improve, but rather because of network traffic and reduced; on the other hand, support for very large files be difficult. Although ext4 support a single file can have a maximum of 16T, but the capacity of the local storage device is limited. So if there are a lot of large file storage needs, consider using Stripe pattern to implement, such as considering new stripe volume dedicated to storing very large files.

Function:
the file stored in the server, as shown above, is stored in the File1 and File2 server1, and stored in File3 Server2, files are stored in random

2. Stripe volume

Stripe pattern corresponds RAID0 fact, in this mode, the system simply on the file offset into N blocks of (N nodes stripe), and then sent to each server node. The server nodes are stored in each of a normal file as a local file system, and extended attribute record with the total number (stripe-count), and the block number of each block (stripe-index). stripe number must equal the number stored in the server brick contained volume, the file is divided into data blocks, to store the Round Robin manner bricks, and transmits the data block size, a big file performance

Function:
The files are stored in different servers, the above figure, File is divided into 6 segments, 1,3,5 server1,2,4,6 placed on server2

3. Replicated volume

Replicated mode, also called AFR (AutoFile Replication), RAID1 equivalent, i.e., the same file stored on a plurality of multiple image storage nodes, each child node has the same replicated directory structure and file. replicated mode is generally not used alone, often in "Distribute + Replicated" or "Stripe + Replicated" form appears. If the storage capacity of two different machines, such as barrel effect then, the storage capacity of the system is determined by the small capacity machines. the number of replica servers must be equal to the number stored in the volume contained brick, high availability. Twenty-two create a mutually-volume backup, storage pool hard disk is damaged, does not affect the use of the data, it requires a minimum of two servers to create a mirrored volume distribution.

Replicated mode is the responsibility carried out at the level of the file (compared to HDFS), and when you create a volume determines the volume of each server node, and can only be adjusted manually. In this case it is relatively inflexible, if a node A out of the question, it must use the new node to replace A, otherwise there will be some problems hidden.

In Replicated mode, each file will have the following several extended attributes:

Read and write data, the specific situation is as follows:

• When reading data: the system will request load balancing to all of the mirrored storage node, while the mechanism is triggered self-heal when the file is accessed, then the system will detect a copy of the agreement (including directories, file content, file attributes Wait). If the inconsistency by changelog will find the correct version, and then repair the file or directory attributes to ensure consistency.
• Write data: the first server to server as a lock, to lock a file or directory, write changelog record the event, and then write data on each mirror node, to ensure the consistency, erasing changelog record, unlock the lock.

If multiple nodes are mirror images in a mirror node there is a problem, the user read / write requests can be normal, and will not be affected. And the question node is replaced, the system will automatically synchronize data in the background to ensure consistency copy. But the system does not automatically need to find another node to replace it, but need to manually add a node, the administrator must promptly to find these problems, otherwise it is difficult to ensure reliability.

Function:
The files are stored in the server, as shown above, File1 exists server1 and server2 Meanwhile, File2, too, server2 equivalent document is a copy of the file in server1.

4. distribute stripe volume

Distributed striped volume, the number of server storage volume contained in the brick must be a multiple stripe (> = 2 times), both distributed and tape-like function. Each file distributed on four shared server, typically used for large file access processing, it requires a minimum of four servers to create a striped volume distribution.

Function:
stores files in different servers, as shown above, File divided into four sections, 1, 3 server1 (EXP1), the 2,4 on server1 (exp2). backup file server2 (exp3) 1,3 storage server1 (exp1) the backup file server2 (exp4) 2,4 storage server1 (exp2) in.

5. distribute replica volume

Distributed replication volume, the number of server storage volume contained in the brick must be a replica of a multiple (> = 2 times), both distributed and replicated functions.

Function:
the random file backup resides on a server, as shown above, server1 (exp1) stored files File1, Server1 (exp2) to store files File2. server2 (exp3) to store backup files File1, server2 (exp4) to store the backup files in File2.

6. stripe replica volume

Function:
The backup file is divided and stored in different servers random, the above chart, is divided into 4 sections File, 1,3 stored on server1 (exp1), 2,4 stored in the server2 (exp4), (exp3 on server1 ) storage server2 (exp4) of backup files, storage server1 (exp1) backup file on server2 in (exp2).

7. distribute stripe replica volume

Distributed striped volume copy data copy distribution strip roll cluster. For best results, you should use distributed at high concurrent parallel stripe copy volume environment very large file access and performance are critical.

Function:
The backup file is divided and stored in different servers in a random, as shown above, is divided into four sections File, 1,3 stored in the server1 (exp1) in 2,4 stored in the server2 (exp3). backup file server1 (exp2) storage server1 (exp1) of, server2 (exp4) storage server2 (exp3) backup file.