World trend, long period of division, together for a long score. I find this statement really set into the history of all things.
From the development history storage point of view, all previously stored locally directly access the hard disk of a local computer (DAS- direct attached storage). Later, in order to share data, increase IOPS, increase Raid, LVM and other means to improve the availability and data protection, and the emergence of centralized storage (disk array). With the X86 server getting cheaper, scalable centralized storage shortage, can not cross-brand integration across models and other issues, began to appear a distributed storage system X86 server-based, distributed storage software as the core.
In the formation of open source software, and is distributed Ceph storage leader, attention is distributed Ceph storage software, rather than a distributed block storage, where a distributed file storage, distributed object store, but three kinds They are supported.
This introduction and build experience using Ceph distributed storage cluster.
Experimental environment: CentOS7
CPU name |
IP addresses |
ceph-admin |
10.1.30.43 |
ceph-node1,ceph-mon |
10.1.30.37 |
ceph-node2,ceph-mon |
10.1.30.38 |
ceph-node3,ceph-mon |
10.1.30.39 |
Ceph-node of the machine, each plate outside the system, add . 1 disks (not less than 20G )
1, environment preparation
1.1 modify the hostname
hostnamectl set-hostname ceph-admin hostnamectl set-hostname ceph-node1 hostnamectl set-hostname ceph-node2 hostnamectl set-hostname ceph-node3
1.2 modify the hosts table
vi /etc/hosts
10.1.30.43 ceph-admin 10.1.30.37 ceph-node1 10.1.30.38 ceph-node2 10.1.30.39 ceph-node3
1.3 turn off the firewall and selinux
systemctl stop firewalld systemctl disable firewalld setenforce 0 vi /etc/selinux/config SELINUX=disabled
1.4 Configuring NTP
yum install ntp ntpdate ntp-doc -y
systemctl restart ntpd systemctl status ntpd
1.5 Configuration yum source
Yum backup of the original file
mkdir / mnt / bak etc. /etc/yum.repos.d/* / mnt / rear /
Download Ali cloud base source and epel source
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
Add ceph source
vim /etc/yum.repos.d/ceph.repo
[ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ gpgcheck=0 priority =1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/ gpgcheck=0 priority =1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS gpgcheck=0 priority=1
Each node is created cephuser user to set sudo permissions
useradd -d /home/cephuser -m cephuser echo "cephuser"|passwd --stdin cephuser echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser chmod 0440 /etc/sudoers.d/cephuser sed -i s'/Defaults requiretty/#Defaults requiretty'/g /etc/sudoers
Test cephuser of sudo privileges
su - cephuser sudo su -
1.6 Configuring SSH-free access secret
ceph-admin generating public and private key files on the node, then ceph-admin node .ssh catalog copy to other nodes
su - cephuser ssh-keygen -t rsa # all the way round cd .ssh/ cp id_rsa.pub authorized_keys
Copy to the node Node
scp -r /home/cephuser/.ssh ceph-node1:/home/cephuser/ scp -r /home/cephuser/.ssh ceph-node2:/home/cephuser/ scp -r /home/cephuser/.ssh ceph-node3:/home/cephuser/
Verify that SSH can have free access to secret
ssh -p22 cephuser@ceph-admin ssh -p22 cephuser@ceph-node1 ssh -p22 cephuser@ceph-node2 ssh -p22 cephuser@ceph-node3
1.7 Preparation disks
( Ceph-node1 , Ceph-node2 , Ceph-node3 three nodes)
Do not use the test disk is too small, otherwise it will error when you add disks later, recommended disk size is 20G and above.
Experiment 3 nodes are VMware virtual machines, each directly add a new disk to 20G. (Note vmware add is / dev / sdb, the original system disk is / dev / sda, if you are using physical machines or other virtualization software, please note the actual modification)
Each node within the operating system disk check
sudo fdisk -l /dev/sdb
Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
Formatting a disk
sudo parted -s /dev/sdb mklabel gpt mkpart primary xfs 0% 100% sudo mkfs.xfs /dev/sdb -f
View disk format ( XFS format)
sudo blkid -o value -s TYPE /dev/sdb
2, installation and deployment
(Only ceph-admin for Ceph-Deploy )
su - cephuser
Installation ceph-deploy
sudo yum update -y && sudo yum install ceph-deploy -y
Create a cluster directory
mkdir cluster cd cluster/
Creating a cluster (fill in behind monit host name of the node, where monit node and management node is the same machine that Ceph-ADMIN )
ceph-deploy new ceph-admin
Modify ceph.conf file (note: mon_host must be public Network ! Network is the same network segment)
vim ceph.conf # Add the following two lines of configuration content
public network = 10.1.30.43/24 osd pool default size = 3
Installation Ceph (process a little long, we need to wait for some time .... )
ceph-deploy install ceph-admin ceph-node1 ceph-node2 ceph-node3
Initialization monit monitoring node, and collect all the keys
ceph-deploy mon create-initial ceph-deploy gatherkeys ceph-admin
Adding OSD to the cluster, check the OSD all available disks on the node
ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3
Use zap option to delete all osd partitions on the node
ceph-deploy disk zap ceph-node1:/dev/sdb ceph-node2:/dev/sdb ceph-node3:/dev/sdb
Prepare the OSD (using the prepare command)
ceph-deploy osd prepare ceph-node1:/dev/sdb ceph-node2:/dev/sdb ceph-node3:/dev/sdb
Activate the OSD (Note Because ceph the disk partition, / dev / sdb disk partition / dev / sdb1 )
ceph-deploy osd activate ceph-node1:/dev/sdb1 ceph-node2:/dev/sdb1 ceph-node3:/dev/sdb1
---------------------------------------------------------------------------------------------
The following error may occur:
[ceph-node1][WARNIN] ceph_disk.main.Error: Error: /dev/vdb1 is not a directory or block device [ceph-node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/vdb1
However, this error did not affect ceph deployment in three osd by the node command displays the disk has been successfully Mount :
[cephuser@ceph-node1 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 4.2G 0 rom sda 252:0 0 70G 0 disk ├─sda1 252:1 0 1G 0 part /boot └─sda2 252:2 0 69G 0 part ├─centos-root 253:0 0 43.8G 0 lvm / ├─centos-swap 253:1 0 3.9G 0 lvm [SWAP] └─centos-home 253:2 0 21.4G 0 lvm /home sdb 252:16 0 20G 0 disk ├─sdb1 252: 17 0 15G 0 part / var / lib / ceph / osd / ceph-0 # mounts to succeed └─sdb2 252:18 0 5G 0 part
[cephuser@ceph-node2 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 4.2G 0 rom sda 252:0 0 70G 0 disk ├─sda1 252:1 0 1G 0 part /boot └─sda2 252:2 0 69G 0 part ├─centos-root 253:0 0 43.8G 0 lvm / ├─centos-swap 253:1 0 3.9G 0 lvm [SWAP] └─centos-home 253:2 0 21.4G 0 lvm /home sdb 252:16 0 20G 0 disk ├─sdb1 252: 17 0 15G 0 part / var / lib / ceph / osd / ceph-1 # mount successful └─sdb2 252:18 0 5G 0 part
[cephuser@ceph-node3 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 4.2G 0 rom sda 252:0 0 70G 0 disk ├─sda1 252:1 0 1G 0 part /boot └─sda2 252:2 0 69G 0 part ├─centos-root 253:0 0 43.8G 0 lvm / ├─centos-swap 253:1 0 3.9G 0 lvm [SWAP] └─centos-home 253:2 0 21.4G 0 lvm /home sdb 252:16 0 20G 0 disk ├─sdb1 252: 17 0 15G 0 part / var / lib / ceph / osd / ceph-2 # mounts to succeed └─sdb2 252:18 0 5G 0 part
View OSD
ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3
........ [Ceph-node1] [DEBUG] / dev / vdb2 ceph journal, for / dev / vdb1 # two partitions as shown below, indicates a success [ceph-node1][DEBUG ] /dev/vdb1 ceph data, active, cluster ceph, osd.0, journal /dev/vdb2 ........ [ceph-node3][DEBUG ] /dev/vdb2 ceph journal, for /dev/vdb1 [ceph-node3][DEBUG ] /dev/vdb1 ceph data, active, cluster ceph, osd.1, journal /dev/vdb2 ....... [ceph-node3][DEBUG ] /dev/vdb2 ceph journal, for /dev/vdb1 [ceph-node3][DEBUG ] /dev/vdb1 ceph data, active, cluster ceph, osd.2, journal /dev/vdb2
With ceph-deploy the configuration file and the admin key copy to the management node and Ceph node, so each time you perform the Ceph command line you need to specify monit node address
And ceph.client.admin.keyring the
ceph-deploy admin ceph-admin ceph-node1 ceph-node2 ceph-node3
Modify the key permissions
sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
Check the ceph state
sudo ceph health
HEALTH_OK
[cephuser@ceph-admin cluster]$ sudo ceph -s
cluster 33bfa421-8a3b-40fa-9f14-791efca9eb96
health HEALTH_OK
monmap e1: 1 mons at {ceph-admin=192.168.10.220:6789/0}
election epoch 3, quorum 0 ceph-admin
osdmap e14: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v29: 64 pgs, 1 pools, 0 bytes data, 0 objects
100 MB used, 45946 MB / 46046 MB avail
64 active+clean
View ceph osd running state
ceph osd stat
osdmap e19: 3 osds: 3 up, 3 in flags sortbitwise,require_jewel_osds
View osd directory tree
ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.04376 root default -2 0.01459 host ceph-node1 0 0.01459 osd.0 up 1.00000 1.00000 -3 0.01459 host ceph-node2 1 0.01459 osd.1 up 1.00000 1.00000 -4 0.01459 host ceph-node3 2 0.01459 osd.2 up 1.00000 1.00000
View monit of service monitoring node
sudo systemctl status ceph-mon@ceph-admin
ps -ef | grep ceph | grep 'cluster'
ceph 28190 1 0 11:44 ? 00:00:01 /usr/bin/ceph-mon -f --cluster ceph --id ceph-admin --setuser ceph --setgroup ceph
Respectively view the Ceph-node1 , Ceph-node2 , Ceph-node3 three nodes osd of service, it has been found in the startup.
[Cephuser @ ceph-node1 ~] $ sudo systemctl status [email protected] # start is start, restart a restart
[cephuser@ceph-node1 ~]$ sudo ps -ef|grep ceph|grep "cluster"
ceph 28749 1 0 11:44 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph cephuser 29197 29051 0 11:54 pts/2 00:00:00 grep --color=auto cluster
[cephuser@ceph-node2 ~]$ sudo systemctl status [email protected]
[cephuser@ceph-node2 ~]$ sudo ps -ef|grep ceph|grep "cluster"
ceph 28749 1 0 11:44 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph cephuser 29197 29051 0 11:54 pts/2 00:00:00 grep --color=auto cluster
[cephuser@ceph-node3 ~]$ sudo systemctl status [email protected]
[cephuser@ceph-node3 ~]$ sudo ps -ef|grep ceph|grep "cluster"
ceph 28749 1 0 11:44 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph cephuser 29197 29051 0 11:54 pts/2 00:00:00 grep --color=auto cluster
3, create a file system
First check the management node status, the default is no management node.
ceph mds stat
e1:
Create a management node ( Ceph-ADMIN as a management node).
Note: If you do not create mds management node, Client client will not be able to mount a normal ceph cluster! !
pwd /home/cephuser cd cluster/ ceph-deploy mds create ceph-admin
View Management node status again, it has been found in the boot in
ceph mds stat
e2:, 1 up:standby
View Status
sudo systemctl status ceph-mds@ceph-admin
ps -ef | grep cluster | grep ceph-mds
ceph 29093 1 0 12:46 ? 00:00:00 /usr/bin/ceph-mds -f --cluster ceph --id ceph-admin --setuser ceph --setgroup ceph
Creating the pool , the pool is ceph logical partitions of data storage , which play a namespace role
ceph osd lspools # first check the pool
0 rbd,
The newly created ceph cluster only rdb a the pool . Then you need to create a new pool
ceph osd pool create cephfs_data 10 # latter figure is the number of PG
pool 'cephfs_data' created
ceph osd pool create cephfs_metadata 10
# Create a pool of metadata
pool 'cephfs_metadata' created
ceph fs new myceph cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1
Check again pool status
ceph osd lspools
0 rbd,1 cephfs_data,2 cephfs_metadata,
Check the mds management node status
ceph mds stat
e5: 1/1/1 up {0=ceph-admin=up:active}
View ceph cluster status
sudo ceph -s
cluster 33bfa421-8a3b-40fa-9f14-791efca9eb96
health HEALTH_OK
monmap e1: 1 mons at {ceph-admin=192.168.10.220:6789/0}
election epoch 3, quorum 0 ceph-admin
fsmap e5: 1/1/1 up {0 = ceph-admin = up: active} # plurality this line state
osdmap e19: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v48: 84 pgs, 3 pools, 2068 bytes data, 20 objects
101 MB used, 45945 MB / 46046 MB avail
84 active+clean
View ceph cluster port
sudo lsof -i:6789
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ceph-mon 28190 ceph 10u IPv4 70217 0t0 TCP ceph-admin:smc-https (LISTEN) ceph-mon 28190 ceph 19u IPv4 70537 0t0 TCP ceph-admin:smc-https->ceph-node1:41308 (ESTABLISHED) ceph-mon 28190 ceph 20u IPv4 70560 0t0 TCP ceph-admin:smc-https->ceph-node2:48516 (ESTABLISHED) ceph-mon 28190 ceph 21u IPv4 70583 0t0 TCP ceph-admin:smc-https->ceph-node3:44948 (ESTABLISHED) ceph-mon 28190 ceph 22u IPv4 72643 0t0 TCP ceph-admin:smc-https->ceph-admin:51474 (ESTABLISHED) ceph-mds 29093 ceph 8u IPv4 72642 0t0 TCP ceph-admin:51474->ceph-admin:smc-https (ESTABLISHED)
If these states are normal, Ceph cluster set up is complete, the next and then describes how to use three different ways to use Ceph.