Ceph distributed storage series (2): deploy three-node ceph cluster using ceph-deploy method

Continuing from the above: Ceph distributed storage series (1): A brief analysis of Ceph’s working principle and architecture. In the past,
ceph was deployed using Deepsea method. I have not used ceph-deploy for a long time. Let’s review it this time and summarize it by the way!

Foreword:

    The ceph-deploy deployment method is officially no longer maintained. It has not been tested on versions after Nautilus (v14.
For details, please view the official instructions.
Official statement: https://docs.ceph.com/en/latest/install/

ceph-deploy is a tool for quickly deploying clusters.
Important：ceph-deploy is no longer actively maintained. It is not tested on versions of Ceph newer than Nautilus. It does not support RHEL8, CentOS 8, or newer operating systems.

1. Deployment preparation

The Ceph version used this time:

• Ceph: Nautilus (14.2.9)
• Ceph-deploy：2.0.1

Node information for this test:

IP address	CPU name	Additional Disk (OSD)	cluster role
192.168.56.125	ceph-node1	A 10G disk (/dev/sdb)	mon, mgr, osd0 (master node)
192.168.56.126	ceph-node2	A 10G disk (/dev/sdb)	osd1
192.168.56.127	ceph-node3	A 10G disk (/dev/sdb)	osd2

If the environment permits, you can use a ceph-admin node to specifically place these components such as mon, mgr, mds, etc., and place the osd on other nodes for easier management.

Server system version

[root@ceph-node1 ~]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

1. Turn off the firewall and selinux

sed -i  "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config
setenforce 0
systemctl stop firewalld
systemctl disable firewalld

2. Configure the hosts file

保证集群内主机名与ip解析正常（每个节点都需要配置）
[root@ceph-node1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.125  ceph-node1
192.168.56.126  ceph-node2
192.168.56.127  ceph-node3
[root@ceph-node1 ~]# ping ceph-node2
PING ceph-node2 (192.168.56.126) 56(84) bytes of data.
64 bytes from ceph-node2 (192.168.56.126): icmp_seq=1 ttl=64 time=0.616 ms
…………

3. Create a deployment user and configure sudo permissions (executed on all nodes)

a. Taking into account the security issues of using the root user, a ceph-admin ordinary user is created here for deployment and operation and maintenance. b.
In addition, ceph-deploy will install software packages on the nodes, so the created user needs to have no password. sudo permissions

[root@ceph-node1 ~]# useradd ceph-admin
[root@ceph-node1 ~]# echo "123456" | passwd --stdin ceph-admin
Changing password for user ceph-admin.
passwd: all authentication tokens updated successfully.

[root@ceph-node1 ~]# echo "ceph-admin ALL = NOPASSWD:ALL" | tee /etc/sudoers.d/ceph-admin
ceph-admin ALL = NOPASSWD:ALL
[root@ceph-node1 ~]# chmod 0440 /etc/sudoers.d/ceph-admin
[root@ceph-node1 ~]# ll /etc/sudoers.d/ceph-admin
-r--r-----. 1 root root 30 Oct 19 16:06 /etc/sudoers.d/ceph-admin

测试
[root@ceph-node1 ~]# su - ceph-admin
Last login: Mon Oct 19 16:11:51 CST 2020 on pts/0
[ceph-admin@ceph-node1 ~]$ sudo su -
Last login: Mon Oct 19 16:12:04 CST 2020 on pts/0
[root@ceph-node1 ~]# exit
logout
[ceph-admin@ceph-node1 ~]$ exit
logout

4. Configure ssh passwordless access (executed on the master node node1)

[root@ceph-node1 ~]# su - ceph-admin
[ceph-admin@ceph-node1 ~]$ ssh-keygen          （每一步都按回车，口令密码留空）
[ceph-admin@ceph-node1 ~]$ ssh-copy-id ceph-admin@ceph-node1
[ceph-admin@ceph-node1 ~]$ ssh-copy-id ceph-admin@ceph-node2
[ceph-admin@ceph-node1 ~]$ ssh-copy-id ceph-admin@ceph-node3

5. Configure ntp time synchronization

The purpose of configuring time synchronization: Only when the time is consistent can the normal operation of the cluster be ensured. The
configuration time synchronization method: node1 connects to the NTP server on the network to synchronize time, and node2 and 3 connect to node1 to synchronize time (that is, node1 is both an NTP server and also for the client)
注：ntpd启动后需要等待几分钟去同步

yum -y intall ntp（安装ntp，全部节点都需要执行）

node1节点操作：
vim /etc/ntp.conf
注释掉默认的配置项：
    #server 0.centos.pool.ntp.org iburst
    #server 1.centos.pool.ntp.org iburst
    #server 2.centos.pool.ntp.org iburst
    #server 3.centos.pool.ntp.org iburst
添加配置项：
server  ntp1.aliyun.com     #阿里云ntp服务器
server 127.127.1.0     #本地ntp服务器，配置此项是为了在外网ntp连接异常的情况下还能保证ntp正常，维护集群稳定

node2/node3节点操作：
vim /etc/ntp.conf
同样注释掉默认的server配置项：
添加配置项：
server 192.168.56.125     #node1-ntp服务器

全部节点都执行：
systemctl restart ntpd
systemctl enable ntpd

查看ntp连接情况和状态
[root@ceph-node1 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*120.25.115.20   10.137.53.7      2 u   41  128  377   30.382   -1.019   1.001
 LOCAL(0)        .LOCL.           5 l  806   64    0    0.000    0.000   0.000
 
 [root@ceph-node2 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ceph-node1      120.25.115.20    3 u   20   64  377    2.143   33.254  10.350

[root@ceph-node1 ~]# ntpstat
synchronised to NTP server (120.25.115.20) at stratum 3
   time correct to within 27 ms
   polling server every 128 s

2. Start deploying the Ceph cluster

1. Add Alibaba Cloud's base source and epel source (executed on all nodes)

备份系统原本的源
[root@ceph-node1 ~]# mkdir /mnt/repo_bak
[root@ceph-node1 ~]# mv /etc/yum.repos.d/* /mnt/repo_bak
添加新源
[root@ceph-node1 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@ceph-node1 ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

2. Add ceph’s yum source (executed on all nodes)

注意事项：
The yum source here determines the version of ceph. The baseurl item in the source rpm-nautilusrepresents nautilusthe rpm package of the ceph version ( nautilus是ceph的14.x版本). If you need to install other versions, you need to replace them with other version numbers. The 12.x version is luminous , the 13.x version is rpm-mimic. Details can be viewed in the ceph official source: https://download.ceph.com/

vim /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph
baseurl=http://download.ceph.com/rpm-nautilus/el7/x86_64
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-nautilus/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-nautilus/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

更新yum缓存及系统软件
yum makecache
yum -y update

You can check the ceph version to determine whether yum is configured correctly.

[root@ceph-node1 yum.repos.d]# yum list ceph --showduplicates |sort -r
 * updates: mirrors.cn99.com
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
 * extras: mirrors.163.com
ceph.x86_64                         2:14.2.9-0.el7                          Ceph
ceph.x86_64                         2:14.2.8-0.el7                          Ceph
ceph.x86_64                         2:14.2.7-0.el7                          Ceph
ceph.x86_64                         2:14.2.6-0.el7                          Ceph
ceph.x86_64                         2:14.2.5-0.el7                          Ceph
ceph.x86_64                         2:14.2.4-0.el7                          Ceph
ceph.x86_64                         2:14.2.3-0.el7                          Ceph
ceph.x86_64                         2:14.2.2-0.el7                          Ceph
ceph.x86_64                         2:14.2.11-0.el7                         Ceph
ceph.x86_64                         2:14.2.1-0.el7                          Ceph
ceph.x86_64                         2:14.2.10-0.el7                         Ceph
ceph.x86_64                         2:14.2.0-0.el7                          Ceph
ceph.x86_64                         2:14.1.1-0.el7                          Ceph
ceph.x86_64                         2:14.1.0-0.el7                          Ceph
 * base: mirrors.163.com
Available Packages

[root@ceph-node1 yum.repos.d]# yum list ceph-deploy --showduplicates |sort -r
 * updates: mirrors.cn99.com
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
 * extras: mirrors.163.com
ceph-deploy.noarch                     2.0.1-0                       Ceph-noarch
ceph-deploy.noarch                     2.0.0-0                       Ceph-noarch
ceph-deploy.noarch                     1.5.39-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.38-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.37-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.36-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.35-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.34-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.33-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.32-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.31-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.30-0                      Ceph-noarch
ceph-deploy.noarch                     1.5.29-0                      Ceph-noarch
 * base: mirrors.163.com
Available Packages

3. Install ceph-deploy (executed on the master node node1)

[root@ceph-node1 ~]# su - ceph-admin
[ceph-admin@ceph-node1 ~]$ sudo yum -y install python-setuptools   #安装ceph依赖包
[ceph-admin@ceph-node1 ~]$ sudo yum install ceph-deploy  （默认会选择安装2.0最新版本）

查看ceph-deploy安装版本
[root@ceph-node1 ~]# ceph-deploy --version
2.0.1

4. Initialize the cluster (executed on the master node node1)

Create the cluster installation directory (ceph-deploy deployment program will output the file to the current directory)

[ceph-admin@ceph-node1 ~]$ mkdir cluster
[ceph-admin@ceph-node1 ~]$ cd cluster/

Create a cluster (the next step is to specify which nodes are used as mon monitors, so select the node where mon is deployed in the plan - node1)

[ceph-admin@ceph-node1 cluster]$ ceph-deploy new ceph-node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy new ceph-node1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f14c44c9de8>
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f14c3c424d0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-node1']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-node1][DEBUG ] connection detected need for sudo
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph-node1][DEBUG ] find the location of an executable
[ceph-node1][INFO  ] Running command: sudo /usr/sbin/ip link show
[ceph-node1][INFO  ] Running command: sudo /usr/sbin/ip addr show
[ceph-node1][DEBUG ] IP addresses found: [u'192.168.56.125']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node1
[ceph_deploy.new][DEBUG ] Monitor ceph-node1 at 192.168.56.125
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-node1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.56.125']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

[ceph-admin@ceph-node1 cluster]$ ls
ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring

Add the following two lines to ceph.conf in the current directory

public_network = 192.168.56.0/24
cluster_network = 192.168.56.0/24

Install the Ceph package to other nodes
( --no-adjust-reposparameter meaning: use the locally configured source, do not change the source. To prevent problems)

[ceph-admin@ceph-node1 cluster]$ ceph-deploy install --no-adjust-repos ceph-node1 ceph-node2 ceph-node3

If an " " error occurs RuntimeError: Failed to execute command: ceph --version, it is caused by server network problems. The downloading speed of the ceph installation package is too slow. It will time out after 5 minutes. You can repeat the execution, or execute yum -y install ceph on all nodes separately.

Initialize mon node

In version 2.0.1 of ceph-deploy, keys will be collected during initialization. There is no need to execute the ceph-deploy gatherkeys
{monitor-host} command.

[ceph-admin@ceph-node1 cluster]$ ceph-deploy mon create-initial

5. Add OSD

If it is a disk with data inside, you need to clear the data first: (For details, see ceph-depoy disk zap --help)

列出所有节点上所有可用的磁盘
[ceph-admin@ceph-node1 cluster]$ ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3
清除数据
sudo ceph-deploy disk zap {
    
    osd-server-name} {
    
    disk-name}
    eg：sudo ceph-deploy disk zap ceph-node2 /dev/sdb

If it is a clean disk, you can ignore the above operation of clearing data and add OSD directly
(I am adding the newly added /dev/sdb disk here)

[ceph-admin@ceph-node1 cluster]$ ceph-deploy osd create --data /dev/sdb ceph-node1
[ceph-admin@ceph-node1 cluster]$ ceph-deploy osd create --data /dev/sdb ceph-node2
[ceph-admin@ceph-node1 cluster]$ ceph-deploy osd create --data /dev/sdb ceph-node3

You can see that cpeh creates the new OSD in LVM format and adds it to the ceph cluster.

[ceph-admin@ceph-node1 cluster]$ sudo pvs
  PV         VG                                        Fmt  Attr PSize   PFree
  /dev/sdb   ceph-ab1b8533-018e-4924-8520-fdbefbb7d184 lvm2 a--  <10.00g    0

6. Allow the host to execute Ceph commands with administrator privileges

Use the ceph-deploy command to copy the configuration file and admin key to each ceph node. Other node hosts can also manage the ceph cluster.

[ceph-admin@ceph-node1 cluster]$ ceph-deploy admin ceph-node1 ceph-node2 ceph-node3

7. Deploy MGR to obtain cluster information

[ceph-admin@ceph-node1 cluster]$ ceph-deploy mgr create ceph-node1

View cluster status

[ceph-admin@ceph-node1 cluster]$ sudo ceph health detail
HEALTH_OK
[ceph-admin@ceph-node1 cluster]$ sudo ceph -s
  cluster:
    id:     e9290965-40d4-4c65-93ed-e534ae389b9c
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-node1 (age 62m)
    mgr: ceph-node1(active, since 5m)
    osd: 3 osds: 3 up (since 12m), 3 in (since 12m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 27 GiB / 30 GiB avail
    pgs:

If you check the cluster status “HEALTH_WARN mon is allowing insecure global_id reclaim”, it is because the unsafe mode is enabled, just disable it:

 [ceph-admin@ceph-node1 cluster]$ sudo ceph config set mon auth_allow_insecure_global_id_reclaim false

Because ordinary users do not have read permission on the key file under /etc/ceph/, ordinary users do not have the right to directly execute ceph commands.
If ceph-admin is needed, ordinary users can also directly call the cluster and increase the read permission on the ceph configuration file
(if desired) Ordinary users on each node can execute ceph related commands, then all nodes must modify the permissions)

[ceph-admin@ceph-node1 ~]$ ll /etc/ceph/
total 12
-rw-------. 1 root root 151 Oct 21 17:33 ceph.client.admin.keyring
-rw-r--r--. 1 root root 268 Oct 21 17:35 ceph.conf
-rw-r--r--. 1 root root  92 Oct 20 04:48 rbdmap
-rw-------. 1 root root   0 Oct 21 17:30 tmpcmU035
[ceph-admin@ceph-node1 ~]$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring
[ceph-admin@ceph-node1 ~]$ ll /etc/ceph/
total 12
-rw-r--r--. 1 root root 151 Oct 21 17:33 ceph.client.admin.keyring
-rw-r--r--. 1 root root 268 Oct 21 17:35 ceph.conf
-rw-r--r--. 1 root root  92 Oct 20 04:48 rbdmap
-rw-------. 1 root root   0 Oct 21 17:30 tmpcmU035
[ceph-admin@ceph-node1 ~]$ ceph -s
  cluster:
    id:     130b5ac0-938a-4fd2-ba6f-3d37e1a4e908
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-node1 (age 20h)
    mgr: ceph-node1(active, since 20h)
    osd: 3 osds: 3 up (since 20h), 3 in (since 20h)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 27 GiB / 30 GiB avail
    pgs:

3. Configure the Mgr-Dashboard module

Open the dashboard module

[ceph-admin@ceph-node1 ~]$ sudo ceph mgr module enable dashboard

If the error is as follows:

Error ENOENT: all mgr daemons do not support module 'dashboard', pass --force to force enablement

That's because ceph-mgr-dashboard is not installed. Just install it on the mgr node.

[ceph-admin@ceph-node1 ~]$ sudo yum -y install ceph-mgr-dashboard

By default, all HTTP connections to the dashboard are secured using SSL/TLS.
To quickly get your dashboard up and running, you can generate and install a self-signed certificate using the following command

[ceph-admin@ceph-node1 ~]$ sudo ceph dashboard create-self-signed-cert
Self-signed certificate created

Create a user with administrator role:

[ceph-admin@ceph-node1 ~]$ sudo ceph dashboard set-login-credentials admin admin
******************************************************************
***          WARNING: this command is deprecated.              ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************
Username and password updated

The "admin admin" I used before seems like I can no longer write it directly like this. I need to write the password in a file and read it, otherwise an error will be reported.
“dashboard set-login-credentials <username> : Set the login credentials. Password read from -i <file>”

Then add the -i parameter to create the same

[ceph-admin@ceph-node1 cluster]$ echo admin > userpass
[ceph-admin@ceph-node1 cluster]$ sudo ceph dashboard set-login-credentials admin -i userpass
******************************************************************
***          WARNING: this command is deprecated.              ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************
Username and password updated

View ceph-mgr service:

[ceph-admin@ceph-node1 ~]$ sudo ceph mgr services
{
    
    
    "dashboard": "https://ceph-node1:8443/"
}

Browser access test:
http://192.168.56.125:8443