分布式存储系统Ceph部署过程 · Jayden's Blog

其他 2020-03-04 15:17:32 阅读次数: 0

本帖子记录的是分布式存储系统Ceph部署过程记录。

环境准备

这里我只是测试环境，所以使用一个监控节点，两个存储节点，具体如下：

ip地址(双网卡) 主机名 ceph磁盘 系统 备注
192.168.0.200/172.16.0.100 ceph-node1 100G CentOS Linux release 7.4.1708 (Core) 作为mds、mon、osd0
192.168.0.201/172.16.0.101 ceph-node2 100G CentOS Linux release 7.4.1708 (Core) 作为osd1
192.168.0.202/172.16.0.102 ceph-node3 100G CentOS Linux release 7.4.1708 (Core) 作为osd2
192.168.0.203/172.16.0.103 ceph-client 挂载点：/cephfs CentOS Linux release 7.4.1708 (Core) ceph客户端

Ceph的文件系统作为一个目录挂载到客户端cephclient的/cephfs目录下，可以像操作普通目录一样对此目录进行操作。

安装前准备(每个节点都需要做以下操作)

分别在ceph的三个节点机（ceph-node1、ceph-node2、ceph-node3）上添加hosts

1
2
3
4
5
6
7
 
$ vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.200 ceph-node1
192.168.0.201 ceph-node2
192.168.0.202 ceph-node3
192.168.0.203 ceph-client

关闭防火墙和selinux

1
2
3
4
 
$ sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
$ setenforce 0
$ systemctl stop firewalld
$ systemctl disable firewalld

创建用户ceph

分别在ceph的三个节点机（ceph-node1、ceph-node2、ceph-node3）上创建用户ceph，密码统一设置为ceph

1
2
3
4
 
$ adduser ceph
$ echo "ceph"|passwd --stdin ceph
$ echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
$ chmod 0440 /etc/sudoers.d/ceph

配置NTP服务

由于Ceph对时间比较敏感，所以建议使用ntp服务保持三台服务器的时间保持同步

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
 
$ yum install -y ntp
$ vi /etc/ntp.conf

# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
 
driftfile /var/lib/ntp/drift
 
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery
 
# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
 
# 允许1-4网段的服务器来校时，不允许客户端来修改，登录ntp服务器 
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap
 
 
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
 
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
 
server time1.aliyun.com
server time2.aliyun.com
server time3.aliyun.com
server time4.aliyun.com
server time5.aliyun.com
server time6.aliyun.com
server time7.aliyun.com
 
#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
 
# Enable public key cryptography.
#crypto
 
includefile /etc/ntp/crypto/pw
 
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography. 
keys /etc/ntp/keys
 
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
 
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
 
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
 
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
 
# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

# Enable Logfile
logfile /var/log/ntp.log



$ systemctl start ntpd
$ systemctl enable ntpd

安装并配置

更换系统源

把系统默认源更换为阿里云源

1
2
3
4
5
6
7
8
 
$ rm -rf /etc/yum.repos.d/*.repo
$ wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
$ wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
$ sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo
$ sed -i 's/$releasever/7/g' /etc/yum.repos.d/CentOS-Base.repo
$ sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo
$ yum clean all
$ yum makecache fast

安装Ceph-Deploy(主节点/管理节点)

以下操作仅在主节点/管理节点中进行，不需要所有节点进行以下操作

1
2
3
 
$ yum install http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/ceph-deploy-1.5.38-0.noarch.rpm
$ ceph-deploy --version
1.5.39

创建CEPH集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
$ sudo -iu ceph
$ mkdir -p ceph-cluster
$ cd /home/ceph/ceph-cluster
$ ceph-deploy new ceph-node-1 ceph-node-2 ceph-node-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /usr/bin/ceph-deploy new ceph-node1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x112b500>
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xf26e18>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-node1','ceph-node2','ceph-node3']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
···
···

编辑CEPH配置文件

在global下添加以下配置：

1
2
3
4
5
 
$ vi ceph.conf
[global]
mon_clock_drift_allowed = 5
osd_journal_size = 20480
public_network=172.16.0.0/24

安装CEPH(所有节点都需要安装)

1
2
3
4
5
 
$ ceph-deploy install --release jewel --repo-url http://mirrors.163.com/ceph/rpm-jewel/el7 --gpg-url http://mirrors.163.com/ceph/keys/release.asc ceph-client ceph-node-1 ceph-node-2 ceph-node-3

## 检查CEPH版本
$ ceph -v
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)

初始化MOM节点(主节点/管理节点)

1
2
3
4
5
6
7
8
9
 
$ ceph-deploy mon create-initial
···
···
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmp8sI0aU

查看集群状态（在OSD节点上查看）

1
2
3
4
5
6
7
8
9
10
11
 
$ ceph -s
     cluster 60597e53-ad29-44bd-8dcd-db6aeae6f580
     health HEALTH_ERR
            no osds
     monmap e2: 3 mons at {ceph-node-1=172.16.0.100:6789/0,ceph-node-2=172.16.0.101:6789/0,ceph-node-3=172.16.0.102:6789/0}
            election epoch 6, quorum 0,1,2 ceph-node-1,ceph-node-2,ceph-node-3
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating

配置管理节点ceph-master

用ceph-deploy把配置文件和admin密钥拷贝到Master节点

1
2
3
4
 
$ ceph-deploy admin ceph-master

## 确保你对ceph.client.admin.keyring有正确的操作权限
大专栏  分布式存储系统Ceph部署过程 · Jayden's Bloge">$ chmod +r /etc/ceph/ceph.client.admin.keyring

添加数据节点(在节点上用root进行操作)

1
2
3
4
5
6
7
8
9
10
11
 
[root@ceph-node1 ~]# mkdir /data
[root@ceph-node1 ~]# mkdir /data/osd0
[root@ceph-node1 ~]# chmod -R 777 /data/
 
[root@ceph-node2 ~]# mkdir /data
[root@ceph-node2 ~]# mkdir /data/osd1
[root@ceph-node2 ~]# chmod -R 777 /data/
 
[root@ceph-node3 ~]# mkdir /data
[root@ceph-node3 ~]# mkdir /data/osd2
[root@ceph-node3 ~]# chmod -R 777 /data/

分别挂载磁盘

1
2
3
4
5
 
[root@ceph-node1 ~]# mount /dev/vdb1 /data/osd0
 
[root@ceph-node2 ~]# mount /dev/vdb1 /data/osd1
 
[root@ceph-node3 ~]# mount /dev/vdb1 /data/osd2

在主节点/管理节点准备OSD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
$ cd /home/ceph/ceph-cluster/
$ ceph-deploy osd prepare ceph-node1:/data/osd0 ceph-node2:/data/osd1 ceph-node3:/data/osd2
······
······
[ceph-node3][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs -- /data/osd2
[ceph-node3][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[ceph-node3][WARNIN] command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
[ceph-node3][WARNIN] command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
[ceph-node3][WARNIN] command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
[ceph-node3][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[ceph-node3][WARNIN] populate_data_path: Preparing osd data dir /data/osd2
[ceph-node3][WARNIN] command: Running command: /sbin/restorecon -R /data/osd2/ceph_fsid.5285.tmp
[ceph-node3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph /data/osd2/ceph_fsid.5285.tmp
[ceph-node3][WARNIN] command: Running command: /sbin/restorecon -R /data/osd2/fsid.5285.tmp
[ceph-node3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph /data/osd2/fsid.5285.tmp
[ceph-node3][WARNIN] command: Running command: /sbin/restorecon -R /data/osd2/magic.5285.tmp
[ceph-node3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph /data/osd2/magic.5285.tmp
[ceph-node3][INFO  ] checking OSD status...
[ceph-node3][DEBUG ] find the location of an executable
[ceph-node3][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph-node3][WARNIN] there are 2 OSDs down
[ceph-node3][WARNIN] there are 2 OSDs out
[ceph_deploy.osd][DEBUG ] Host ceph-node3 is now ready for osd use.

激活OSD（仍然在主节点/管理节点上操作）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
$ ceph-deploy osd activate ceph-node1:/data/osd0 ceph-node2:/data/osd1 ceph-node3:/data/osd2
······
······
[ceph-node3][WARNIN] start_daemon: Starting ceph osd.4...
[ceph-node3][WARNIN] command_check_call: Running command: /usr/bin/systemctl disable ceph-osd@4
[ceph-node3][WARNIN] command_check_call: Running command: /usr/bin/systemctl disable ceph-osd@4 --runtime
[ceph-node3][WARNIN] command_check_call: Running command: /usr/bin/systemctl enable ceph-osd@4
[ceph-node3][WARNIN] Created symlink from /etc/systemd/system/ceph-osd.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[ceph-node3][WARNIN] command_check_call: Running command: /usr/bin/systemctl start ceph-osd@4
[ceph-node3][INFO  ] checking OSD status...
[ceph-node3][DEBUG ] find the location of an executable
[ceph-node3][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph-node3][WARNIN] there are 2 OSDs down
[ceph-node3][WARNIN] there are 2 OSDs out
[ceph-node3][INFO  ] Running command: sudo systemctl enable ceph.target

开机挂载磁盘（所有OSD节点都要操作）

挂载好的新磁盘会在下一次重启或开机的时候失效，因此需要在每台机上对新磁盘进行挂载，并编辑/etc/fstab文件

1
2
3
4
5
6
7
8
9
10
11
 
[root@ceph-node1 ~]# cat /etc/fstab
........
/dev/vdb1               /data/osd0                   xfs     defaults        0 0
 
[root@ceph-node2 ~]# cat /etc/fstab
........
/dev/vdb1               /data/osd1                   xfs     defaults        0 0
 
[root@ceph-node3 ~]# cat /etc/fstab
........
/dev/vdb1               /data/osd2                   xfs     defaults        0 0

分发配置和密钥（主节点/管理节点上执行）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
$ sudo -iu ceph
$ cd /home/ceph/ceph-cluster
$ ceph-deploy admin ceph-node1 ceph-node2 ceph-node3
······
······
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node1
[ceph-node1][DEBUG ] connection detected need for sudo
[ceph-node1][DEBUG ] connected to host: ceph-node1 
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node2
ceph@ceph-node2's password: 
[ceph-node2][DEBUG ] connection detected need for sudo
ceph@ceph-node2's password: 
[ceph-node2][DEBUG ] connected to host: ceph-node2 
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node3
ceph@ceph-node3's password: 
[ceph-node3][DEBUG ] connection detected need for sudo
ceph@ceph-node3's password: 
[ceph-node3][DEBUG ] connected to host: ceph-node3 
[ceph-node3][DEBUG ] detect platform information from remote host
[ceph-node3][DEBUG ] detect machine type
[ceph-node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

查看集群状态

1
2
3
4
5
6
7
8
9
10
11
12
 
$ ceph health
HEALTH_OK

$ ceph -s
    cluster 12d4f209-69ea-4ad9-9507-b00557b42815
     health HEALTH_OK
     monmap e1: 3 mons at {ceph-node-1=172.16.0.100:6789/0,ceph-node-2=172.16.0.101:6789/0,ceph-node-3=172.16.0.102:6789/0}
            election epoch 2, quorum 0 ceph-node1
     osdmap e14: 3 osds: 3 up, 3 in
      pgmap v26: 64 pgs, 1 pools, 0 bytes data, 0 objects
            15459 MB used, 45944 MB / 61404 MB avail
                  64 active+clean

重新部署（温馨提示）

部署过程中如果出现任何奇怪的问题无法解决，可以简单的删除所有节点，一切从头再来：

1
2
3
 
$ ceph-deploy purge ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd1 ceph-osd2
$ ceph-deploy purgedata ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd1 ceph-osd2
$ ceph-deploy forgetkey

创建文件系统

创建文件系统的步骤参考官网
对于一个刚创建的MDS服务，虽然服务是运行的，但是它的状态直到创建 pools 以及文件系统的时候才会变为Active.

还没有创建时候的状态

1
2
 
$ ceph mds stat
e1: 0/0/0 up

创建管理节点（ceph-node1节点上）

1
2
3
4
5
6
7
8
9
10
11
12
13
 
$ sudo -iu ceph
$ cd /home/ceph/ceph-cluster
$ ceph-deploy mds create ceph-node1 ceph-node2 ceph-node3
······
······
[ceph-node2][INFO  ] Running command: service ceph start mds.ceph-node2
[ceph-node2][DEBUG ] === mds.ceph-node2 === 
[ceph-node2][DEBUG ] Starting Ceph mds.ceph-node2 on ceph-node2...
[ceph-node2][WARNIN] Running as unit ceph-mds.ceph-node2.1496648453.865988091.service.
[ceph-node2][INFO  ] Running command: systemctl enable ceph
[ceph-node2][WARNIN] ceph.service is not a native service, redirecting to /sbin/chkconfig.
[ceph-node2][WARNIN] Executing /sbin/chkconfig ceph on
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

注意：如果不创建mds管理节点，client客户端将不能正常挂载到ceph集群！！

创建pool

1
2
 
$ ceph osd lspools
0 rbd,

新创建的ceph集群只有rdb一个pool。这时需要创建一个新的pool
新建pool，可参考官网

创建新的pool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
$ ceph osd pool create cephfs_data 32                //用于数据的pool。后面的数字是PG的数量
pool 'cephfs_data' created
$ ceph osd pool create cephfs_metadata 32            //用于元数据的pool
pool 'cephfs_metadata' created

$ ceph fs new myceph  cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1

$ ceph osd lspools
0 rbd,1 cephfs_data,2 cephfs_metadata,

## 检验
$ ceph mds stat
e5: 1/1/1 up {0=ceph-node1=up:active}


$ ceph -s
    cluster 12d4f209-69ea-4ad9-9507-b00557b42815
     health HEALTH_OK
     monmap e1: 1 mons at {ceph-node-1=172.16.0.100:6789/0,ceph-node-2=172.16.0.101:6789/0,ceph-node-3=172.16.0.102:6789/0}
            election epoch 2, quorum 0 ceph-node1
     mdsmap e5: 1/1/1 up {0=ceph-node1=up:active}
     osdmap e19: 3 osds: 3 up, 3 in
      pgmap v37: 84 pgs, 3 pools, 1962 bytes data, 20 objects
            15460 MB used, 45943 MB / 61404 MB avail
                  84 active+clean

$ ceph osd tree
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 0.29279 root default                                     
-2 0.09760     host ceph6                                   
 2 0.09760         osd.2       up  1.00000          1.00000 
-3 0.09760     host ceph7                                   
 3 0.09760         osd.3       up  1.00000          1.00000 
-4 0.09760     host ceph8                                   
 4 0.09760         osd.4       up  1.00000          1.00000

猜你喜欢

转载自www.cnblogs.com/lijianming180/p/12409598.html
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
周排行
Metasploit文件目录与入侵基本概念
跨域(CORS)请求问题[No 'Access-Control-Allow-Origin' header is present on the requested resource]常见解决方案
CodeIgniter 源码解读之 CodeIgniter.php（二）
SAS入门之（四）改变数据类型
初识元组
[数学建模]数学建模算法和模型（B站视频）（二）
Nginx 服务器源码安装配置流程
C#实现语音视频录制 【基于MCapture + MFile】
开发进度4
下载安装vue的方法网址
每日归档
更多
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022