Ceph 提供iSCSI存储

Tgtd+Ceph部署

一、yum安装tgt

[root@c720181 ~]# yum --enablerepo=epel -y install scsi-target-utils libxslt

（1）安装 scsi-target-utils

安装 scsi-target-utils

# yum --enablerepo=epel -y install scsi-target-utils libxslt

查看版本

# tgtadm --version
1.0.55

（2）启动 tgtd 服务

//设置开机启动
systemctl enable tgtd

//启动
systemctl start tgtd

（3）查看是否支持 rbd

# tgtadm --lld iscsi --mode system --op show | grep rbd

如果有类似于上面的输出结果，说明你的tgt加载了rbd模块。该模块实际上是一个名为bs_rbd.so的动态链接库。如果命令没有输出，则说明你的tgt还不支持rbd，你需要安装最新的tgt，或者直接下载编译最新的源码。

二、编译安装tgt

(1) 准备编译环境

[root@c720181 ~]# yum install git install librados2-devel librbd1-devel gcc rpm-build

(2)获取Tgt

扫描二维码关注公众号，回复： 7138576 查看本文章

[root@c720181 ~]# git clone https://github.com/fujita/tgt.git

(3) 进入目录后编辑

[root@c720181 tgt]# vim Makefile

export ISCSI_RDMA
export CEPH_RBD = 1 #修改为=1
export GLFS_BD
export SD_NOTIFY

[root@c720181 tgt]# vim scripts/tgtd.spec

...

%{_mandir}/man5/*
%{_mandir}/man8/*
%{_initrddir}/tgtd
/usr/lib/tgt/backing-store/bs_rbd.so
/etc/bash_completion.d/tgt
%attr(0600,root,root) %config(noreplace) /etc/tgt/targets.conf

...

(4)编译

[root@c720181 tgt]# make rpm
Building version: 1.0.79-v1.0.79
Creating rpm build dirs under /root/tgt/pkg
Creating tgz scsi-target-utils-1.0.79-v1.0.79.tgz
Creating rpm
Wrote: /root/tgt/pkg/SRPMS/scsi-target-utils-1.0.79-v1.0.79.src.rpm
Failed to build rpm. LOG: /tmp/build-pkg.sh-386879.log
make: *** [rpm] Error 1
[root@c720181 tgt]# vim /tmp/build-pkg.sh-386879.log

error: Failed build dependencies:
        libibverbs-devel is needed by scsi-target-utils-1.0.79-v1.0.79.x86_64
        librdmacm-devel is needed by scsi-target-utils-1.0.79-v1.0.79.x86_64
        libaio-devel is needed by scsi-target-utils-1.0.79-v1.0.79.x86_64
        docbook-style-xsl is needed by scsi-target-utils-1.0.79-v1.0.79.x86_64c

解决办法：安装依赖包

[root@c720181 tgt]# yum install -y libibverbs-devel librdmacm-devel libaio-devel docbook-style-xsl

重新编译：

[root@c720181 tgt]# make rpm
Building version: 1.0.79-v1.0.79
Creating rpm build dirs under /root/tgt/pkg
Creating tgz scsi-target-utils-1.0.79-v1.0.79.tgz
Creating rpm
Wrote: /root/tgt/pkg/SRPMS/scsi-target-utils-1.0.79-v1.0.79.src.rpm
Wrote: /root/tgt/pkg/RPMS/x86_64/scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm
Wrote: /root/tgt/pkg/RPMS/x86_64/scsi-target-utils-debuginfo-1.0.79-v1.0.79.x86_64.rpm
Done.

(5)生成安装文件

[root@c720181 tgt]# ls pkg/RPMS/x86_64/
scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm scsi-target-utils-debuginfo-1.0.79-v1.0.79.x86_64.rpm

[root@c720181 tgt]# cp pkg/RPMS/x86_64/scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm /root/

(6)在3个MON节点安装Tgt rpm包

[root@c720181 ~]# rpm -ivh scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm --force
Preparing... ################################# [100%]
Updating / installing...
1:scsi-target-utils-1.0.79-v1.0.79 ################################# [100%]

[root@c720182 ~]# rpm -ivh scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm --force
Preparing... ################################# [100%]
Updating / installing...
1:scsi-target-utils-1.0.79-v1.0.79 ################################# [100%]

[root@c720183 ~]# rpm -ivh scsi-target-utils-1.0.79-v1.0.79.x86_64.rpm --force
Preparing... ################################# [100%]
Updating / installing...
1:scsi-target-utils-1.0.79-v1.0.79 ################################# [100%]

三、使用 rbd 作为 target

1. 创建 rbd 池和 image

[ceph-admin@c720181 my-cluster]$ ceph osd pool create tgtpool 8
pool 'tgtpool' created
[ceph-admin@c720181 my-cluster]$ rbd create --size 10240 tgtpool/image01
[ceph-admin@c720181 my-cluster]$ rbd info tgtpool/image01
rbd image 'image01':
   size 10GiB in 2560 objects
   order 22 (4MiB objects)
   block_name_prefix: rbd_data.2a8276b8b4567
   format: 2
   features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
   flags:
   create_timestamp: Fri Aug 30 18:57:33 2019

2、通过配置文件来创建 target

[root@c720181 ~]# vim /etc/tgt/targets.conf

include /etc/tgt/conf.d/*.conf
<target iqn.c720181.kevin.cn:cephtgt.target0>
    bs-type rbd
    backing-store tgtpool/image01
    initiator-address 172.22.69.0/24
</target>

[root@c720181 ~]# systemctl start tgtd

3、查看激活的 target

[root@c720181 ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.c720181.kevin.cn:cephtgt.target0
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 10737 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rbd
            Backing store path: tgtpool/image01
            Backing store flags:
    Account information:
    ACL information:
        172.22.69.0/24

四、客户端安装iSCSI Initor程序并挂载

(1)安装iSCSI Initor程序

[root@client ~]# yum -y install iscsi-initiator-utils

(2)扫描iSCSI Target

[root@client ~]# iscsiadm -m discovery -t sendtargets -p c720181（tgt服务端的IP或主机名）
172.22.69.63:3260,1 iqn.c720181.kevin.cn:cephtgt.target0

(3)挂载目标设备

       如果是挂载这个网络上所有的节点，使用以下命令即可

iscsiadm -m node --login

       如果是挂载指定的一个存储设备，则需要使用以下命令

iscsiadm -m node -T <target_name>:<protocol> -p <server_ip>:<port> -l

       使用示例

[root@client ~]# iscsiadm -m node -T iqn.c720181.kevin.cn:cephtgt.target0 -p 172.22.69.63:3260 --login
Logging in to [iface: default, target: iqn.c720181.kevin.cn:cephtgt.target0, portal: 172.22.69.63,3260] (multiple)
Login to [iface: default, target: iqn.c720181.kevin.cn:cephtgt.target0, portal: 172.22.69.63,3260] successful.

取消挂载目标设备
iscsiadm -m node –T iqn.c720181.kevin.cn:cephtgt.target0 -p 172.22.69.63:3260 -u

其中末尾的u为unlogin的缩写

(4)验证本地挂载是否成功

[root@client ~]# lsscsi
[2:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb

[root@client ~]# fdisk -l

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4194304 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes

五、windows客户端连接
打开软件
首先打开控制面板，然后搜索管理工具，双击打开Iscsi发起程序

挂载设备
在下图中的目标中输入iscsi所在的服务器IP地址，点击快速连接，就可以发现该服务器上可用的Iscsi硬盘，选择后即可点击连接。如果搜索服务器没搜索到，需在服务端的/etc/tgt/targets.conf的需要用到的硬盘的target中加入windows客户端的IP地址。

initiator-address <windows_client_ip_address >