Ceph distributed storage series (7): Installation, configuration and common commands of object storage RGW and S3cmd

Continuing the above Ceph distributed storage series (2): deploying a three-node ceph cluster using ceph-deploy method

这里就不详细介绍对象存储了，感兴趣可以在这里查看：

      Ceph distributed storage series (1): A brief analysis of the working principle and architecture of Ceph
      Ceph distributed storage series (6): The differences, advantages and disadvantages of object storage, block storage, and file storage

Foreword:

This article introduces two major pieces of content:

1. Installation, use and common commands of RGW in clusters deployed by ceph-deploy
2. Installation, use and common commands of object storage management tool S3cmd

1. RGW installation

Operate based on cluster deployed by ceph-deploy

Note: If it was not installed by ceph-deploy before, it is best to use the previous source code installation to prevent errors.

1. Install and enable RGW components

$ cd /root/cluster/
$ ceph-deploy rgw create ceph-node1  （本次是测试环境，就用一个rgw网关节点来测试，实际环境中最好要三个）

查看集群中rgw状态
$ [ceph-admin@ceph-node1 ~]$ ceph -s
  cluster:
    id:     9575c356-be14-4455-8225-9788bba9d026
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-node1 (age 6h)
    mgr: ceph-node1(active, since 6h)
    osd: 3 osds: 3 up (since 4h), 3 in (since 4h)
    rgw: 1 daemon active (ceph-node1)
.......

查看服务端口是否正常
[root@ceph-node1 ~]# lsof -i:7480
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
radosgw 841 ceph   46u  IPv4  23934      0t0  TCP *:7480 (LISTEN)
radosgw 841 ceph   47u  IPv6  23936      0t0  TCP *:7480 (LISTEN)

You need to execute
cluster in the cluster directory. Stored in the directory is the configuration file generated by ceph-deploy when initializing the cluster.

2. Enable the object storage function in the default dashboard

1、创建dashboard使用的用户
（要使用仪表板的对象网关管理功能，需要提供system启用该标志的用户的登录凭据）
$ radosgw-admin user create --uid=<user_id> --display-name=<display_name> --system
eg：radosgw-admin user create --uid=dashboard-admin  --display-name=admin --system

记下此命令的输出中的access_key和secret_key。

2、可以手动查看用户的两个key
$ radosgw-admin user info --uid=<user_id>
eg：radosgw-admin user info --uid=dashboard-admin

3、将两个key保存到单独一个文件中，以便下一步使用
$ echo D3HTA2TRXBBE514USEQT > access_key
$ echo AZxoYU6u3DkLUw9OMnRewfx73DxhjpDICwSjEIwH > secret_key

4、最后，向仪表板提供凭据：
$ ceph dashboard set-rgw-api-access-key -i <file-containing-access-key>
$ ceph dashboard set-rgw-api-secret-key -i <file-containing-secret-key>
eg:
ceph dashboard set-rgw-api-access-key -i access_key
ceph dashboard set-rgw-api-secret-key -i secret_key
最后刷新dashboard页面即可

3. rgw common commands

查询（罗列）所有用户
radosgw-admin user list

创建rgw用户
radosgw-admin user create --uid="testuser" --display-name="first user"

查看rgw用户信息
radosgw-admin user info --uid="testuser"

删除用户
radosgw-admin user rm --uid="testuser"

暂定用户
radosgw-admin user suspend --uid="testuser"

启用用户
radosgw-admin user enable --uid="testuser"

创建子用户（swift接口使用）
radosgw-admin subuser create --uid=testuser_1 --subuser=testuser:swift --access=full

查看bucket桶
radosgw-admin bucket list

禁止删除bucket桶
radosgw-admin bucket delete disable --bucket=<bucket_name>

2. Installation, configuration and use of S3cmd

Amazon Amazon S3 has launched two tools for operating object storage clusters, s3cmd is one of them

1. s3cmd: It is a command line operation tool in Linux. It can not only upload, download, synchronize, but also set permissions.
2. s3brower: A graphical operation tool available in Windows, which is more convenient to operate. You can directly operate it by installing and configuring two keys.

I won’t introduce too much about s3brower. I mainly use s3cmd here.

1. S3cmd installation

$ yum -y install s3cmd   
#epel源中就包含这个包，没有的话可以试下wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

2. Create a user-generated key used by s3cmd

格式：$ radosgw-admin user create --display-name="[name]" --uid=[uid]
eg：radosgw-admin user create --uid=s3user  --display-name=admin

It is still necessary to save the two values ​​​​of access_key and secret_key.

3. Configure s3 so that it can connect to the ceph cluster

根据提示输入accessKey,securityKey 生成基本的配置文件
[root@ceph-node1 ~]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: D3HTA2TRXBBE514USEQT     ##此处填上一步获取到的access_key
Secret Key: AZxoYU6u3DkLUw9OMnRewfx73DxhjpDICwSjEIwH     #此处填上一步获取到的secret_key
Default Region [US]:   #默认即可，直接回车

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]:    #默认即可，直接回车

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:    #默认即可，直接回车

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no     #不使用https，填写no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:      #默认即可，直接回车

New settings:
  Access Key: D3HTA2TRXBBE514USEQT
  Secret Key: AZxoYU6u3DkLUw9OMnRewfx73DxhjpDICwSjEIwH
  Default Region: US
  S3 Endpoint: s3.amazonaws.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n    #测试访问，此时还没配置完，不测试，填写no

Save settings? [y/N] y    #是否保存，是，填写y
Configuration saved to '/root/.s3cfg'

还没结束，还要修改刚生成的/root/.s3cfg中的三处配置

cloudfront_host = [serverIP]（改成自己的服务端的IP）
host_base = [serverIP]:[Port]（改成自己的服务端的IP和端口）
host_bucket = [serverIP]:[Port]/%(bucket)（改成自己的服务端的IP和端口）

示例：
本地的ceph集群环境，rgw默认端口为7480
cloudfront_host = 192.168.1.51
host_base = 192.168.1.51:7480
host_bucket = 192.168.1.51:7480

4. Test the s3cmd command

创建名为test-bucket的bucket
[root@ceph-node1 ~]# s3cmd mb s3://test-bucket
Bucket 's3://test-bucket/' created

查看bucket桶列表
[root@ceph-node1 ~]# s3cmd ls
2021-10-29 07:13  s3://test-bucket
即s3配置正常，可正常连接集群

5. Common commands of S3cmd

Operations for buckets:

创建bucket
$ s3cmd mb s3://{
    
    bucket_name}

删除bucket（bucket需为空）
$ s3cmd rb s3://{
    
    bucket_name}

查看bucket列表或bucket内文件列表
s3cmd ls
s3cmd ls s3://{
    
    bucket_name}

Operations on files in bucket:

上传文件到bucket中
$ s3cmd put fio-fio-3.10.zip s3://test-bucket

删除文件
s3cmd del s3://test-bucket/file.txt

批量删除文件
s3cmd del s3://test-bucket/aa*
s3cmd del s3://test-bucket/test/*

批量上传文件
$ s3cmd put test/* s3://test-bucket

递归上传文件（可上传整个文件夹-包含文件夹）
   #-r	递归参数，全称为：--recursive
$ s3cmd put -r /root/test s3://test-bucket

同步目录下文件至bucket中（应该类似于git合流代码）
s3cmd sync ./test/ s3://test-bucket

复制bucket中文件到其他bucket中
s3cmd cp s3://test-bucket/aaaa s3://test-bucket-2

下载文件
s3cmd get s3://test-bucket/file.txt
s3cmd get s3://test-bucket/file.txt /root/test/

Operations for permissions:

将文件权限设置为所有人可读
$ s3cmd setacl --acl-public s3://test-bucket/file.txt

将bucket中整个文件夹设置权限为私有读（递归权限，文件夹下所有文件都生效）
$ s3cmd setacl --acl-private -r s3://test-bucket/test/

End……