ceph object storage construction

Minimum machine:

1.osd1 192.168.1.2

2.osd2 192.168.1.3

3.monitor + radosgw 192.168.1.4

4.admin 192.168.1.5

Preliminary preparation:

0. All machines are initialized, change the hostname, ntp check time, close selinux, and close the firewall

1. Create a common user ceph_deploy on all machines (the new version of ceph must be started with a common user) and ensure that the new users of all machines have sudo privileges

useradd ceph_deploy

echo "{username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/{username}

sudo chmod 0440 /etc/sudoers.d/{username}

2. All machine hosts are added to each other for resolution

vim /etc/hosts
osd1 192.168.1.2
osd2 192.168.1.3
monitor 192.168.1.4
admin 192.168.1.5

3. All machines ceph_deploy user keys are mutually secret-free

ssh-keygen && ssh-copy-id ceph_deploy@{hostname}

4. The admin machine downloads and installs the ceph-deploy program

sudo subscription-manager repos --enable=rhel-7-server-extras-rpms

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

cat >/etc/yum.repos.d/ceph.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

sudo yum update

sudo yum install ceph-deploy

5. Modify the ~/.ssh/config file on the admin node, so that ceph-deploy can log in to other nodes with the username you created without specifying –username {username} every time you execute ceph-deploy.

Host node1
   Hostname node1
   User {username}
Host node2
   Hostname node2
   User {username}
Host node3
   Hostname node3
   User {username}

Create a cluster

Create a directory on the admin node as the admin directory

mkdir ceph

Create a cluster

ceph-deploy new monitor

After completion, there should be a Ceph configuration file, a monitor keyring and a log file in the current path.

change ceph.conf

 vim ceph.conf

 osd pool default size = 2

 如果你有多个网卡，可以把 public network 写入 Ceph 配置文件的 [global] 段下。详情见网络配置参考。

 public network = {ip-address}/{netmask}

Install Ceph

ceph-deploy install osd1 osd2 monitor admin

Go to each node to execute ceph –version to check the ceph installation result

ceph --version  

Configure initial monitor(s) and collect all keys:

ceph-deploy mon create-initial

After doing the above, these keyrings should appear in the current directory:

{cluster-name}.client.admin.keyring
{cluster-name}.bootstrap-osd.keyring
{cluster-name}.bootstrap-mds.keyring
{cluster-name}.bootstrap-rgw.keyring

configure osd node

Add a hard disk to osd1 and osd2 respectively, create partition /dev/sdb1, format it as xfs file system

add osd node

ceph-deploy osd prepare osd1:/dev/sdb1 osd2:/dev/sdb1

activate osd node

ceph-deploy osd activate osd1:/dev/sdb1 osd2:/dev/sdb1

Check whether the /dev/sdb1 partition is mounted successfully

In the two osd nodes, add the /dev/sdb1 partition to /etc/fstab and mount it automatically at boot

Building an Object Storage Gateway

install rgw

ceph-deploy install --rgw monitor

ceph-deploy rgw create monitor

Append the following configuration to the monitor node ceph.conf

vim /etc/ceph/ceph.conf 
[client.rgw.client-node]
rgw_dns_name = ceph.closeli.cn  请填写主域名,解析时采用泛解析
rgw_frontends = "civetweb port=80"

Generate a Ceph Object Gateway username and key

sudo ceph auth get-or-create client.radosgw.gateway osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring

Create a new RADOSGW user for S3 access

sudo radosgw-admin user create --uid="testuser" --display-name="First User"

python test code:

import boto.s3.connection

access_key = '1GB703PJH538VUE6H1AS'
secret_key = '2mAsFWEuVFV5ixqkOZ3XAtZvjpC7UbT7gtxChaCU'

conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        host = 'ceph.closeli.cn',
        is_secure=False,
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )

conn.create_bucket('my-new-bucket') #创建bucket

for bucket in conn.get_all_buckets():  #显示该用户下所有bucket
        print(f"{ bucket.name}\t{bucket.creation_date}")

a=conn.get_bucket('my-new-bucket') 获取指定bucket

key = a.new_key('hello.txt')  #创建一个文件

key.set_contents_from_string('Hello World!') #写入一个文件

for key in a: #获取文件内容
  print(f"{key.name}\t{key.size}\t{key.last_modified}")

key = a.get_key('hello.txt') #获取文件并下载

key.get_contents_to_filename('/Users/yeshaobin/Downloads/hello.txt')

key.generate_url(3600, query_auth=True, force_http=True) #获取文件url下载地址

print(plans_url)