Ceph RGW负载均衡和高可用配置

配置主要参考

（配置负载均衡有多种方法，这里是使用keepalived 用于高可用，haproxy 用于负载均衡）

注意：负载均衡节点不能与任何RGW节点重合。

架构示意图如下：

我的三个RGW分别为

R3S25：172.16.50.25

R3S43：172.16.50.43

R3S44：172.16.50.44

两个LB服务器

R3S24：172.16.50.24

R3S42：172.16.50.42

VIP

虚拟ip：172.16.50.100

1.安装keepalived haproxy

yum install keepalived haproxy -y

选取两个均衡节点(均衡节点不能和rgw节点重合)，下面在均衡节点（LB）执行。

开启linux ip转发功能

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

sysctl -p

允许绑定到非本地ip

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

sysctl -p

检查：

/usr/sbin/sysctl net.ipv4.ip_nonlocal_bind

/usr/sbin/sysctl net.ipv4.ip_forward

cat /proc/sys/net/ipv4/ip_forward

查看是否看起了ip转发功能

如果上述文件中的值为0,说明禁止进行IP转发；如果是1,则说明IP转发功能已经打开。

2.修改keepalived + Haproxy配置文件

下面所有操作均只需在LB（负载均衡器）节点执行

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
vrrp_script chk_haproxy {
    script "killall -0 haproxy" # 检查haproxy进程
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface enp175s0f1    # 似情况修改
    virtual_router_id 1      # id号局域网唯一
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.50.100        # VIP，配好了，客户端连接该ip即可高可用
    }
    track_script {
        chk_haproxy
    }
}

haproxy 的配置文件

vim /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    stats refresh           30s
    stats uri /stats #统计页面url
    stats realm baison-test-Haproxy #统计页面密码框上提示文本
    stats auth admin:admin123 #统计页面用户名和密码设置
    stats hide-version #隐藏统计页面上HAProxy的版本信息

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend rgw *:80     # 侦听端口
    mode http
    default_backend    rgw
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend rgw
    mode http
    balance    roundrobin     # 均衡模式，当前为轮循
    server R4S43 172.16.50.43:7480 check   # 填写真实网关IP和端口
    server R4S44 172.16.50.44:7480 check
    server R3S25 172.16.50.25:7480 check

3. 启动服务

systemctl start keepalived

systemctl enable keepalived

systemctl start haproxy

systemctl enable haproxy

关闭防火墙

systemctl stop firewalld

查看状态

systemctl status keepalived

systemctl status haproxy

关闭电脑的防火墙

在浏览器中打开http://ip:80/stats

ip你当前BL服务器的ip，即：172.16.50.24

输入账号/密码：admin/admin123

即可在web页面中看到当前负载均衡的状态