文章目录

LVS 项目介绍
ipvsadm 功能
keepalived + ipvsadm

LVS 项目介绍

http://www.linuxvirtualserver.org/whatis.html

ipvsadm 功能

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.ipvsadm.html#using_ipvsadm

注意事项： put the names of the realservers in /etc/hosts， ipvsadm by default outputs the names of the realservers rather than the IPs， or you could use ipvsadm -n which outputs the IPs of the realservers instead.

功能/用途：（缺点：不能自动移除故障的RS，或添加修改后的RS,可以使用keepalived来支持)

添加服务 add services: add a service with weight >0 [ 0 to (2^24-1) ]
shutdown (or quiesce) services: set the weight to 0.
This allows current connections to continue, untill they disconnect or expire, but will not allow new connections. When there are no connections remaining, you can bring down the service/realserver.
删除服务delete services: this stops traffic for the service (the connection will hang), but the entry in the connection table is not deleted till it times out. This allows deletion, followed shortly thereafter by adding back the service, to not affect established (but quiescent) connections.
备份/恢复配置
$ ipvsadm-sav > ipvsadm.sav
and then after reboot, restore the ipvsadm settings, with ipvsadm-restore
$ ipvsadm-restore < ipvsadm.sav
Both of these commands can be part of an ipvsadm init script.

a, 使用场景

In a system where 700Mbit/s traffic is flowing through it
Concurrent connection number is about 420.000
Our main purpose for using LVS is to direct 80. port requests into number of squid servers (~80 servers)

b, 内存和性能

Each entry takes 128 bytes：the entries for each connection are stored in a hash table
LVS-DR + squid 缓存服务 + 420000并非连接 = 内存消耗【420000 x 128 x [RTTmin/RTTmin+maxIdleTime] 】bytes
粗算估计：LVS-DR + 4G内存~= 500万并发连接

c, 调度算法

round robin (rr), weighted round robin (wrr)
least connected (lc), weighted least connection (wlc)
DH: destination hash，makes a hash from the target IP and all requests to that IP will be sent to the same realserver
LBLC：a persistent memory algorythm, maybe lblc = dh + wlc.
SH: source hash
persistence：持久连接

d, 使用ipvsadm配置: LVS-DR

http://www.linuxvirtualserver.org/VS-DRouting.html
在这里插入图片描述
TCP报文：收发

客户端报文SrvMac->to-> VIP：balancer 接收后选择一个RS并转发给它；
RS拆报文，看到是VIP发过来的，而自己也有一个VIP, 于是封装报文VIP->to -> SrvMac

e, RS/LVS balancer配置VIP说明

RS和LVS balancer 配置 VIP=172.26.20.110

角色	执行动作
LVS balancer:172.26.20.111	ifconfig eth0:0 $VIP/24 broadcast $VIP up
	echo 1 > /proc/sys/net/ipv4/ip_forward
	ipvsadm -A -t $VIP:23 -s wlc
	ipvsadm -a -t $VIP:23 -r 172.26.20.112 -g
RS:172.26.20.112	ifconfig lo:0 $VIP/32 broadcast $VIP up #or ip addr add $VIP/32 dev lo label lo:0
	route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
	route add -host $VIP dev lo:0

On lo the interface will answer to all addresses covered by the netmask. This is how 127.0.0.1/8 on lo ends up answering 127.0.0.0-127.255.255.255.
So if you add 172.16.4.222/16 to eth0 then it will answer 172.16.4.222 and only 172.16.4.222. But if you add the same thing to lo then it will answer 172.16.0.0-172.16.255.255. So you need to use 172.16.4.222/32 instead.

#普通网卡eth0： vip  子网掩码/24 == > 仅响应vip
ifconfig eth0:0 192.168.10.10 netmask 255.255.255.0 broadcast 192.168.10.255 up
   -> Add 192.168.10.10 to eth0

#回环网卡lo： vip  子网掩码/24 == > 响应0-255个IP
ifconfig lo:0 192.168.10.10 netmask 255.255.255.0 broadcast 192.168.10.255 up
   -> Add 192.168.10.0 - 192.168.10.255 to lo
#			vip段ip.0  子网掩码/24 == > 响应0-255个IP		
ifconfig lo:0 192.168.10.0 netmask 255.255.255.0 broadcast 192.168.10.255 up
   -> Same as above, add 192.168.10.0 - 192.168.10.255 to lo
#			vip  子网掩码/32 == > 仅响应vip	
ifconfig lo:0 192.168.10.10 netmask 255.255.255.255 broadcast 192.168.10.10 up
   -> Add 192.168.10.10 to lo

keepalived + ipvsadm

https://www.keepalived.org/doc/introduction.html
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.failover.html#keepalived_vrrpd
功能/用途

Health checking for LVS systems: realserver 故障检查(移除故障的RS, 可以自动添加修复后的RS)
Implementation of the VRRPv2 stack to handle load balancer failover：VIP故障转移

[root@c72 ~]# ip a
6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.254/32 scope global vrrp.51
       valid_lft forever preferred_lft forever

[root@c72 ~]# tailf /var/log/messages
Dec 30 11:39:12 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
Dec 30 11:39:15 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
Dec 30 11:39:18 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Check on service [192.168.56.7]:80 failed after 3 retry.
Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Removing service [192.168.56.7]:80 from VS [192.168.56.254]:80


Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: HTTP status code success to [192.168.56.7]:80 url(1).
Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: Remote Web server [192.168.56.7]:80 succeed on service.
Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: Adding service [192.168.56.7]:80 to VS [192.168.56.254]:80

组成部件

keepalived daemon:
一个主（父）进程： in charge with forked children process monitoring.并重启失败的子进程
两个子进程： one responsible for VRRP framework and the other for healthchecking.
genhash utility

#配置virtual_server :HTTP_GET --> digest
[root@lvs-c6 ~]# grep digest /etc/keepalived/keepalived.conf
           url {
    
    
              path /index.html
              #digest 640205b7b0fc66c1ea91c463fac6334c
              status_code 200
            }
                      
[root@lvs-c6 ~]# genhash -s 192.168.56.7 -p 80 -u /index.html
MD5SUM = 3ab3417b4624b998d70adf9ac9fbb8ae

配置keepalived 使用VMAC地址通信

https://www.keepalived.org/doc/software_design.html#note-on-using-vrrp-with-virtual-mac-address

To reduce takeover impact, some networking environment would require using VRRP with VMAC address. To reach that goal Keepalived VRRP framework implements VMAC support by the invocation of ‘use_vmac’ keyword in configuration file

查看系统（MASTER和BACKUP) 是否支持 macvlan

[root@lvs-c6 ~]# lsmod | grep macvlan
macvlan                10356  0

#配置内核arp参数
[root@lvs-c6 ~]# tail /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.eth1.arp_filter = 1
[root@lvs-c6 ~]# sysctl -w

MASTER配置

global_defs {
    
    
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    
    
    state MASTER
    interface eth1
    virtual_router_id 51
    #use_vmac interface_prefix_name
    use_vmac
       vmac_xmit_base

    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.56.254
    }
}

virtual_server 192.168.56.254 80 {
    
    
    delay_loop 6
    lb_algo rr
    lb_kind DR
    #使用lvs持久连接：保证session连接同一个主机
    persistence_timeout 5
    protocol TCP

    #sorry_server 192.168.200.200 1358

    real_server 192.168.56.8 80 {
    
    
        weight 1
        HTTP_GET {
    
    
            url {
    
    
              path /index.html
              #digest 640205b7b0fc66c1ea91c463fac6334d
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.56.7 80 {
    
    
        weight 1
        HTTP_GET {
    
    
            url {
    
    
              path /index.html
              #digest 640205b7b0fc66c1ea91c463fac6334c
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}


#启动后，生成一个网卡 ,名称为 vrrp.{virtual_router_id}
[root@lvs-c6 ~]# ip a
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:8c:e4:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.6/24 brd 192.168.56.255 scope global eth1
    inet6 fe80::a00:27ff:fe8c:e4d6/64 scope link
       valid_lft forever preferred_lft forever
6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.254/32 scope global vrrp.51
    inet6 fe80::200:5eff:fe00:133/64 scope link
       valid_lft forever preferred_lft forever

生成LVS服务

当realserver故障时，会自动删除它；恢复后，会自动添加
当keepalived-MASTER故障时，BACKUP自动切换到VIP

[root@lvs-c6 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.56.254:80 rr persistent 5
  -> 192.168.56.7:80              Route   1      0          0
  -> 192.168.56.8:80              Route   1      0          0

在这里插入图片描述

BACKUP配置

同上：只有以下差别:state , priority

[root@c72 ~]# cat /etc/keepalived/keepalived.conf
...
vrrp_instance VI_1 {
    
    
    state BACKUP
	...
	 priority 60
...

#启动后，生成一个网卡 ,名称为 vrrp.{virtual_router_id}
[root@c72 ~]# ip a
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:9b:af:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.77/24 brd 192.168.56.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe9b:af24/64 scope link
       valid_lft forever preferred_lft forever
6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff

LVS集群: linux virtual server