background

In the last chapter ( LVS NAT model ), we introduced and actually built a set of LVS NAT experiment environment. In the experiment environment, lvs is a single node, and the forwarding rules of lvs are manually configured through ipvsadm. We mentioned Keepalived before introducing lvs. Keepalived is designed for LVs. In the next article, we will introduce the DR model of LVS, and build a high-availability experiment of the LVS DR model through keepalived + lvs.

LVS DR Model Introduction

Manipulate and encapsulate new MAC addresses! !

Virtual Server via Direct Routing (VS-DR): Realize the virtual server with direct routing technology. This method can be used when the computers participating in the cluster and the computer as the control and management computer are on the same network segment. When the control and management computer receives the request packet It is sent directly to the nodes participating in the cluster. The direct routing mode is special, and it is hard to say what it is similar to. The former mode basically works on the network layer (three layers), while the direct routing mode should work on the data link upper floor (second floor).

As shown in the figure above, both the Director and the REAL SERVER are configured with the same IP (VIP), the Director configures the IP to the external network card, and the Real server configures the IP to the lo network card. Configure arp_ignore to 1 (the purpose is to let the data packet send an apr request, only the Director will respond to the arp request), and all REAL SERVERs will keep silent on the ARP request of their own IP. After receiving the data packet, the Director finds out the corresponding REAL SERVER according to the scheduling algorithm, changes the destination MAC address to the MAC of the REAL SERVER and sends it to the REAL SERVER. At this time, the REAL SERVER receives the data packet through the network card eth0. Since the lo network card on the Real Server is also configured with a VIP, the RS receives the data packet. After processing, it is directly returned to the client (arp_announce needs to be configured here to modify the source ip address of the returned data packet.). Since the DR needs to change the layer 2 packet header, the DR and the REAL SERVER must be in a broadcast domain, which can also be simply understood as being on the same switch.

1. When the user requests the target website, the destination IP is VIP and the destination port is 80 through dns query, so the client establishes a connection with our VIP and port 80. When the data packet arrives at the LAN where the VIP is located, in the same network segment, the two hosts communicate through the physical address of the second layer instead of the IP address, so the IP address needs to be converted into a MAC address, so an apr request will be issued to query The mac address corresponding to the VIP. ==Linux host has such a feature. Suppose we have two network cards on the host, such as eth0 and eth1. When arp requests the mac address of eth1, eth1 will reply. This is a matter of course, but eth0 will also be "kind" Help eth1 answer this arp request. ==We configured VIP on the lo network card of Real Server, but we only want the network card on Director to respond to our arp request. Therefore, we need to change some of our kernel parameters, and the specific meanings are described later. At this time, the data packet gets the transmission address of the Layer 2 Director.

net.ipv4.conf.lo.arp_ignore = 1 
net.ipv4.conf.all.arp_ignore = 1

2. When the user request reaches DirectorServer, the requested data packet will first go to the PREROUTING chain of the kernel space. At this time, the source IP of the packet is CIP, and the destination IP is VIP.

3. The PREROUTING check finds that the destination IP of the data packet is the local machine, and sends the data packet to the INPUT chain.

4. IPVS compares whether the service requested by the data packet is a cluster service. If so, modify the source MAC address in the request message to the MAC address of DIP, modify the target MAC address to the MAC address of RIP, and then send the data packet to POSTROUTING chain. At this time, neither the source IP nor the destination IP has been modified, only the MAC address whose source MAC address is DIP, and the destination MAC address which is RIP MAC address.

5. Since DS and RS are in the same network, they are transmitted through Layer 2. The POSTROUTING chain checks that the target MAC address is the MAC address of RIP, then the data packet will be sent to the Real Server at this time.

6. After the client's request is forwarded by the Director and reaches the Realserver through link layer addressing, since the lo interface of the Realserver is configured with a VIP (the target IP in the request is exactly the VIP), the request is received and processed. After the processing is completed, the response message is sent to the eth0 network card through the lo interface (this network card generally refers to the network card on the same network segment as the scheduler) and then sent out. At this time, the source IP address is VIP, and the destination IP is CIP. ==If the source address is VIP and the data packet is sent out, then two mac address records corresponding to the VIP will be generated on the switch, one is the mac address record of the Director, and the other is the mac address record of the Real server, which will It will cause the real VIP to be unable to receive the request. == Therefore, arp_announce is to be configured here to modify the destination address of the source ip.

# 配置arp_announce=2，选择该主机发送网卡上最合适的本地地址作为arp请求的源IP地址。
net.ipv4.conf.lo.arp_announce = 2 
net.ipv4.conf.all.arp_announce = 2

Features of DR mode:

1. Ensure that the front-end routing sends all packets with the destination address of VIP to Director Server instead of RS.
2. RS can use private addresses; it can also be public network addresses. If public network addresses are used, RIP can be performed through the Internet at this time. Direct access
3. RS and Director Server must be in the same physical network
4. All request messages pass through Director Server, but response messages must not enter Director Server
5. Does not support address translation or port mapping
6. RS can be most common operating systems
7. The gateway of RS is never allowed to point to DIP
8. The lo interface on RS configures the IP address of VIP

LVS-DR experiment

The role of LVS:

1. Do four-layer load balancing, and forward the data to the seven-layer load of the backend NGinx;

2. In the DR model, lvs changes the source and destination MAC addresses of data packets for data forwarding;

Note: The port of the back-end RS must be the same as the port accessed by the user, because the DR model forwards data by modifying the MAC mirror image, and there is no source-destination ip and source-destination port of the data packet.

The role of keepalived:

1. Make high availability for lvs, generate vip, and avoid single point of failure;

2. lvs does not have an availability check function for the back-end RS. keepalived can be used to solve this problem. When the back-end rs server fails, keepalived can remove the faulty machine from the forwarding cluster according to the rules.

1. Network planning

equipment	ip	gateway
Client	10.10.10.128/24	-
Route-link1	10.10.10.129/24	-
Route-link2	192.168.10.10/24	-
LVS-vip	192.168.10.100/24	192.168.10.10
nginx-1	192.168.10.12/24	192.168.10.100
nginx-2	192.168.10.13/24	192.168.10.100
web-1	192.168.10.15/24	-
web-2	192.168.10.16/24	-

2. Route node configuration

Configure the route forwarding of the Route node, and configure the device ip. Combined with the actual production environment, port mapping needs to be enabled on the router.

##  由于是linux模拟的路由器，设置Route节点ip转发功能
root@Route:~# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
root@Route:~# sysctl -p
 
 
##开启端口映射
root@Route:~# iptables -t nat -A PREROUTING -d 10.10.10.129 -j DNAT --to 192.168.10.200

3. LVS node configuration

1. Configure route forwarding;
2. Configure IP address, gateway pointing to route;
3. Configure keepalived to set vip.
4. Configure ipvs forwarding

3.1 Configure route forwarding

##  主备节点均配置
 
root@Lvs-Master:~#echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
root@Lvs-Master:~# sysctl -p

3.2 Configure ip and gateway

3.3 configure keepalived

Master node

root@Lvs-Master:~# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from [email protected]
   smtp_server 192.168.65.129
   smtp_connect_timeout 30
   router_id keepalived-MASTER
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    garp_master_delay 10
    smtp_alert
    virtual_router_id 51
    priority 100
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.200
    }
}
##配置集群访问的IP+port
virtual_server 192.168.10.200 80 {
    ## 健康检查时间（检查后端rs的连通性），单位S
    delay_loop 6
    #配置负载均衡算法
    lb_algo rr
    # 设置LVS模型（NAT,DR,TUN）
    lb_kind DR
    # 设置回话持续时间
    persistence_timeout 50
    # 设置协议
    protocol TCP


    #配置后端RS节点（RS-1）
    real_server 192.168.10.12 80 {
        # 设置权重
        #设置健康检查
        TCP_CHECK {
            #检查后端的80端口
            connect_port 80
            # 超时时间
            connect_time 3
            # 重试次数
            retry 2
            #时间间隔
            delay_before_retry 3
           
        }
    }

        #配置后端RS节点（RS-2）
    real_server 192.168.10.13 80 {
        # 设置权重
        #设置健康检查
        TCP_CHECK {
            #检查后端的80端口
            connect_port 80
            # 超时时间
            connect_time 3
            # 重试次数
            retry 2
            #时间间隔
            delay_before_retry 3
           
        }
    }
}

Backup node

root@LVS-Backup:~# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from [email protected]
   smtp_server 192.168.65.129
   smtp_connect_timeout 30
   router_id keepalived-MASTER
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    garp_master_delay 10
    smtp_alert
    virtual_router_id 51
    priority 10
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.200
    }
}
##配置集群访问的IP+port
virtual_server 192.168.10.200 80 {
    ## 健康检查时间（检查后端rs的连通性），单位S
    delay_loop 6
    #配置负载均衡算法
    lb_algo rr
    # 设置LVS模型（NAT,DR,TUN）
    lb_kind DR
    # 设置回话持续时间
    persistence_timeout 50
    # 设置协议
    protocol TCP


    #配置后端RS节点（RS-1）
    real_server 192.168.10.12 80 {
        # 设置权重
        weight 1
        #设置健康检查
        TCP_CHECK {
            #检查后端的80端口
            connect_port 80
            # 超时时间
            connect_time 3
            # 重试次数
            retry 2
            #时间间隔
            delay_before_retry 3
           
        }
    }

        #配置后端RS节点（RS-2）
    real_server 192.168.10.13 80 {
        # 设置权重
        weight 1
        #设置健康检查
        TCP_CHECK {
            #检查后端的80端口
            connect_port 80
            # 超时时间
            connect_time 3
            # 重试次数
            retry 2
            #时间间隔
            delay_before_retry 3
           
        }
    }
}

4. Nginx node

1. Configure the loopback interface ip as VIP;
2. Configure arm to ignore;
3. Configure the route to point to the router;
5. Configure Nginx load balancing (7-layer load)

4.1 Configure the loopback interface

##  所以RS节点均要配置
root@proxy-2:~#ifconfig lo:0 192.168.10.200 netmask 255.255.255.255

4.2 Configure arp to ignore

##  所有RS节点均要配置
root@proxy-1:~# sysctl -p
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_ignore = 1
root@proxy-1:~#

4.3 Configure routing

## 所有RS节点均要配置
root@proxy-1:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.10.10   0.0.0.0         UG    20100  0        0 ens33
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 ens33
192.168.10.0    0.0.0.0         255.255.255.0   U     100    0        0 ens33
root@proxy-1:~#

4.4 Configure load balancing

##  节点1
root@proxy-1:~# cat /etc/nginx/conf.d/web1.conf 
upstream lvsservers {
	server 192.168.10.15:80;
	server 192.168.10.16:80;
}

server {
	listen 80;
	server_name lvs_test.com www.lvs_test.com;
	root /https;

	location / {
		proxy_pass http://lvsservers;
		include proxy_params;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}

}



##节点2
root@proxy-2:~# cat /etc/nginx/conf.d/web2.conf 
upstream lvsservers {
	server 192.168.10.15:80;
	server 192.168.10.16:80;
}
server {
	listen 80;
	server_name lvs_test.com www.lvs_test.com;

	location / {
		proxy_pass http://lvsservers;
		include proxy_params;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}

}

5. Configure the web node

##  web1节点
root@web-1:~# cat /etc/nginx/conf.d/web.conf 
server {
	listen 80;
	server_name lvs_test.com www.lvs_test.com;
 
	location / {
		root /https;
		index index.html;
	}
}
root@web-1:~# cat /https/index.html 
web1
root@web-1:~#
 
 
##web2节点
root@web-2:~# cat /etc/nginx/conf.d/web.conf 
server {
	listen 80;
	server_name lvs_test.com www.lvs_test.com;
 
	location / {
		root /https;
		index index.html;
	}
}
root@web-2:~# cat /https/index.html 
web2
root@web-2:~#