Article directory
1. Overview of LVS
LVS is the abbreviation of Linux Virtual Server. It is a high-availability and high-performance load balancing technology based on the Linux kernel. It can distribute requests from clients to multiple servers, realize load balancing of multiple servers, and improve the performance and availability of the entire system.
LVS technology mainly includes the following components:
- LVS scheduler: responsible for receiving client requests and distributing them to real backend servers, and distributing them according to different load balancing algorithms.
- Real server: handles requests from the scheduler and returns responses, providing actual services.
- Keepalived: A high-availability component of LVS, which is used to monitor the status of the LVS scheduler and automatically switch to the standby scheduler in case of failure to ensure high availability of services.
- IPVS: A module that implements LVS technology in the kernel, and implements functions such as load balancing algorithm and request distribution.
LVS technology is widely used in Internet services, online games, data centers and other fields, which can improve system performance and availability and reduce system maintenance costs.
2. Basic operation of LVS
1) Basic command operation
1. Add rules
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p timeout] [-M netmast] [--pepersistence_engine] [-b sched-flags]
2. Delete rules
ipvsadm -D -t|u|f service-address
3. Clear all the contents of the definition
ipvsadm -C
4. Overload
ipvsadm -R
6, save
ipvsadm -S [-n]
7. Add and change RS rules
ipvsadm -a|e -t|u|f service-address -r server-address [-g|i|m] [-w weight]
8. Delete the RS rule
ipvsadm -d -t|u|f service-address -r server-address
9. View the list of rules
ipvsadm -Ln|l [options]
--numeric, -n: 以数字形式输出地址和端口号
--exact: 扩展信息，精确值
--stats: 统计信息
--rate: 输出速率信息
10. Clear the counter
ipvsadm -Z [-t|u|f service-address]
11. ipvs rules
/proc/net/ip_vs
12. ipvs connection
/proc/net/ip_vs_conn
2) Save and reload rules
1, save
It is recommended to save to/etc/sysconfig/ipvsadm
ipvsadm-save -n > /PATH/TO/IPVSADM_FILE
ipvsadm -Sn > /PATH/TO/IPVSADM_FILE
systemctl stop ipvsadm.service
2. Overload
ipvsadm-restore < /PATH/TO/IPVSADM_FILE
ipvsadm -R < /PATH/TO/IPVSADM_FILE
systemctl restart ipvsadm.service
3. Explanation of the actual operation of the four modes of LVS
1) NAT mode
1. Design points
- RIP and DIP are in the same IP network, and the gateway of RIP should point to DIP
- Support port mapping
- Director should open the core forwarding function
2. Configuration
1. Manage cluster services: add, modify, delete
- increase, change
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p timeout]
- delete:
ipvsadm -D -t|u|f service-address
- service-address:
-t|u|f:
-t：TCP协议的端口，VIP：TCP_PORT
-u：UDP协议的端口，VIP：UDP_PORT
-f：firewall MARK，标记，一个数字
[-s scheduler]：指定集群的调度算法：默认为wlc
2. RS on the management cluster: add, modify, delete
- 增、改：ipvsadm -a|e -t|u|f service-address -r server-address [-g|i|m] [-w weight]
- server-address:
rip[:port] If port is omitted, no port mapping - option: lvs type:
-g：gateway, dr类型，默认
-i：ipip, tun类型
-m：masquerade，nat类型
-w weight：权重
3、ipvs scheduler
- ipvs scheduler : Whether to consider the current load status of each RS according to its scheduling.
Two types: static method and dynamic method - Static method : schedule only according to the algorithm itself
1、RR: roundrobin, 轮训
2、WRR: Weighted RR, 加权轮训
3、SH: Source Hashing, 实现session sticky, 源IP地址hash; 将来自同一个IP地址的请求始终发往第一次挑中的RS，从而实现会话绑定
4、DH: Destination Hashing; 目标地址哈希，将发往同一个目标地址的请求始终转发至第一次挑中的RS, 典型使用场景是正向代理缓存场景中的负载均衡，如：带宽运营商
4. Experiment: Realize LVS in NAT mode (must go back the same way)
ip_forward=1
route add default gw 192.168.0.201
# -t:tcp, -s wrr:加权 轮训
ipvsadm -A -t 172.20.0.200:80 -s wrr
# -m: NAT模式；默认：DR模式，不支持映射到不同端口；-w：权重,默认是1
ipvsadm -a -t 172.20.0.200:80 -r 192.168.30.17:8080 -m -w 3
ipvsadm -a -t 172.20.0.200:80 -r 192.168.30.27:8080 -m
2.router：路由器配置
ip_forward=1
route add default gw 192.168.0.200
2) DR mode
1. In the DR model, VIPs need to be configured on each host. There are three ways to resolve address conflicts :
- Do static binding on the front-end gateway
- Use arptables in each RS
- Modify kernel parameters in each RS to limit the level of arp response and notification
2. Limit response level: arp_ignore
0: The default value, which means that any local interface can be configured to respond at any address1: Only when the request target IP is configured on the interface of the local host that receives the request message, a response will be given
3. Limit announcement level: arp_announce
0: Default value, notify all information of all interfaces of this machine to the network of each interface1: Try to avoid notifying the interface information to the non-directly connected network2: It is necessary to avoid advertising interface information to non-networks
4. Experiment: Realize LVS in DR mode ( without backtracking )
-
Step 1 : Prepare 3 virtual machines
-
Step 2 : First configure the network of the 3 virtual machines
eth0 配置在一个网段
DIP,RIP配置在一个网段
- Step 3 : Configure the VIP of lvs
ifconfig ens33:0 192.168.182.100/24
echo "1" > /proc/sys/net/ipv4/ip_forward
- Step 4 (RS) : Adjust RS response, notification level (every RS is configured)
echo 1 > /proc/sys/net/ipv4/conf/ens33/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/ens33/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
- Step 5 : Configure VIP on RS
ifconfig lo:8 192.168.182.100 netmask 255.255.255.255
- Step 6 : Start the httpd service on RS
yum install httpd -y
cd /var/www/html
vi index.html
service httpd start
客户端验证：RIP:80能显示
VIP:80不能显示
- Step 7 : Install LVS—ipvsadm
yum install ipvsadm -y
# -t:tcp, -s rr:轮训
ipvsadm -A -t 192.168.182.100:80 -s rr
# -m: NAT模式；默认：DR模式，不支持映射到不同端口；-w：权重,默认是1；-g：DR模型，-m：NET模型
ipvsadm -a -t 192.168.182.128:80 -r 192.168.182.129 -g
ipvsadm -a -t 192.168.182.128:80 -r 192.168.182.130 -g
ipvsadm -ln
# 浏览器刷新：访问vip
ipvsadm -lnc
netstat -natp
3) TUN mode
4) FULL-NAT mode
4. Keepalived + LVS actual operation
Keepalived is a high availability software for Linux platform. It implements the Virtual Router Redundancy Protocol (VRRP) and health check functions that can be used to ensure high availability of services across multiple servers. Keepalived can detect server failures, and when the primary server goes down, it will automatically promote the backup server to the primary server to ensure service continuity and availability.
Keepalived can dynamically allocate virtual IP addresses between the active and standby servers, enabling clients to seamlessly switch between the active and standby servers, improving service availability. Additionally, Keepalived supports text file-based configuration and SNMP-based monitoring. It works with commonly used load balancers such as HAProxy, Nginx, etc.
Overall, Keepalived is a powerful tool for providing high availability services. It is a free and open source software widely used in enterprise and personal server environments.
The architecture diagram is as follows:
1) keepalived installation and basic operations
# 安装
yum install keepalived -y
# 启动
service keepalived start
# 配置文件位置
/etc/keepalived/keepalived.conf
# 查看日志
tail -f /var/log/message
2) Specific configuration steps
-
[Step 1] Prepare at least four virtual machines
-
【Step 2】Adjust the response and notification level of RS (every RS is configured)
echo 1 > /proc/sys/net/ipv4/conf/ens33/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/ens33/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
- Step 3: Configure VIP on RS, remember (DR mode) don’t forget to configure VIP on RS, otherwise data packets will be discarded
ifconfig lo:8 192.168.182.100 netmask 255.255.255.255
- Step 4: Install and start the httpd service in RS
yum install httpd -y
cd /var/www/html
echo "from ooxxip" > index.html
service httpd start
- Step 5: Install keepalived on two keepalived machines (one master and one backup)
yum -y install keepalived
yum -y install ipvsadm
- Step 6: Configure the keepalived configuration file
Master node configuration:
# /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
# vrrp_strict 如果还是访问不了VIP，可以把这行注释掉
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
# 主
state MASTER
# 备
# state BACKUP
interface ens33
virtual_router_id 51
# 主
priority 100
# 备
# priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.182.100/32 dev ens33 label ens33:7
}
}
virtual_server 192.168.182.100 80 {
delay_loop 6
lb_algo rr
lb_kind DR nat_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 192.168.182.130 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.182.131 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
Slave node configuration:
# 复制一份修改好的配置到从主机，修改 1、state BACKUP，2、priority 50
cd /etc/keepalived/ && scp ./keepalived.conf [email protected]:`pwd`
# 启动
/bin/systemctl start keepalived.service
# RS切记关闭防火墙，如果web也访问不了，lvs也可以关闭防火墙试试
systemctl stop firewalld
# 查看是否配置了VIP
#查看是否配置了规则
# 主未挂，备则会自动配置规则,但是不会配置VIP，保证只有一个VIP对外提供服务
# 一旦主挂了，则备接管（测试，down掉主网卡：ifconfig ens33 down）
Cons: split-brain problem
[Analysis reason]: keepalived itself is not highly available, the main process may be killed, but after the process is killed, the VIP is not recycled, resulting in the failure of the main keepalived to broadcast, and the standby keepalived cannot get the main broadcast signal, resulting in the backup will also be configured On the VIP, so that both the master and the backup have VIPs. Finally, the three-way handshake of CIP accessing the VIP may be broken up to the master and backup keepalived, and the connection cannot be established, resulting in inaccessibility.
【solution】:
- Write an automatic script to periodically check whether the main keepalived process is still alive, and if it is killed, restart the main keepalived service
- Switch to a more advanced high-availability technology (zookeeper), and there will be corresponding articles on zookeeper in the future
Finally, attach the keepalived configuration file description:
! Configuration File for keepalived
global_defs {
#全局定义部分
notification_email {
#设置报警邮件地址，可设置多个
[email protected] #接收通知的邮件地址
}
notification_email_from [email protected] #设置 发送邮件通知的地址
smtp_server smtp.163.com #设置 smtp server 地址，可是ip或域名.可选端口号 （默认25）
smtp_connect_timeout 30 #设置 连接 smtp server的超时时间
router_id LVS_DEVEL #主机标识，用于邮件通知
vrrp_skip_check_adv_addr
vrrp_strict #严格执行VRRP协议规范，此模式不支持节点单播
vrrp_garp_interval 0
vrrp_gna_interval 0
script_user keepalived_script #指定运行脚本的用户名和组。默认使用用户的默认组。如未指定，默认为keepalived_script 用户，如无此用户，则使用root
enable_script_security #如过路径为非root可写，不要配置脚本为root用户执行。
}
vrrp_script chk_nginx_service {
#VRRP 脚本声明
script "/etc/keepalived/chk_nginx.sh" #周期性执行的脚本
interval 3 #运行脚本的间隔时间，秒
weight -20 #权重，priority值减去此值要小于备服务的priority值
fall 3 #检测几次失败才为失败，整数
rise 2 #检测几次状态为正常的，才确认正常，整数
user keepalived_script #执行脚本的用户或组
}
vrrp_instance VI_1 {
#vrrp 实例部分定义，VI_1自定义名称
state MASTER #指定 keepalived 的角色，必须大写 可选值：MASTER|BACKUP
interface ens33 #网卡设置，lvs需要绑定在网卡上，realserver绑定在回环口。区别：lvs对访问为外，realserver为内不易暴露本机信息
virtual_router_id 51 #虚拟路由标识，是一个数字，同一个vrrp 实例使用唯一的标识，MASTER和BACKUP 的 同一个 vrrp_instance 下 这个标识必须保持一致
priority 100 #定义优先级，数字越大，优先级越高。
advert_int 1 #设定 MASTER 与 BACKUP 负载均衡之间同步检查的时间间隔，单位为秒，两个节点设置必须一样
authentication {
#设置验证类型和密码，两个节点必须一致
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
#设置虚拟IP地址，可以设置多个虚拟IP地址，每行一个
192.168.119.130
}
track_script {
#脚本监控状态
chk_nginx_service #可加权重，但会覆盖声明的脚本权重值。chk_nginx_service weight -20
}
notify_master "/etc/keepalived/start_haproxy.sh start" #当前节点成为master时，通知脚本执行任务
notify_backup "/etc/keepalived/start_haproxy.sh stop" #当前节点成为backup时，通知脚本执行任务
notify_fault "/etc/keepalived/start_haproxy.sh stop" #当当前节点出现故障，执行的任务;
}
virtual_server 192.168.119.130 80 {
#定义RealServer对应的VIP及服务端口，IP和端口之间用空格隔开
delay_loop 6 #每隔6秒查询realserver状态
lb_algo rr #后端调试算法（load balancing algorithm）
lb_kind DR #LVS调度类型NAT/DR/TUN
#persistence_timeout 60 同一IP的连接60秒内被分配到同一台realserver
protocol TCP #用TCP协议检查realserver状态
real_server 192.168.119.120 80 {
weight 1 #权重，最大越高，lvs就越优先访问
TCP_CHECK {
#keepalived的健康检查方式HTTP_GET | SSL_GET | TCP_CHECK | SMTP_CHECK | MISC
connect_timeout 10 #10秒无响应超时
retry 3 #重连次数3次
delay_before_retry 3 #重连间隔时间
connect_port 80 #健康检查realserver的端口
}
}
real_server 192.168.119.121 80 {
weight 1 #权重，最大越高，lvs就越优先访问
TCP_CHECK {
#keepalived的健康检查方式HTTP_GET | SSL_GET | TCP_CHECK | SMTP_CHECK | MISC
connect_timeout 10 #10秒无响应超时
retry 3 #重连次数3次
delay_before_retry 3 #重连间隔时间
connect_port 80 #健康检查realserver的端口
}
}
}
vrrp_instance VI_2 {
#vrrp 实例部分定义，VI_1自定义名称
state BACKUP #指定 keepalived 的角色，必须大写 可选值：MASTER|BACKUP 分别表示（主|备）
interface ens33 #网卡设置，绑定vip的子接口，lvs需要绑定在网卡上，realserver绑定在回环口。区别：lvs对访问为外，realserver为内不易暴露本机信息
virtual_router_id 52 #虚拟路由标识，是一个数字，同一个vrrp 实例使用唯一的标识，MASTER和BACKUP 的 同一个 vrrp_instance 下 这个标识必须保持一致
priority 90 #定义优先级，数字越大，优先级越高。
advert_int 1 #设定 MASTER 与 BACKUP 负载均衡之间同步检查的时间间隔，单位为秒，两个节点设置必须一样
authentication {
#设置验证类型和密码，两个节点必须一致
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
#设置虚拟IP地址，可以设置多个虚拟IP地址，每行一个
192.168.119.131
}
}
virtual_server 192.168.119.131 80 {
#定义RealServer对应的VIP及服务端口，IP和端口之间用空格隔开
delay_loop 6 #每隔6秒查询realserver状态
lb_algo rr #后端调试算法（load balancing algorithm）
lb_kind DR #LVS调度类型NAT/DR/TUN
#persistence_timeout 60 #同一IP的连接60秒内被分配到同一台realserver
protocol TCP #用TCP协议检查realserver状态
real_server 192.168.119.120 80 {
weight 1 #权重，最大越高，lvs就越优先访问
TCP_CHECK {
#keepalived的健康检查方式HTTP_GET | SSL_GET | TCP_CHECK | SMTP_CHECK | MISC
connect_timeout 10 #10秒无响应超时
retry 3 #重连次数3次
delay_before_retry 3 #重连间隔时间
connect_port 80 #健康检查realserver的端口
}
}
real_server 192.168.119.121 80 {
weight 1 #权重，最大越高，lvs就越优先访问
TCP_CHECK {
#keepalived的健康检查方式HTTP_GET | SSL_GET | TCP_CHECK | SMTP_CHECK | MISC
connect_timeout 10 #10秒无响应超时
retry 3 #重连次数3次
delay_before_retry 3 #重连间隔时间
connect_port 80 #健康检查realserver的端口
}
}
}
LVS + Keepalived introduction and practical operation demonstration are here first. If you have any questions, please leave me a message, or follow my public account [Big Data and Cloud Native Technology Sharing], and related technical articles will continue to be updated in the future~.