The most comprehensive in history. Introduction to working principle, detailed explanation of DR mode, with practice questions

1. Cluster and distributed

System performance expansion method: scale (ratio)

• scale UP (increase in proportion): vertical expansion, upward expansion, enhancement, more powerful computers running the same service
• Scale Out (expanding outward): horizontal expansion, outward expansion, adding equipment, running multiple servers in parallel, scheduling allocation problems, cluster

Vertical expansion no longer mentions:

As the need for computer performance increases, its price will also double

The performance of a single computer is limited, and it is impossible to expand vertically indefinitely.

Multi-core CPU means that even a single computer can be parallelized, so why it is best to use parallel technology in the beginning

1.2. Cluster

Cluster A single computer system formed by combining multiple computers to solve a specific problem

Cluster is divided into three types

• ​ LB Load Balancing. Load balancing , composed of multiple hosts, each host only undertakes a part of the access request
• HA High Availiablity, high availability , avoid SPOF (single point of connection failure)
• High-performance computing, high-performance www.top500.org

1.3, distributed system

Distributed storage: Ceph, GlusterFS, FastDFS, MogileFS

Distributed computing: hadoop, Spark

Distributed common applications

• Distributed applications -services are split according to functions, using microservices
• Distributed static resources -static resources are placed on different storage clusters
• Distributed data and storage- use key-value caching system
• Distributed computing- use distributed computing for special services, such as Hadoop clusters

1.4. Cluster and Distributed

Cluster : The same business system is deployed on multiple servers. In the cluster, there is no difference in the functions implemented by each server, and the data and code are the same

Distributed: A business is split into multiple sub-businesses, or different businesses themselves are deployed on multiple servers. In distributed, the functions implemented by each server are different, and the data and code are also different. The functions of each distributed server are combined, and the complete business distribution is improved by shortening the execution time of a single task. Efficient, while the cluster improves efficiency by increasing the number of tasks executed per unit time.

For large-scale websites, there are many visiting users, and a cluster is implemented, a load balancing server is deployed in the front, and the following servers complete the same business. If a user accesses the corresponding business, the load balancer decides which server to complete the response according to the load situation of which server at the back end, and if one server goes down, other servers can come up. Each distributed node completes a different business. If one node goes down, the business may fail

1.4.1 Principles of cluster design

Scalability -the horizontal expansion capability of the cluster

Availability - no downtime

Performance- access response time

Capacity ---Maximum concurrent throughput per unit time

1.4.2 Cluster design and implementation

Improve hardware resource performance —from the entrance firewall to the back-end web server, use higher-performance hardware resources

Multi-domain name— DNS polling A record resolution
Multi-entry —resolve A record to multiple public network IP entrances
Multi-computer room —same city + remote disaster tolerance
CDN (Content Delivery Network)—based on GSLB (Global Server Load Balance) to achieve global load balancing , Such as: DNS

1.4.3 Business level

Layering: Security layer, load layer, static layer, dynamic layer, (cache layer, storage layer) Persistence and non-persistence.
Segmentation: segmentation of large services into small services based on functions.
Distributed: For services in special scenarios, use distributed Calculation

2 Introduction to Linux Virtual Server

LVS: Linux Virtual Server, load scheduler, kernel integration, Zhang Wensong (huaming Zhengming), Ali's four-layer SLB (Server Load Balance) is implemented based on LVS+keepalived

2.1 The working principle of LVS

The VS forwards the scheduling to a certain RS according to the target IP, target protocol and port of the request message, and selects the RS according to the scheduling algorithm. LVS is a kernel-level function that works in the position of the INPUT chain and "processes" the traffic sent to INPUT

2.1 LVS cluster system architecture

2.3 LVS function and organizational structure

The application scenario of load balancing is a high-traffic business, which improves the availability and reliability of applications.

2.3.1 Applied to high-traffic businesses

If your application has high traffic, you can distribute traffic to different ECS (Elastic
Compute Service) instances by configuring listening rules . In addition, you can use the session persistence function to forward requests from the same client to the same backend ECS

2.3.2 Extended application

According to the needs of business development, ECS instances can be added and removed at any time to expand the service capabilities of the application system , which is suitable for various Web servers and App servers.

2.3.3 Eliminate single points of failure

You can add multiple ECS instances under the load balancing instance . When some of the ECS instances fail, load balancing will automatically shield the faulty ECS instances and distribute requests to the normally running ECS ​​instances to ensure that the application system can still work normally

2.3.4 Intra-city disaster recovery (multi-zone disaster recovery)

In order to provide more stable and reliable load balancing service, Ah Li cloud has deployed more than load balancing is available in each district in order to achieve the same geographical area disaster recovery . When the main availability zone has a computer room failure or unavailability, the load balancer still has the ability to switch to another backup availability zone in a very short period of time (for example, about 30s interruption) to restore service capabilities; when the main availability zone is restored, the load balancer will also automatically Switch to the main availability zone to provide services.

When using load balancing, you can deploy load balancing instances in regions that support multiple availability zones to achieve intra-city disaster recovery. In addition, it is recommended that you consider the deployment of back-end servers based on your own application needs. If you add at least one ECS instance to each Availability Zone, then the efficiency of load balancing services under this deployment mode is the highest.

As shown in the figure below, bind ECS instances in different availability zones under the load balancing instance. Under normal circumstances, user access traffic will be forwarded to ECS instances in the primary and backup availability zones at the same time; when Availability Zone A fails, user access traffic will only be forwarded to the ECS instances in the backup availability zone. This deployment can not only avoid the unavailability of external services due to the failure of a single availability zone, but also reduce the delay through the choice of availability zones between different products.

If the deployment scheme shown in the following figure is adopted, that is, multiple ECS instances are bound to the primary availability zone of the load balancing instance, but there is no ECS instance in the backup availability zone. When the primary availability zone fails, business will be interrupted because there is no ECS instance in the backup availability zone to receive requests. This deployment method is obviously to obtain low latency at the expense of high availability.

2.3.5 Cross-regional disaster recovery

You can deploy load balancing instances in different regions and mount ECSs in different availability zones in the corresponding regions. The upper layer uses cloud resolution as a smart DNS to resolve domain names to the service addresses of load balancing instances in different regions to achieve global load balancing. When a certain area is unavailable, suspend the corresponding analysis to realize that all users' access is not affected.

2.4 Terminology in LVS Cluster Type

VS: Virtual Server, Director Server (DS), Dispatcher (scheduler), Load Balancer
RS: Real Server (lvs), upstream server (nginx), backend server (haproxy)
CIP: Client IP
VIP: Virtual serve IP VS extranet IP virtual IP address
DIP: Director IP VS Intranet IP is mainly used to communicate with internal hosts IP address
RIP: Real server IP Real IP
access process: CIP <--> VIP == DIP <--> RIP

3 LVS working mode and related commands

3.1 Working modes of LVS cluster (four types)

lvs-nat: modify the target IP of the request message, DNAT of multi-target IP Dnat is just a point-to-point service
lvs-dr: manipulate and encapsulate the new MAC address
lvs-tun: add a new IP header lvs- in addition to the original request IP message
fullnat: modify the source and destination IP of the request message

1. The user sends a request to

2. Scheduling algorithm/rewrite packet

3. Processing the request

4. Rewrite reply through load balancing machine

5, then do reply

lvs-nat: The essence is a DNAT with multiple target IPs, which realizes forwarding
by modifying the target address and target port in the request message to the RIP and PORT of a selected RS (1) RIP and DIP should be on the same IP network, And the private network address should be used; the RS gateway must point to DIP
(2) Both the request message and the response message must be forwarded by the Director, which is easy to become a system bottleneck.
(3) Support port mapping and modify the target PORT of the request message
( 4) VS must be a Linux system, RS can be any OS system

SRC IP source address

Dst IP destination address

Port destination address

3.2 !!!LVS DR mode!!!

LVS-DR: Direct Routing, direct routing, LVS default mode, the most widely used, by re-encapsulating a MAC header for the request message for forwarding, the source MAC is the MAC of the interface where the DIP is located, and the destination MAC is the selected RS The MAC address of the interface where RIP is located; the source IP/PORT and destination IP/PORT remain unchanged

Features of DR mode:

1. Director and each RS are configured with VIP

2. Ensure that the front-end router sends the request message whose target IP is VIP to the Director

   • Statically bind the MAC address of VIP and Director at the front-end gateway

   • Use arptables tool on RS

   arptables -A IN -d $VIP -j DROP
   arptables -A OUT -s $VIP -j mangle --mangle-ip-s $RIP

   • Modify the kernel parameters on RS to limit the arp notification and response level

/proc/sys/net/ipv4/conf/all/arp_ignore
/proc/sys/net/ipv4/conf/all/arp_announce

3. RS RIP can use private network address or public network address; RIP and DIP are on the same IP network; RIP gateway cannot point to DIP to ensure that the response message will not pass through the Director
4. RS and Director must be on the same physical network
5. The request message goes through the Director, but the response message does not go through the Director, but is sent directly to the Client by the RS
6. Does not support port mapping (the port cannot be modified)
7. No need to enable ip_forward
8. RS can use most OS systems

4. Experiment: In DR mode, vip is not implemented on the same network segment (exaggerated network segment)

4.1 Preparation of the experimental environment

Five machines used in the experiment

• Client
• router
• LVS server
• Back-end RS1 server
• Backend RS2 server

1. One route host network card is bridged to the host, and the other is in NAT mode. And turn on the core forwarding function.

vim /etc/sysctl.conf
#添加一下行
net.ipv4.ip_forward=1
sysctl -p

2. The back-end server RS ​​installs httpd separately and provides a different homepage for testing.

yum install httpd -y
systemctl start httpd
echo webRS1>/var/www/html/index.html
echo webRS2>/var/www/html/index.html

3. LVS server installs two network cards

4.2 Specific steps of the experiment

1. Configure LVS server

#安装ipvsadm
yum install ipvsadm -y
#配置两块网卡的ip 分别为DIP :192.168.16.8 VIP :10.0.0.0.100/32
#启用核心转发
vim /etc/sysctl.conf
#添加一下行
net.ipv4.ip_forward=1
sysctl -p
#配置后端RS服务器 (分别再后端服务器上面关闭lo网卡 ARP应答和广播请求
#管理ARP广播请求和应答
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
#配置lo网卡差点VIP
ifconfig lo:1 10.0.0.100/32
＃使用curl命令行工具可以发现网页以1:2的权重被访问
while true; do curl 10.0.0.100;done

5. In CentOS7.6, the nfs client uses /etc/fatab to automatically mount at boot

#创建共享文件夹
mkdir /data/nfs -pv
setfacl -m u:nobody:rwx /data/nfs
#安装nfs和rpc
yum install nfs-utils rpcbind -y
#修改配置文件
vim /etc/exports
/data/nfs       10.0.0.0/24(rw,all_squash,async)
#重读配置文件
exportfs -r
systemctl start nfs-server rpcbind
#在centos7 修改/etc/fstab 添加一下行
vim /etc/fstab
/data           10.0.0.8:/data/nfs  nfs _netdev,noexec      0 0

6. In CentOS7.6, the nfs client uses autofs to realize real-time mounting when using nfs

Two methods

Method one: relative path method

yum install autofs -y
vim /etc/auto.master
/data   /etc/data.nfs
vim /etc/data.nfs
*   10.0.0.8:/data/nfs

Method two: absolute path method

vim /etc/auto.master
/-  /etc/nfs.auto
vim /etc/nfs.auto
/data/nfs       -ftype=nfs,noexec       10.0.0.8:/data/nfs