Meituan 2 side: 5 nines, high availability 99.999%, how to achieve it?

Said it in front

In the reader community (50+) of the 40-year-old architect Nien , some friends have recently obtained interview qualifications from first-tier Internet companies such as NetEase, Youzan, Xiyin, Baidu, NetEase, and Didi, and met a few Very important interview questions:

Question 1: How to achieve high availability in your system?

Question 2: How to achieve 4 nines high availability of 99.99%?

Note that recently a friend of Meituan 2mian encountered this problem again: 5 nines are highly available at 99.999%. How to achieve it?

Nien reminded that the issue of high availability is the core knowledge of architecture and a key online problem.

Therefore, here Nien will give you a systematic and systematic thread pool sorting, so that you can fully demonstrate your strong "technical muscles" and make the interviewer "can't help himself and drool" .

This question and the reference answers are also included in the V63 version of our " Nion Java Interview Guide " for reference by subsequent friends to improve everyone's 3-level architecture, design, and development levels.

For the latest PDFs of "Nien Architecture Notes", "Nien High Concurrency Trilogy" and " Nien Java Interview Guide ", please go to the official account [Technical Freedom Circle] to obtain

Article directory

Three high principles of architectural design

Nowadays, the requirements for developing a software system are getting higher and higher. If you understand some "architecture design" requirements, you will know that a good software architecture should follow the following three principles:

high performance
High concurrency
high availability

High performance means that the system has greater traffic processing capabilities and lower response latency.

For example, 10W concurrent requests can be processed in 1 second, and the interface response time is 5 ms, etc.

High concurrency means that the system can be expanded at the minimum cost when iterating new functions. When the system encounters traffic pressure, the system can be expanded without changing the code.

High availability is usually measured by two indicators:

MTBF (Mean Time Between Failure): Indicates the time between two failures, which is the average time for the system to "normally operate". The longer this time, the higher the stability of the system.
Failure recovery time MTTR (Mean Time To Repair): Indicates the "recovery time" after a system failure. The smaller this value, the smaller the impact of the failure on the user.

The relationship between availability and these two:

Availability = MTBF / (MTBF + MTTR) * 100%

The result of this formula is a "proportion". Usually we use "N nines" to describe the availability of a system.

You can see from this picture that in order to achieve availability of more than four nines, the downtime in a year is 52 minutes, and the average daily failure time must be controlled within 10 seconds.

System failure is actually inevitable, especially the larger the system, the greater the probability of problems.

3 Nines Architecture – Universal Architecture

On the Alibaba Cloud platform, for small and medium-sized enterprises, the business volume is not particularly large, and the requirements for remote disaster recovery are not particularly strong.

The following high-availability solution can be used (as shown below: Figure 6),

You can choose to purchase cloud products in the same region.

It is recommended to select cloud products in the same availability zone or different availability zones in the same region in a VPC network environment.

At the same time, it is recommended to have at least two ECS servers to avoid single points of failure. Purchase SLB on the front end to provide load functions, so that when the back-end ECS resources are in short supply, they can be directly expanded horizontally without affecting the business.

In addition, try not to deploy database services and application services on the same ECS.

It prevents resource preemption between different services and facilitates daily management and later expansion. It is recommended to purchase RDS products directly for the database server. Data security is guaranteed and you do not need to spend too much energy on operation and maintenance management.

Introduction to high-availability SLB components

The Alibaba Cloud SLB component uses the open source software LVS+keeplived to achieve Layer 4 load balancing.

Layer 7 uses Taobao's Tengine to achieve load balancing on layer 7.

All load balancing uses cluster deployment, and real-time session synchronization between clusters eliminates server single points, improves redundancy, and ensures service stability. Multiple physical computer rooms are deployed in various regions to achieve disaster recovery in the same city.

The overall design of SLB makes its availability as high as 99.99%.

It can be flexibly expanded according to the application load, and external services can not be interrupted even if any SLB fails or traffic fluctuates.

What is Multi-AZ?

The availability zone of a cloud product refers to a set of independent infrastructure. The infrastructure (network, power, air conditioning, etc.) between different availability zones is independent of each other. That is, an infrastructure failure in one availability zone does not affect the other availability zone.

In order to provide more stable and reliable load balancing services to users, Alibaba Cloud Load Balancing has deployed multiple availability zones in each region to achieve cross-machine rooms in the same region.

When the computer room in the primary availability zone fails or is unavailable, load balancing still has the ability to switch to another computer room in the backup availability zone and restore services in a very short period of time (about 30 seconds); when the primary availability zone recovers, load balancing also has the ability to It will automatically switch to the computer room in the main availability zone to provide services.

Regarding the load balancing active and standby availability zones, please note:

SLB supports mounting backend ECS across availability zones, that is, as long as the ECS and SLB instances are in the same region. SLB can distribute traffic to ECS in different availability zones at the same time.
Under normal circumstances, the SLB instance in the standby zone is in standby state. You cannot manually switch the active and backup working status of an SLB instance. Only when Alibaba Cloud detects that the entire availability zone is unavailable, such as a power outage in the entire computer room or an interruption in the optical cable at the exit of the computer room, will the load balancer switch to the standby availability zone. If an instance fails, it will be switched to the standby zone.
SLB and ECS are different clusters. When the SLB in Availability Zone A is unavailable, ECS does not necessarily become unavailable. Therefore, if the SLB primary and secondary switches are switched only due to SLB cluster failure, the SLB in the standby Availability Zone can still distribute traffic to ECS in different Availability Zones. However, when all clusters in the entire availability zone are powered off or optical cables are interrupted, all services in the availability zone (including but not limited to SLB, ECS, etc.) will not work properly.

List of active and standby availability zones

The following table lists the active and standby availability zones in each region of Alibaba Cloud. You can also query the available active and standby availability zones through the DescribeZones interface.

The following table lists the active and backup availability zones in each region. You can also query the available active and backup availability zones through the DescribeZones interface.

High-availability architecture with four nines—same-city disaster recovery

For medium and large users who want their business systems to have intra-city disaster recovery capabilities, they can consider making a complete set of backups of the original application architecture between different availability zones in the same city.

If there is a fault such as a power failure in the IDC computer room or a fire, the front-end can switch DNS to restore business in time.

As shown below:

High-availability architecture with five nines—off-site disaster recovery

For some large enterprises that require both intra-city and remote disaster recovery in terms of business security, service availability and data reliability, this disaster recovery architecture can be used to solve single computer room failures and respond to earthquakes. Wait for catastrophic failure.

Alibaba Cloud's high-speed channels can be used for private network communication between different regions, ensuring real-time data synchronization between databases and minimizing data transmission delays.

When a fault occurs, the front-end DNS can be used to implement second-level switching and restore services in a timely manner.

As shown below:

Virtual private network (VPC) to achieve interoperability in different regions

When ECSs are located in different regions, how to achieve intranet interoperability between them?

First, intranet interconnection between ECSs can be achieved through a virtual private network (VPC).

VPC is a highly customizable network isolation environment provided by Alibaba Cloud. Users can create their own private network within the VPC and connect ECS and RDS instances in different regions through the VPC intranet.

Secondly, users can achieve intranet interoperability between regions through inter-VPC peering connections. By creating peering connections, VPCs in different regions can communicate directly, allowing ECS and RDS instances in different regions to access each other through the intranet. Peer-to-peer connections provide low-latency, high-bandwidth network communications, ensuring data security and speed.

In addition to VPC peering connections, users can also choose to use Alibaba Cloud's intranet dedicated line access (VBR) service. Through VBR, users can build their own dedicated lines to connect private networks in different regions to achieve intranet interoperability between ECS and RDS. This method is suitable for scenarios that require large amounts of data transmission or have higher requirements for network reliability.

In addition, in order to enhance the security of intranet interconnection, users can use security groups and access control lists (ACLs) for network access control.

Security groups can define inbound and outbound rules to control access to ECS and RDS; ACLs can finely control traffic between subnets and VPCs based on source and destination IP addresses, protocols, port numbers and other information.

In general, when ECS and RDS are located in different regions, intranet interoperability can be achieved through VPC peering connections or intranet dedicated line access. These methods all provide a safe and reliable cloud network environment and meet users' needs for intranet communication in a distributed architecture.

Be aware of cross-region delays

The main challenge facing remote multi-activity is network delay. From Beijing to Shanghai, it is 1,468 kilometers. Even at the speed of light, a round trip requires close to 10 ms. During the actual test, it was found that the network delay from Shanghai to Beijing is generally 30 ms.

Reference Case: Introduction to High Availability of Alibaba Cloud Relational Database RDS

Alibaba Cloud Relational Database (RDS for short): It is a stable, reliable, elastically scalable online database service.

RDS adopts an active-standby architecture by default (the standby instance is normally invisible to users). The two instances are located on different servers and automatically synchronize data. When the primary instance becomes unavailable, the system automatically switches the database connection to the standby instance. The switching takes minutes and does not require manual intervention. It is all completed automatically by the system and no changes are required to the application system. This architecture is sufficient to meet the high availability requirements of 90% of users.

As shown below:

If users have higher requirements for system availability, they hope to achieve disaster recovery in the computer room.

Alibaba Cloud RDS can choose to purchase multi-availability zone RDS. Multi-availability zone is a physical area composed of multiple single availability zones in the same region at the single-availability zone level.

Compared with single-AZ RDS instances, multi-AZ RDS instances can withstand higher levels of disasters, as shown in the following figure:

RDS intra-city disaster recovery architecture

Figure 4: RDS intra-city disaster recovery architecture

In addition to intra-city disaster recovery, for users with strong demand for data reliability, such as financial business scenarios with regulatory requirements, RDS provides off-site disaster recovery instances to help users improve data reliability.

RDS uses Data Transfer Service (DTS) to achieve real-time synchronization between the main instance and the remote disaster recovery instance.

Both the master instance and the disaster recovery instance build an active-standby high-availability architecture. When a sudden natural disaster occurs in the area where the master instance is located, and neither the master node (Master) nor the backup node (Slave) can connect, the remote disaster recovery instance can be switched. As the master instance, after modifying the database link address on the application side, the application's business access can be quickly restored.

As shown below:

Figure: RDS remote disaster recovery architecture

Remote multi-active data consistency: DTS component

In the multi-active remote architecture, in order to support the flexible switching of business traffic between regions, the problem of data synchronization between various business centers must be solved.

Alibaba Cloud Data Transfer Service DTS supports two-way synchronization between RDS instances to achieve data synchronization between business centers and ensure global consistency of data, thereby achieving rapid replication of the multi-active technology architecture in remote locations.

Since 2013, the data transmission service DTS has steadily supported the underlying global data synchronization of Alibaba's off-site Duohuo (three business centers) for four consecutive years.

Since providing services to users on Alibaba Cloud in 2014, DTS has provided reliable and stable data streaming services for tens of thousands of users.

DTS supports data synchronization between data layers in the remote multi-active architecture to achieve global data consistency. The following is a simple remote multi-active business architecture diagram:

As shown in the figure above, the business divides traffic into various business centers (also called units) according to a certain dimension.

The selection of segmentation dimensions should follow the following principles:

(1) After splitting, it is necessary to implement single-point writing of the business. For example, according to the segmentation of members, the access of the same member can only be written in a single point in a certain business center.

(2) The split dimension should try to ensure that the business is closed within the unit, that is, all business requests can be completed within the unit to reduce cross-regional access calls.

For services with a wide distribution of users, the deployment area of the service center can be selected according to the distribution of users.

For example, for international business, you can choose multiple locations such as China, Europe, and North America to deploy business centers. Business requests from users near the area will directly land in the nearby area to minimize user access delays and effectively improve user experience.

When the traffic is divided into each unit, data will be written to the data layer of each unit. Bidirectional data synchronization of the data layer is performed through DTS to achieve global data consistency. When a business center (unit) fails, the traffic segmentation rules can be modified to switch the traffic to other units in seconds, thus effectively ensuring the continuous availability of the business and perfectly avoiding the economic losses caused by the failure and the impact on the company's brand. Impact.

Say it at the end

High availability related interview questions are very common interview questions. If everyone can answer the above content fluently and thoroughly, the interviewer will basically be shocked and attracted by you.

In the end, the interviewer loved it so much that he "can't help himself and his mouth watered" . The offer is coming.

During the learning process, if you have any questions, you can come and talk to Nien, a 40-year-old architect.

references

https://zhuanlan.zhihu.com/p/549472160

https://zhuanlan.zhihu.com/p/96917394