author
Kong Lingzhen, Chief Architect of Douyu, is fully responsible for the planning and construction of Douyu's total station technical architecture system. He has more than 10 years of experience in the architecture of medium and large Internet products, and is good at architecture and solution design in high concurrency and high availability scenarios.
Yu Jing, Douyu's technical support operation and maintenance expert, is responsible for Douyu's high-availability infrastructure construction. He is good at technical fields such as registration center and monitoring system.
Tang Cong, senior engineer of Tencent Cloud, author of the Geek Time column "etcd Practical Course", active etcd contributor, mainly responsible for product development and design of Tencent Cloud's large-scale k8s/etcd platform, stateful service containerization, and offline co-location .
Chen Peng, the product architect of Tencent Cloud Container Service, has been focusing on the cloud native field for many years, and has helped a large number of users to transform and implement cloud native containerization. He has rich first-line practical experience and has published a large number of cloud native technical articles.
Business Background and Pain Points
As an industry-leading game live broadcast platform, Douyu Live provides hundreds of millions of Internet users with high-quality live game viewing, interaction and entertainment services every day.
With the popularity of the live broadcast market in recent years, Douyu live broadcast platform, as an Internet company with excellent reputation and experience in the industry, has also experienced a blowout growth in the number of users. The stability and technical challenges brought by the massive users to the platform are also becoming more and more intense. The old architecture of Douyu is shown in the figure below. Whether it is business support or architecture design, there are certain risks and hidden dangers.
Douyu old architecture
Figure 1 Douyu old architecture
In order to bring a better usability experience to users, Douyu urgently needs to solve the problem of a single data center and upgrade the old architecture from a single data center to multiple data centers.
Multiple data center challenges
In the process of realizing single-active upgrade to multi-active, in order to ensure trouble-free migration and upgrade, we face a series of challenges, such as:
How do stateful services such as etcd, zookeeper, etc. synchronize with multiple data centers? Applications have a complex tree or mesh of dependencies on each other, where should I start migrating? What dimension should be used to divide the boundary of the target, and how to avoid the business being welded together, resulting in a situation where there is no way to start? If there is a problem after the migration, how to recover quickly without implicated in the business that has been successfully migrated? Due to the process of upgrading from single-active to multi-active, there are many systems involved. This article will be the first in a series of multi-active transformation of Douyu Live. It only focuses on the registration center module, so we will first introduce to you the etcd and etcd behind the registration center. zookeeper.
The role of zk/etcd
Dubbo solves the problem of service registration and discovery under large-scale clusters through the registry. The following is the architecture diagram of the registry:
dubbo supports the zookeeper registry by default. Although the new version also has etcd implementation, there is no precedent for this implementation to be put into large-scale production, and it is rare for the Java technology stack to use etcd as the registry.
When zookeeper is used as the dubbo registration center, its registration relationship is a tree structure, and the detailed structure is shown in the following figure:
Because zookeeper stores data based on a file system-like tree structure, etcd uses key-value pairs for storage, the difference between the two will bring greater difficulties to the synchronization of registration relationships.
In addition, if you migrate from zookeeper to etcd, during the entire migration process: the existing online services cannot be damaged, let alone stopped; if the migration fails, you can also fall back to zookeeper.
New architecture for dual-active and multi-active in the same city
In order to achieve multi-active, we successfully solved the above challenges through technical means and operation and maintenance concepts such as cross-data center synchronization services, service dependency sorting and boundary division, and controllable changes, and designed the following new architecture to achieve multi-active ,As shown below:
Figure 2 Douyu's new architecture
Under the new architecture, traffic can be scheduled in a fine-grained manner by domain name or even URL, and the RPC layer also has the ability to automatically call nearby. The partial architecture diagram of the registry is as follows:
Figure Sandouyu registration center old architecture
Registration center multi-activity plan selection and goals
During the multi-active transformation of the registration center, we are faced with multiple solutions, as shown in the following table:
Due to historical reasons, we have two sets of registries, zookeeper (hereinafter referred to as zk) and etcd, and we have four technology stacks, namely Java, Go, C++, and PHP. Therefore, there are still some shortcomings in the field of registries. etcd to solve the pain points and achieve the following goals:
Reduce maintenance costs: Previously, two sets of registries, zk+etcd, were required to be operated and maintained. What is more difficult is that zk+etcd also needs to be adapted to the multi-active solution, which doubles the R&D cost of the multi-active registry. Since etcd is a part of k8s, operation and maintenance of etcd is inevitable. This is the first reason to choose etcd.
Embrace a more prosperous ecosystem: etcd has cloud-native hosting solutions, some manufacturers manage 10K node-level k8s clusters through etcd, etcd also comes with various peripheral tools such as proxy, cache, mirror, etcd, and the java side dubbo also supports etcd as registration In the center, etcd has a better development prospect than zk, which is the second reason for choosing etcd.
Enhanced cross-language capability: etcd can communicate based on http or grpc protocol, and supports long polling, with strong cross-language capability. However, zk needs to introduce a dedicated client. Except for the java client, other language clients are not yet mature. And we have 4 R&D languages including JAVA, Go, C++, PHP, etc. This is the third reason to choose etcd.
Based on the above reasons, we chose the fourth scheme, and the four new architectures of the scheme are shown in the following figure:
Figure Sidouyu registration center new structure
Difficulties and challenges in the registration center
In order to realize the new registration center and achieve our desired design goals, the registration center faces the following difficulties and challenges in the transformation process:
How to solve the multi-data center synchronization problem of zk? In particular, the zookeeper watch mechanism is unreliable, and there may be a problem of missing watch events? (Correctness) How to solve etcd's multi-datacenter synchronization problem? From the following solution selection, we can see that the community currently does not have any mature solutions available for production environments. (Correctness) How to solve cross-datacenter read performance issues? (Performance) How to solve the problem of service stability across data centers? What should I do if the network link, such as an intranet dedicated line, is interrupted? In terms of synchronization service design, will the etcd/zk synchronization service enter the full synchronization logic with extremely slow performance, whether the synchronization service itself has high availability, etc.? In disaster recovery testing, how should we design test case verification? In terms of operation and maintenance, how can we quickly find hidden dangers, eliminate potential failures, and build a visual and flexible multi-activity operation and maintenance system? (stability, operability)
Analysis on the Difficulties of Multiple Activities in the Registration Center
How to ensure the interoperability of new and old services during the migration process?
Develop zk2etcd
Many of our java-developed businesses use the dubbo framework for service governance, and the registry is zookeeper. We hope that all the businesses developed by java and go will use etcd as the registry, which will also pave the way for the possibility of cross-language calls.
Due to the large number of businesses, the transformation and migration cycle will be very long, which is expected to last 1~2 years. During this process, we need to synchronize the registration data in zookeeper to etcd, and synchronize in real time, and ensure data consistency and high availability. At present, there is no tool to meet our needs on the market, so we cooperated with the Tencent Cloud TKE team to develop a zk2etcd to synchronize the zookeeper data to etcd, and it has been open sourced. We will introduce the overall solution in detail.
How to implement etcd remote disaster recovery?
Through the zk2etcd synchronization service, we successfully solved the zookeeper data migration problem, so that the registry data of new and old businesses are stored in etcd.
Therefore, the importance of etcd is self-evident, its availability determines our overall availability, and the current deployment architecture of Douyu Live relies heavily on a core computer room. Once the core computer room fails, it will lead to overall unavailability. Therefore, the next pain point of Douyu Live is to improve the availability of etcd, hoping to realize etcd's cross-city and remote disaster recovery capabilities.
Douyu Live's ideal etcd cross-city synchronization service should have the following features:
After etcd is deployed across cities for disaster recovery, the read and write performance does not decrease significantly, which can meet the basic demands of business scenarios. The synchronization components reach the level available in the production environment, with complete consistency detection, logs, metrics monitoring, etc. Businesses that do not require strong data consistency can access etcd cluster services in the same region nearby, and businesses that demand strong consistency can access the main etcd cluster. After the primary cluster fails, business operation and maintenance can quickly promote the standby cluster to the primary cluster based on consistency monitoring. So what are the options? What are the advantages and disadvantages of each program? Finally, the following options were evaluated:
Single-cluster multi-site deployment solution
etcd community make-mirror solution etcd community learner solution Tencent cloud etcd-syncer solution single-cluster multi-location deployment solution The single-cluster multi-location deployment solution diagram is as follows:
In this scheme, etcd Leader node replicates data to Follower nodes in various regions through the Raft protocol.
The advantages of this scheme are as follows:
After each area network is interconnected, the deployment is simple and no additional components are required to operate and maintain
Strong and consistent data synchronization across cities, in a 3-node deployment scenario, any city failure can be tolerated without any data loss
After introducing its advantages, let's look at its disadvantages, as follows:
In the scenario of 3-node deployment, any write request requires at least two nodes to respond and confirm, and different nodes are deployed in various places, the ping delay will increase from a few milliseconds to about 30ms (Shenzhen-Shanghai), which will lead to a sharp drop in write performance. .
The default read request of etcd is linear read. When the follower node receives the read request initiated by the client, it also needs to obtain relevant information from the leader node, and confirm that the local data catches up with the leader before returning the data to the client to avoid reading old data. In this process, the etcd read delay will increase and the throughput will decrease.
The quality of networks deployed across cities is also prone to fluctuate, resulting in jitter in service quality.
When a client accesses the etcd cluster configuration, in order to prevent a single point of failure, multiple etcd nodes must be configured, which in turn may cause the client to access the remote etcd nodes, resulting in increased service request delays.
etcd community make-mirror solution
After introducing the single-cluster multi-location deployment solution, let's take a look at the make-mirror solution provided by the etcd community. Its schematic diagram is as follows:
In this solution, we deploy a set of independent etcd clusters in different cities, and realize cross-city data replication through the make-mirror tool provided by the etcd community.
The principle of the make-mirror tool is as follows:
After specifying the prefix for data synchronization, traverse all the data under the prefix from the main cluster through the etcd Range read interface and write it to the destination etcd. (full synchronization) Then specify the "version number" returned by the read request through the etcd Watch interface, and monitor all change events after this version number. After make-mirror receives the key-value change event pushed by the main etcd cluster, it writes data to the hot standby cluster through the txn transaction interface. (Incremental synchronization) The advantages of this scheme are as follows:
The main etcd cluster has high read and write performance, and is generally not affected by cross-regional network delays and network quality fluctuations. If the business can tolerate short-term inconsistencies, you can visit the nearest etcd cluster. If the business requirements are strong and consistent, you can access the main etcd cluster through an intranet dedicated line. The etcd cluster does not depend on the high version of etcd After introducing its advantages, let's look at its disadvantages, as follows:
When the write request is large, the standby cluster may have a certain amount of data behind, and dirty data may be read. After the make-mirror synchronization link provided by the community is interrupted, it will enter the full synchronization mode again after exiting and restarting. The performance is poor and cannot meet the requirements of the production environment. The make-mirror tool that comes with the community lacks a series of features such as leader election, data consistency detection, logs, and metrics, and is not usable in production environments. Synchronization of non-key-value data, such as auth authentication related data, lease data, etc., is not supported.
etcd community learner scheme
After introducing the make-mirror scheme of the etcd community, let's take a look at the learner scheme provided by the etcd community. Its schematic diagram is as follows:
Its core principle is as follows:
The etcd raft algorithm library has supported the learner node since 2017. For details, please refer to pr 8751. The etcd community officially supports the Learner node in version 3.4 launched in August 2019. It joins the cluster as a non-voting member node, does not participate in cluster elections and other voting, and only performs data replication. After the Leader receives the write request, it synchronizes the logs to the Follower and Learner nodes, and uses a data structure called Progress in memory to maintain the log synchronization progress information of the Follower and Learner nodes. When the gap between the data of the learner node and the data of the leader is small, it can be promoted as a member node that can vote to join the cluster. The advantages of this scheme are as follows:
After the interconnection of various regional networks, the deployment is simple, just add a Learner node to the etcd cluster, no need to operate and maintain additional components. The Learner node can synchronize any type of data, such as key-value, auth authentication data, and lease data. After introducing its advantages Then, let's look at its shortcomings, as follows:
The learner node only allows serial reading, that is, if the business reads the nearest data, it will read the old data. Depends on the higher version of etcd, etcd 3.4 and above only supports the Learner feature, and only allows one Learner node. After the main cluster fails completely, the Learner node cannot be quickly promoted to a writable independent etcd cluster. After introducing several existing solutions, we found that none of them can meet the demands of the business production environment, so we have completed the implementation of etcd synchronization services available in the production environment by ourselves, which will be introduced in detail in the chapter on the implementation of the overall solution.
How to ensure the stability and operability of etcd and zk synchronization services?
In order to ensure the stability of etcd and zk synchronization services, 5 types of common faults are simulated, and the self-healing ability of the service under these typical fault scenarios is tested. The detailed test plan is as follows.
failure scenario
Redis flash (depending on zk2etcd service), for example: redis version upgrade, non-smooth expansion.
zk2etcd is offline, for example: OOM, container eviction, host failure.
etcd2etcd offline, for example: OOM, container eviction, host failure
Network outages, such as OOM, container eviction, host failure.
In a weak network environment, for example, the public network is temporarily replaced after the leased line is disconnected.
There are various actual triggering reasons for the above five scenarios, and only one situation needs to be simulated.
Exercise program
Redis flashes: Simulate that redis is unreachable by changing the host, and the automatic correction stops at this time; after the simulated redis is restored, the automatic correction also automatically resumes.
zk2etcd offline: simulate zk2etcd hang by killing the container node, k8s will automatically pull up within 15 seconds, after the pull is completed, the synchronization is normal and the data is consistent.
etcd2etcd offline: simulate zk2etcd hang by killing the container node, k8s will automatically pull up within 15 seconds, after the pull is completed, the synchronization is normal and the data is consistent.
Network flashing: By changing the host to simulate zk and etcd unreachable, the synchronization is interrupted at this time, then remove the host to simulate network recovery, after recovery, the synchronization is normal and the data is consistent.
Weak network environment: The weak network environment is simulated by switching to the public network. After switching to the public network, the synchronization efficiency is reduced by less than 4 times, and one full synchronization can still be completed within one minute.
In addition, for operability issues, both etcd and zk synchronization services provide detailed metrics and logs. We have configured visual observation views and alarm policies for each core scenario and abnormal scenario.
The overall plan is implemented
Overall structure
The multi-active architecture diagram of etcd cluster is as follows:
illustrate
Black solid line: Dedicated line access under normal conditions
Black dotted line: Access via public network
Red solid line: dedicated line access after the etcd cluster is switched between active and standby
Red dotted line: public network access after etcd cluster switchover
The etcd2etcd/zk2etcd data synchronization service diagram is as follows:
zk synchronization service engineering practice
The storage structure of zookeeper and etcd is inconsistent, which increases the difficulty of synchronization. zookeeper storage is a tree structure, while etcd v3 is a flat structure. Zookeeper cannot list all keys according to prefix like etcd; etcd cannot query child nodes in a directory through list chilren like zookeeper, which also increases the difficulty of synchronization.
How to perceive data changes in zookeeper? Zookeeper's watch is not like etcd, which can simply perceive the addition of any key. It needs to recursively watch all nodes. After receiving the ChildrenChanged event, get all the child nodes under the corresponding node of the event, and then carry out the data in etcd. By comparison, the newly added data can be obtained and put into etcd synchronously. Similarly, you can use the recursive method to watch the deletion events of all nodes, and delete the data in etcd synchronously.
In addition, zookeeper's watch has a congenital defect. The watch is a one-time use, so it must be re-watched every time an event is received. In theory, it is possible to lose events between two watches, mainly in the same key multiple times in a row. changes may occur. If a missing event occurs, the data consistency will be destroyed. We have introduced the ability of automatic diff and correction, that is, to calculate the difference between the data in zookeeper and etcd, each time will go through two rounds of diff calculation, because in the case of frequent data changes , a round of diff calculation often has some "pseudo-differences" that are not caused by strong consistency synchronization, and these differences will be automatically fixed when the diff calculation results are obtained.
How to resolve coexistence with etcd2etcd? When under the same path, there is data written by etcd2etcd synchronously and data written by zk2etcd. In the automatic correction logic of zk2etcd, the difference will be calculated and corrected, but we do not want to delete the data written by etcd2etcd by mistake. data. We solved this problem by introducing redis for zk2etcd to store state. When zk2etcd writes or deletes data synchronously to etcd, it also records and deletes in redis synchronously:
Then, when zk2etcd automatically corrects and calculates the difference, it only considers the data written by this tool to avoid accidentally deleting the data written by other synchronization tools.
etcd2etcd engineering practice
In order to solve the etcd synchronization problem, we have investigated the following two solutions. Next, we will introduce its principle in detail:
mirror-plus version of etcd-syncer
First of all, let's introduce the mirror-plus solution of etcd-syncer. As the name suggests, it is an enhanced version of make-mirror in the etcd community. In order to solve various defects of make-mirror, it implements the following features and advantages:
Supports multiple synchronization modes, full synchronization, and resumable transmission at breakpoints, no longer worry about the jitter of the private line and public network network, high availability, and the instance responsible for the replication of the same data path supports multi-copy deployment. After one copy fails, the other copies will be in 5 seconds After obtaining the lock, based on the progress of the previous instance synchronization, perform fast recovery Support consistency check (full data check, snapshot check) Support multi-instance concurrent replication to improve performance (different instances are responsible for different paths), it is recommended to configure multi-instance production environment , each instance is responsible for good operation and maintenance capabilities of different paths, based on k8s deployment one-click deployment, rich metrics, logs, and complete e2e test cases covering core scenarios (http/https scenarios, service interruption, network exception, etc.) Then it What's the downside? Because its core principle still relies on the mvcc+watch feature of etcd, the data cannot guarantee strong consistency and only synchronize key-value data.
Breakpoint resuming depends on the retention time of the mvcc historical version, and it is best that the business can save historical data for at least 1 hour. When the write request is large, the standby cluster may have a certain amount of data behind, and dirty data may be read. Synchronization of non-key-value data, such as auth authentication related data, lease data, etc., is not supported.
Raft version of etcd-syncer
In order to solve all types of data synchronization problems and eliminate the dependence on etcd mvcc historical data, Tencent Cloud can also provide the raft version of etcd-syncer, a synchronization solution based on Raft logs.
Its deployment diagram is shown below. The etcd-syncer synchronization service joins the main etcd cluster as a learner-like node.
The main etcd cluster leader synchronizes the Raft log data to etcd-syncer through MsgApp/Snapshot and other messages, etcd-syncer parses the Raft log, and applies the Txn/Delete/Auth requests corresponding to the Raft log entries to the destination etcd cluster.
It has the following advantages:
Has all the features and benefits of the mirror-plus version of etcd-syncer without relying on etcd mvcc historical data.
Based on the Raft log synchronization data underlying etcd, various types of data such as key-value, auth, lease, etc. can be synchronized.
Does not depend on a higher version of etcd.
Complete disaster recovery test
grpc-proxy
This program introduces the grpc-proxy proxy service, which is also the first time to use it. In order to understand the performance of this proxy service, we used the benchmark that comes with etcd to test the read and write, and also wrote a small tool to do the watch test. The following is part of the test content.
write test
Direct access to the load balancing entry of the etcd service
The situation of accessing etcd service through grpc-proxy proxy
The grpc-proxy proxy can normally write and write keys when the endpoints are configured to go through a dedicated line or public network. Under the condition that the total number of keys is fixed, the greater the number of connections and clients, the lower the total time consumption. The greater the total number of keys, the single The average time (Average) of each write will increase, but it is still in milliseconds. When the total number of keys written at one time is 100,000, an error of too many requests will appear when directly connected to etcdserver, but grpc-proxy has no public network. The performance of the private line is lower than that of the private line. The average time spent on the grpc-proxy proxy is increased compared to the direct connection, but it meets the requirements for reading the test.
Direct access to the load balancing entry of the etcd service
The situation of accessing etcd service through grpc-proxy proxy
The grpc-proxy proxy can read normally when the endpoints are configured on a dedicated line or on the public network
The average time taken to use the grpc-proxy proxy has increased compared to the direct connection, but it is within an acceptable range
watch test
According to an etcdwatcher service written by ourselves, we can perform a watch test on grpc-proxy: the total number of watchers, update frequency, and test time can be set, and a briefing report can be printed at the end
./etcdwatch -num=100 -span=500 -duration=10 -endpoint=http://grpc-proxy-addr:23791 test done total 100 task 0 task failed current revision is 631490 least revision is 631490 0 task is not synced
Parameter Description:
num number of tasks
span update interval, in milliseconds
duration total test time, in seconds
current revision: represents the revision written
least revision: Indicates the revision with the slowest synchronization among num tasks
If failed is 0, it means normal; if task not sync occurs, it means watch and put are out of sync
From the above test results: the number of failed is 0, and the watch test is normal
zk2etcd
We are using version 1.2.5, which is deployed through the deployment method of k8s
Simulate zk server disconnection
Scenario resolves addresses by injecting errors into hosts
During the phenomenon period, the error log monitoring indicator of the zk disconnection was not found. No abnormality was found. After that, the restart was performed, and the fixed operand did not increase. The bug of the key. After restarting the zk2etcd service instance, the phenomenon of the fixed operation convex increase may be observed)
Simulate redis disconnection
Simulated operation 09:56:49 Inject redis into hosts Error resolution address 10:07:34 Restore redis 10:16:00 Restart synchronization service pod (operation restart is to observe whether full sync is normal)
During the phenomenon period, the number of fixed operations did not increase, and other monitoring indicators found no obvious abnormality. After the instance restarted, there was no convex increase in the number of fixed operations.
Simulate etcd disconnection
Simulated operation 16:15:10 etcd server lost connection
16:30 Recovery
16:45 Restart pod
During the phenomenon period, the number of fixed operations did not increase, and other monitoring indicators did not find obvious abnormalities
After restarting, the number of fixed operations has increased (not sure if full sync did not take effect, or if there was just an update and repair after restarting)
Summarize
As long as the full sync mechanism works normally, after each abnormal scenario occurs, it can be recovered after the next full sync is triggered
The minimum time interval for recovery depends on the set full sync regular execution interval (default is 5m), and the business adjusts the parameters by itself to tolerate this interval time
In addition, in order to avoid the occurrence of an exception, the full sync mechanism runs regularly but fails to sense the situation. To be on the safe side, you can restart the zk2etcd service as soon as possible.
For the additional etcd public network test, the time-consuming operation of full sync completed and zk and etcd operations has a certain (second level) increase compared to the internal network.
etcd2etcd
The synchronization service of etcd2etcd, I use deployment double copy deployment
Multi-copy backup capability
It is expected that the standby node will take over the synchronization task after 5s after the worker node fails
Test solution etcd syncer dual instance deployment
Kill the running worker node for observation
Conclusion Whether it is incremental synchronization or full synchronization, the active-standby switch can work normally (it should be noted that when the active-standby switch occurs in the full synchronization, it will become incremental synchronization, which may lead to slower comparisons)
Resume transfer capability
Expect to resume synchronization from the breakpoint after failure recovery
In fact, in the first part, the standby node takes over the synchronization work after switching to the main node, and the fast_path becomes 1, which also proves the ability to resume the transmission from a breakpoint. We also add a few additional verification scenarios:
(a) Short-time failure
failure scenario
During the synchronization process from the central etcd cluster to the hot standby cluster, because the key of -etcd-syncer-meta- also exists in the central etcd cluster as the source, the synchronization service is triggered to report an error (the same key cannot be contained in the same txn). data discrepancy
Phenomenon
Add the synchronization service running parameters to filter -etcd-syncer-meta-, and then observe that after catching up with the data for a period of time, the final number of misses decreases to reach the same level
(b) Prolonged failure
failure scenario
Stop the deployment deployment of the sync service
Wait for the data difference between the etcd clusters on both sides, and a compact occurs before starting the synchronization service
Phenomenon
After data discrepancy occurs and compaction occurs, restart the synchronization service. The log is as follows: Full synchronization is triggered due to compacted occurrence.
Synchronization service monitoring indicators: (a) dst miss key decreased quickly; (b) src miss key increased and continued to decline
analyze
After the synchronization service is stopped, the number of keys in the source etcd has changed a lot, and the monitoring chart shows that there is a drop during the period, indicating that the key has been deleted.
A small problem is also exposed here. When the src miss key occurs, it cannot be automatically repaired at present, and manual access is required to clean up the redundant key.
- reset triggers full synchronization. When a major difference occurs in synchronization (for example, dst miss occurs) for emergency repair, configure the --reset-last-synced-rev parameter to delete the breakpoint resume information to trigger full synchronization to repair the difference
Phenomenon Due to some abnormality, a dst miss (the example of the yellow line in the figure) occurs synchronously. To fix, run the new instance with the --reset-last-synced-rev parameter added
analyze
slow_path is 1, indicating that full synchronization is triggered (the green line example in the figure)
The dst miss value of the green line instance has not increased, indicating that the consistency has been reached
- Network failure, the dedicated line between the two etcd clusters is interrupted
Incremental synchronization
Full synchronization
Test program
When the private line is interrupted and the public network is switched, the etcd cluster access address in the running parameters needs to be modified, that is, a restart will occur (the restart scenario test has been covered before, and will not be repeated here)
Summarize
The etcd-syncer synchronization service has a good master-standby mechanism, which can switch in a timely and effective manner
The performance of resuming transmission after a short-term failure is as expected; for a long-term failure, when the complex situation of compact occurs at the same time, and the src miss occurs after the synchronization is restored, manual access may be required.
By configuring the --reset-last-synced-rev parameter, the exception repair of src miss has a better effect
about Us
For more cases and knowledge about cloud native, you can pay attention to the public account of the same name [Tencent Cloud Native]~
Welfare: Reply to [Manual] in the background of the official account, you can get "Tencent Cloud Native Roadmap Manual" & "Tencent Cloud Native Best Practice"~