- ADVANCED concurrent principle
⽆ state
⽆ state to apply it to facilitate expansion Horizontal
stateful configuration ⽆ Center Weighted achieved by state configuration
practice: Disconf, Yaconf, Zookpeer, Consul, ConfD, Diamond,
Xdiamond other
split
system dimensions: The system function, Business splitting, such as shopping, settlement, orders, etc.
Functional dimensions: fine-grained splitting of system functions.
Read and write dimensions: split according to the characteristics of read and write ratios; read more, consider multi-level cache; write more, allow test
considered sub-library sub-table
AOP dimensions: The access patterns according ⾏ split into AOP, such as commodity ⻚ ⽐ can be divided into
the CDN, ⻚ ⾯ rendering system, the CDN is ⼀ a system AOP
modules dimensions: the overall division of the Web code structure, Service, DAO
service-oriented
service-oriented evolution: In-process service-stand-alone remote service-cluster manual registration service-automatic registration and
discovery service-service grouping, isolation, routing-service governance
consider service grouping, isolation, current limiting, Blacklist, timeout, retry mechanism, routing, fault
compensation, etc.
Practice: Utilize Nginx, HaProxy, LVS, etc. to achieve load balancing, ZooKeeper, Consul,
etc. to achieve automatic registration and discovery of services.
Message queue target
: Service decoupling ( One-to-many consumption), asynchronous processing, traffic peak-shaving buffering, etc.
Mass flow buffering: Sacrifice consistency to ensure final consistency (Case: inventory deduction, deduction is now made in
Redis, record deduction log, pass The background process will apply the deduction log to the DB)
Data proofreading: Solve the problem of message loss under the asynchronous message mechanism.
Data heterogeneity.
Data heterogeneity: Receive data changes through the message queue mechanism and store original
data. Closed data loop: Shield multiple data sources and store data heterogeneously to form a closed-loop
cache silver bullet
User layer:
DNS cache
Browser DNS cache
Operating system DNS cache
Local DNS service provider cache
DNS server cache
Client-side cache
Browser cache (Expires, Cache-Control, Last-Modified, Etag)* App
client cache (js/css/ image...)
Proxy layer:
CDN cache (generally built based on ATS, Varnish, Nginx, Squid, etc., edge node-
sub-level node-middle node-origin site) Interface
layer:
Nginx as an example:
Proxy_cache: proxy cache, you can Stored to /dev/shm or SSD
FastCGI Cache
Nginx+Lua+Redis: Business data cache
PHP as an example:
Opcache: Opcodes application
layer for caching PHP :
Static
business data cache (Redis/Memcached/local files, etc.)
Message queue
Data layer:
NoSQL: Redis, Memcache, SSDB, etc.
MySQL: Innodb/MyISAM, etc. Query Cache, Key Cache, Innodb
Buffer Size, etc.
System layer:
CPU: L1/L2/L3 Cache/NUMA
memory
disk: disk itself cache, dirty_ratio/dirty_background_ratio, array card Concurrency of own
cache - High availability principle
Downgrade
Downgrade switch centralized management: push switch configuration information to each
application, downgradable multi-level read service: such as service call downgraded to read-only local cache
switch front-end: such as Nginx+lua (OpenResty) Configure a downgrade strategy to divert traffic; based on
this, a gray-scale strategy can be made.
Business downgrade: high concurrency, ensuring core functions, secondary functions can be changed from synchronous to asynchronous strategies or
blocking functions
Flow limit
: prevent malicious requests*** or system beyond the peak
practice:
malicious traffic only request access to the Cache
penetrate the back-end to apply it flow Use the Nginx limit processing
malicious IP Using Nginx Deny policy or refuse iptables
cut traffic
to do this: the shield machine failure
practice:
the DNS: change Domain name resolution, such as DNSPOD can add a backup IP, when a normal IP fails
, it will automatically switch to the backup address; effective practice is slower
HttpDNS: In order to bypass the precise traffic scheduling implemented by the operator LocalDNS
LVS/HaProxy/Nginx : Remove the faulty node
and roll back. When the
release version fails, you can quickly roll back to the previous stable version at any time - Business design principles
anti redesign
idempotent design
process definition
state of the state machine
back system operation can be fed
back office systems approval of
documentation that annotation
backup - Conclusion It
is necessary to plan and design first, to have plans for existing problems and plans for the future; the
technical debts will be repaid sooner or later. - The difference between distributed and cluster:
Distributed refers to the distribution of different businesses in different places. Cluster refers to the clustering of several servers to
achieve the same business - Distributed transaction:
- Submission in the second stage:
a. Concept: Participants inform the coordinator of the success or failure of the operation, and the coordinator
decides whether each participant should submit the operation or not based on the feedback information of all participants.
b. Function: Mainly ensure the originality of distributed transactions; the first phase is the preparation phase, and the second
phase is the commit phase;
c. Disadvantages: not only must lock all the resources of the participants, but also lock the coordination The resource, the cost is
large. One sentence summary is: 2PC is very inefficient and unfriendly to high concurrency. - Three-phase commit:
a. Concept: The three-phase commit protocol introduces a timeout mechanism in both the coordinator and participants, and
splits the first phase of the two-phase commit protocol into two steps: inquiry, and then lock resources, Finally
, it is actually submitted. In this way, the three-phase submission has three stages: CanCommit, PreCommit, and
DoCommit.
b. Disadvantages: If after entering PreCommit, the Coordinator sends an abort request.
Assuming that only one Cohort receives and performs the abort operation,
other Cohorts that are unknown about the system status will choose to continue Commit according to the 3PC. At this time, the system status
Inconsistent. - Flexible transaction:
a. Concept: The so-called flexible transaction is a rigid transaction that is relatively forced to lock the table. Flow below:
If the transaction of server A is executed smoothly, then transaction A will be submitted first. If transaction B is also executed
smoothly, then transaction B will also be submitted, and the entire transaction will be completed. However, if YES transaction B failure
fails, the transaction rollback in B, then transaction A has been committed, it is necessary to perform it a compensation operation
for the transaction have been submitted for A Perform inverse operation of the operation, does not return to the The
status of transaction A before execution .
b. Disadvantages: The business is too intrusive and compensatory operations are required. It lacks universality and cannot be promoted on a large scale
. - The RabbitMQ implementation of the final message consistent solution:
a. Implementation: send confirmation + message persistence + consumer confirmation. - When to use distributed development:
a. Advantages:
i. Module decoupling: split modules, use interface communication, reduce the coupling between modules.
ii. Split projects, different teams are responsible for different subordinates Project: Split the project into several
sub-projects, and different teams are responsible for different sub-projects.
iii. Improve project scalability: When adding functions, you only need to add another sub-project and call it. The connection of
other systems is fine.
iv. Distributed deployment: Distributed deployment can be carried out flexibly.
v. Improve code reusability: Just like the service layer, if you don’t use the distributed rest service
architecture, it will be available in the mobile wap mall, WeChat Each end of the mall, pc, android, and ios needs
to write a service layer logic. The amount of development is large, and it is difficult to maintain and upgrade. At this time, a
distributed rest service method can be adopted and a service layer can be used publicly.
a. Disadvantages:
i. The interaction between systems
requires the use of remote communication, which increases the workload of access development; ii. There is a delay in network requests;
iii. Transaction processing is more troublesome and requires the use of distributed transactions. - cdn (Multi-activity in different places)
1. Multi-activity in different places: Multi-activity in different places refers to the business scenario where multiple sites distributed in different places provide services to the outside at the same time
. Multi-activity in different places is a type of high-availability architecture design. The main difference from traditional disaster recovery design is
"multi-activity", that is, all sites provide external services at the same time.
2. Two-
site disaster tolerance switchover scheme: Disaster tolerance is the most core part of multiple activities in different places. Take the architecture diagram of multiple activities in two cities as an example:
A
complete set of business systems are deployed in two cities (city 1 is located in South China 1 region, and city 2 is located in East China 1 region) .
Order business is classified according to "user_id"% 100. Under normal circumstances:
[00~49] All reading and writing are in the main database of the database instance in city 1.
[50～99] All the reading and writing of the sub-pages are in the main database of the database instance in city 2.
"Main database instance of city 1" and "Main database instance of city 2" establish two-way replication
of DTS .
When an abnormality occurs, a disaster recovery switchover is required. There are four possible scenarios:
the second and third abnormal situations are all handled by the second case, so whether it is all
APP Server anomalies, all database anomalies, or the entire city, Just follow the city-level
disaster recovery plan and directly switch the APP Server and database to another city.
3. Multiple cities and different places:
Multiple cities and different places, multiple jobs mode refers to the deployment of multiple jobs in different places among 3 or more cities. In this mode
, there are central nodes and unit nodes:
central node: refers to that the incremental data of the unit node needs to be synchronized to the central node in real time, and the
central node synchronizes all the distributed incremental data to other unit nodes.
Cell node: the node corresponding to the reading and writing of the branch. The node needs to synchronize the increment of the branch to the
central node and receive the incremental data of other partitions from the central node.
The figure below is a diagram of the multi-activity architecture in three cities. East China 1 is the central node, and South China 1 and North China 1 are
unit nodes.
9. How to deal with downtime in a distributed environment?
1. Dubbo: The server is down and zk is temporarily deleted;
2. Springcloud: Send one-hop detection every 30s to re-enter the lease. If the client cannot
renew the lease multiple times , it will be removed from the server registration center within 90s.
3. Apm monitoring:
10.