======= opening Tucao: the most recent period coincided with the Mid-Autumn Day double, but its heavy workload, the basic hard to ensure there is time to write articles =======
"Scalable service architecture framework and middleware" and "the principle of the distributed services architecture, design and actual combat" is tied to matching look, this marketing tools, clothes.
This book describes some of the conventional frame assembly used in a distributed system, such as distributed ID, a message queue buffer, the RPC frame, ES like. Much greater role in the book is a general introduction, knowledge of the point spread, the initial entry, the book posted the source code which is hard to seriously learn to read line by line. Want more in-depth study, more work needs to accumulate in the usual extensive project experience, or see more open source projects, so to summarize and extraction.
Each chapter introduces a component, extract some of their own find useful content, collate, and then to be understood. (Mainly forced himself written summary form the habit of looking at a lot of books, are always easy to forget that the effect is minimal)
Chapter 1 How to design a high-performance distributed generation is never repeated number is
1. Why not just adopt UUID?
Although the UUID to ensure uniqueness, but can not meet many other features business system needs, such as ordering a rough time, and the solution can be anti-manufacturability (that were so, ID is distributed according to the time needed to reflect the increasing features, and from the ID string to resolve the meaning of certain services), while the UUID long, representing a large space, poor performance.
2. Based on that database to achieve it?
I.e., to ensure uniqueness across databases ID adjustment step by the self-energizing field or database sequence, but this scheme strongly relies on a database.
Implementation, I can see: Reconstruction - design of distributed ID
3. The basic needs of distributed ID
(1) Globally Unique
Distributed system guarantees only a pessimistic strategy is to use a distributed lock or lock, but it would greatly reduce performance. Thus using the ordering time, and using the self-energizing sequence at a certain time unit to achieve globally unique.
(2) rough ordered
The biggest problem is the UUID disorder.
(3) the inverse solution
That can be seen from the ID string of the meaning of certain businesses, such as when generated, which business function modules with related.
(4) can be produced
Fa does not depend (if the system collapse), but also by certain rules to process the data manually.
(5) High Performance
Generating a new business, it is necessary to generate a new ID, you very high performance requirements. ID generated depends on network I / O and CPU performance, network I / O is generally not a bottleneck.
(6) High Availability
Fa device should meet the HA cluster, also has a retry mechanism. In extreme cases, but also a local fault-tolerant solution.
4. How to ensure the performance requirements?
Early in the project proposed performance requirements, performance tests to validate the project.
The acquired ID of the distributed several methods
(1) REST Method
Provide a HTTP interface to obtain
(2) RPC service method
Export service mode RPC services through Dubbo
(3) embedding method
The Fa service embedded into business projects, and provides local service within the JVM process
Chapter 2 Design and implementation of flexible extended frame message queue
1. Background
It applied to the message queue multiple asynchronous processing, clipping and high concurrent scenarios such decoupling between the system modules.
2. threading model
(1) synchronous threading model
The client uses a stream for each consumer threads, each responsible consumption messages from the queue Kafka, and in the same thread processing business.
(2) asynchronous threading model
The client uses a stream for each consumer threads, each responsible consumption messages from the queue Kafka, and deliver the message to the consumer to get the back end of the asynchronous thread pool, thread pool in asynchronous processing business.
The back-end business asynchronous thread pool can be divided into:
1> all consumers stream shared thread pool
This model can be created fewer objects thread pool, save a little memory
2> each stream exclusive thread pool
Asynchronous traffic using different thread pool to handle different message streams inside, isolated from each other, independently of each other, do not affect each other. For example, to distinguish between normal user message, the user message and VIP, which can be processed in a different business thread pool.
3. Exception Handling
For messages generated in the processing business exception, the current service processing in an upper layer captures the Throwable, special error recovery log records information about the error, according to a subsequent manual handling error recovery log an error message, or may be redone clean data. Listener systems are also contemplated, exception handling is achieved using pluggable listener mode exception handler, and the like.
4. graceful shutdown
By registering the JVM hook to exit a graceful shutdown.
========================= continue to look next ===================== =====
================================= See Part ============= =========================
Chapter 3 lightweight sub-library sub-table database architecture and framework
1. Definition of the sub-library sub-table
(1) Vertical Split
The business dimension, the original library (table) split into a plurality of libraries (Table), each library (Table) different from the original structure.
In this manner a single addition to the decomposition pressure of a single database table, and also for realizing separation of hot and cold.
1 "in Mysql, the more cold data query, update less, suitable for use MyISAM engine, and thermal data updated more frequently, suitable for use InnoDB storage engine.
2 "read how much cold write data from the library can be configured more queries to resolve a large number of pressure; thermal data can be used to build the structure of a plurality of main sub-library library part tables.
3 "for some special active data or hot data, you can also consider using Memcache, Redis cache and the like, and other accumulated to a certain amount and then update the database. For example, in Zambia the number of records microblogging point service, the number of points praise is in the cache memory for each additional 1000 points praise, just write data once.
(2) horizontal split
The slice (Sharding) algorithm, a database (table) split into a plurality of libraries (Table), each library (Table) still retains the original structure.
Segmentation dimensions are:
1 "Press hash slice
Advantage is that uniform data slice, pressure data of better dispersion, the dispersion drawback is the data for the query requirement needed for the polymerization process.
2 "time slice
Suitable for a significant time data characteristics.
2. The three kinds of sub-library sub-table of implementation
(1) The client fragment
Use sub-library sub-table database application layer slices logic operation directly, fragmentation rules need to be synchronized, each application layer are embedded logic implementing a slice operation (fragmentation rules) among a plurality of nodes in the same application. Jar generally in the form of packets.
(2) fragmentation agent
Increasing a proxy layer at the application and the database layer, the routing rules slice layer is disposed in the agent, the agent layer provides the external interface to a JDBC compliant to the application layer. Our company is using this framework: Mycat.
(3) support for distributed database transactions
The scalable features and packaging to achieve a distributed transaction distributed internal database implementation, transparent to the user. For example: OceanBase, TiDB and other products.
3. Distributed Transaction
There are three solutions: two-phase commit protocol, its utmost to ensure mode, transaction compensation mechanism.
Good practice and the nature of the buffer cache used in Chapter 4 of
To resolve mass by caching and message queue read and write requests to the pressure caused by the back-end database services.
1. False Sharing
When reading and writing different variables at the same time means that the same cache line in multiple threads, although there is no relationship between these variables, but still need to be synchronized between multiple threads, resulting in performance degradation.
2. JVM memory model object
1 "Java object has all the object header of 8 bytes, the first four bytes and a hash code to the lock state holding object, the first 3 bytes used to store the hash code, the last byte with to store the state of the lock, once the object is locked, the four bytes are taken out outside the object, and the link pointer; the remaining four bytes of memory used to refer to their class objects. The size of each object will be aligned to a multiple of 8 bytes, not 8-byte portions need to be filled.
2 "In order to ensure efficiency, Java compiler when compiling Java objects, Java objects will sort the field by field type. By filling any field between long integer variable, the variable focus isolated in different cache line, by reducing the pseudo synchronization, can greatly improve the efficiency of the multi-core CPU.
3 "lock-free queue Disruptor known to increase the efficiency by solving the problem of false sharing, after which by filling the variable sequence and the variable RingBuffer BatchEventProcessor the cursor, and align with the cache line size of 64 bytes, to solve the competition dummy The problem.
3. Common Distributed cache: Redis, Memcached, Tair
4. The penetration cache, the cache and cache concurrency avalanche
(1) penetrate cache
Refers to the use key does not exist in a lot of high concurrent queries, which can not lead to a cache hit, each request must penetrate to the back-end database systems query the database too much pressure, or even make the database service was crushed to death.
Normally cached null value, the same query request is received again, and if a cache hit is empty, it will return directly transmitted to the database does not penetrate, to avoid penetrating the cache.
But sometimes malicious attackers guess this scheme, each time using different parameters to a query, which is we need to filter the input parameters.
(2) the cache concurrency
Usually occurs at high concurrency scenarios, when a cache key expired, because the cache key access request is too large, multiple requests at the same time found the cache expires, so multiple requests simultaneously access the database to query the latest data, and will write cache, this will result in an increase in load applications and databases, reduce performance.
There are three ways to solve:
1 "Distributed Lock
2 "Local lock
3 "soft expired
Just do not use expired time caching services provided, but the business layer in the data storage expiration time information from business processes to determine whether expired and updated.
(3) cache avalanche
It refers to the cache server reboot or a large number of cache concentrated in one period of time failure, resulting in instantaneous load elevated pressure to the back-end database.
Typically use different data for different failure times.
5. Practice good cache design (good look)
(1) core and non-core business use different cache instance, be isolated physically.
(2) cache timeout setting is very important. To digital, desirably less than 100 milliseconds.
(3) all cache instances need to add monitoring, reliable monitoring needs to be done to slow queries, large object, memory usage.
(4) restricted by the specifications of each application key using a unique prefix.
(5) any cache key must set the cache expiration time and expiration time can not concentrate at some point.
Lower (6) Usually, the order is to read the cache, the database; write order is to the database, when the cache.
(7) When using the cache, there must be downgraded, especially for key business sectors. The common solution is to allow sufficient capacity in the database level.
(8) If a large number of key-value pairs to be saved, then use the database type string keys, each key and set the expiration time, please do not aggregate data in a hash key internal storage without boundaries.
==================================== continue to look next ========== =====================================
================================== ============ See Part I ============================
Chapter 5 Big Data tool of Elasticsearch
1. Lucene
Lucene based underlayer is achieved, based on the design principle inverted table, using a storage mode segments, when it is almost completely avoid the occurrence of read and write locks, greatly improved read and write performance.
2. Fragmentation
(1) When you delete a file, Lucene add a file .del in the index file, specifically for data storage id deleted.
(2) In order to improve write performance, Lucene and each new piece of data is not increased by one segment, instead of using delay tactics to write, whenever there is new data, it will be written into memory first, and then batch written to disk.
Lucene or Elasticsearch and so can not be called real-time search engine, can only be called a quasi-real-time search engine.
3. segment consolidation strategy
When too many indexes in the middle of, not only a serious drain on server resources, but also affect the performance of retrieval. It must be consolidated segment operating a regular basis.
The combined ideas Lucene segment: segments are grouped according to the size of the first segment, then the segments belonging to the same group are combined. When Lucene will reach a certain size in the size of the segment, or the amount of data inside the segment reaches a certain number of pieces, it will not be merged.
4. Lucene similarity score
The two most classic text similarity algorithm: an algorithm based on probability-based algorithms and vector space model (BM25)
5. 3C and cerebral fissure
(1) consensus (Consensus)
All nodes in a distributed system must reach a consensus on a given data or node status.
(2) concurrent (Concurrency)
Write requests are sent to the main time slice, while the form is sent in parallel to the backup slice, but the delivery time of these requests may be disordered. In this case, Elasticsearch with optimistic concurrency control to ensure that the new version of the data will not be overwritten the old version of the data.
(3) Consistency (Consistency)
Elasticsearch cluster to ensure consistency of approach is written before writing check how many slices can be written and, if the write condition is reached, then write, otherwise, Elasticsearch will wait for more fragments appear, by default One minute.
(4) Split Brain
In Elasticsearch cluster master node checks whether other nodes in the cluster by the ping command in an available state, while the non-master node will be checked by the ping command is the master node in a usable state.
6. Performance Tuning
(1) Write Optimization
Batch submission, optimize storage devices, the rational use of segment merge, refresh reduce the number of (in order to improve write performance, using delay tactics write, the data is about to be written in the first write to memory, when more than one second delay when (the default), it will trigger a refresh, data refresh will refresh the memory in the form of segments of the operating system's file cache system), reduce the number of flush, reducing the number of copies
(2) read optimization
Avoid large result sets and deep plowing, select the appropriate route, SearchType, periodically delete
7. Elasticsearch heap memory allocation
(1) should not exceed 50% of the physical memory
(2) heap memory size should not exceed 32GB
Java uses the memory pointer compression technology to solve this problem.
Chapter 6 Secret comprehensive distributed regular tasks
1. how to make one timed task only one server at a time in the implementation of the trigger?
(1) is only performed on one server
But this has obvious drawbacks: a single point of risk and uneven distribution of resources.
(2) configure operating parameters dispersion
For example, create a configuration item specifies the class name of the scheduled task server needs to perform, so you can manually dispersed scheduled tasks to different servers when deploying the service.
(3) through a global "lock" the implementation of mutually exclusive
Use ZooKeeper, Redis or database (relying on a unique index) and other means.
2. TBSchedule
(1 Introduction
Ali open distributed scheduling framework, changes can be made dynamically batch tasks are executed in parallel is dynamically allocated to a plurality of machines JVM, and the like has the advantage of failover.
(2) TBSchedule dynamically scalable principle
TBSchedule work between a host processor and the hardware abstraction layer of a scheduling server IStrategyTask, by the scheduling server complete decoupling layer. Processor ScheduleProcessor work is truly responsible for specific tasks processing, multi-threaded processors inner workings of the way parallel processing tasks, and scheduling server is a one to one relationship. The number dependent on the scheduling server hosts can be customized distribution or uniform distribution.
(3) TBSchedule fragmentation mechanism
TBSchedule batch task to be fragmented, logical tile itself is defined by the consumer according to specific business TBSchedule, a fragment corresponding to a dispatch processor, but the processor may correspond to a plurality of scheduled tasks slices, and further by the dispatch processor processor concurrent operation corresponding to the process.
3. Elastic-Job
(1 Introduction
Elastic-Job is Dangdang CAPE source of distributed scheduling solution that supports mission fragmentation function, can make full use of resources.
(2) Elastic-Job into Elastic-Job-Lite and Elastic-Job-Cloud composition
Elastic-Job-Lite is positioned as a lightweight non-central solutions, providing a distributed task in the form of Jar package of coordinated services. The Elastic-Job-Cloud using Mesos + Docker solutions, providing additional resource management, application distribution, and process isolation and other services.
(3) slicing task
1 "aims to disperse a task to be performed on different machines, either stand-alone computing power to solve the limited problem, but also part of the mission failed to reduce the impact on the overall system. Developers need to address fragmentation own a correspondence relationship with the real data items, keeping in mind the task failure retry idempotent.
2 "fragment is typically greater than the number of entry to the server, the server is preferably a multiple of the number is greater than, the job will rational use of distributed resources, dynamically allocated slice item.
============================= continue to look next ================= ======================
See Part ===================== ========================= ===========
Development and comparison of Chapter 7 RPC service analysis
1. RPC uses the client / server model, the requesting program is a client, the service provider is a server.
RPC is built on language level, is cross-language, which in the OSI model, the intermediary between the session layer and the presentation layer.
Key 2. RPC achieve transparent remote procedure calls
Is to create a client stub (client stub), the stub (stub) as agent (agent) mode where the agent (agent), after generating proxy code, agent code will be able to communicate with a remote server, the process of communication by RPC framework to achieve, and the caller as easy as calling native code. In the client view, stub functions just like ordinary local function the same, but actually contains the code over a network to send and receive messages.
(1) The client stub method will be packaged and encapsulated into parameters and sent to one or more network server message body. Encapsulated message to the network parameters in a process known as encoding (encode), all of it will serialize the data byte array format.
(2) the server stub (server stub) receiving a message sent by the client, and the message is encoded parameters (decode), it will usually parameter from the standard format into a network specific language format.
(3) After the server stub return values are encoded and serialized, transmitted to the client through one or more network messages.
3. In the JVM uses RMI's, Java supports two modes of operation: marking dirty data and clean up. When the object is still in use, the local JVM periodically sends a call to mark dirty data to the server's JVM. Dirty marker server based on the given time interval periodically re-send a heartbeat message. When the client is no more local referencing the remote object, it will send a clean-up call to the server.
Chapter 8 Dubbo combat and source code analysis
1. Dubbo configuration of three categories
(1) Service Discovery class: Indicates that the configuration items for registration and discovery service, the purpose is to allow consumers to find providers.
(2) service management class: Indicates that the configuration items for the relationship between governance services, or provide convenient conditions for development and testing.
(3) tuning categories: indicates that the configuration item for performance tuning, have different effects on performance of different options.
2. Set a timeout by the service provider, as the service provider more clearly how long a method needs to be performed.
3.
(1) If the service needs to warm up time, such as initializing the cache, waiting for the resources in place and so on, you can use attributes delay service delays exposure.
(2) excessive if a concurrent services, beyond the capacity of a server control property --- using executes concurrently.
(3) Client concurrency control, by limiting actives property.
4. Service Isolation
Intended to limit the scope of the propagation range and when the system fails, thus ensuring that only the problem of unavailable services, other services or normal.
Isolation generally thread isolation, process isolation, reading and writing isolated, isolated from the cluster and the engine room isolation, and isolation Dubbo also provides grouping, namely using the group attribute grouping.
NIO-based non-blocking mechanism implemented 5. Dubbo when the asynchronous call.
In the process of remote calls if abnormal or need correction, you can use the Dubbo event notification mechanism.
6. Dubbo Supported protocols
Dubbo communication protocol --- small packets, high concurrency service
--- Hessian protocol to transfer data providers and large number of services than consumers
HTTP protocol or the Hessian --- for external and internal communication of a scene, if you want to penetrate the firewall restrictions
Precautions 7. Dubbo agreement
(1) consumers in practice more than the number of data providers
(2) large packet can not pass
(3) single long recommended asynchronous connection
Long connection cable less handshaking verification, and the use of asynchronous I / O, the thread pool can be reused, to prevent C10K problem.
8. Dubbo an I / O thread model
Dubbo service provider There are two main types of thread pools: one is the I / O processing thread pool; the other is the service scheduling thread pool.
Dubbo default configuration of unlimited size CachedThreadPool thread pool, which means it will not all service requests are refused, but Dubbo limits the number of I O / thread, the default is the number of nuclear +1.
9. Service fuse and service degradation
(1) Service fuse
It is a protective measure to prevent the general software system for some reason the service appeared overload, causing the entire system to malfunction.
(2) Service downgraded
In the case of the sharp increase in pressure on the server, according to the current business situation and traffic strategically for some services and pages downgrade to free up server resources and ensure the normal operation of the core tasks.
10. Dubbo elegance is accomplished by stopping the JDK ShutdownHook.
11. Online Troubleshooting (here knock blackboard)
(1) identify problems
1 "system-level monitoring
CPU utilization of the system, system load, memory usage, network I / O load, disk load, I / O wait, use the swap area, the number of threads and the number of open file handles, etc.
2 "application level monitoring
Response time to service interface, throughput, call frequency, success rate and volatility Interface Interface
3 "layer of monitoring resources
Database (load, the SQL slow, the number of connections), the cache (the number of connections, memory occupancy, throughput, response time) and a message queue (response time, throughput, load, backlog) monitoring
(2) Orientation
Consider the following questions:
1 "question whether the system has recently been on the line?
2 "is dependent on the underlying platform and resources on line or upgrade?
3 "depend on whether the system has recently been on the line?
4 'operations staff whether they have been operational changes in the system?
5 "if there are fluctuations in the network?
6 "Recent business whether the amount?
7 "service consumer if there are promotions?
(3) to solve the problem
Each system will be designed for a variety of serious cases and demotion stop switch, so the first use of stop-loss policy when a serious problem occurs, locate and solve problems in recovery problems before. To solve the problem to locate the problem-based, must be clearly locate the root cause of problems, and then come up with effective solutions to the problem, remember before there is no clear reason, do not use every possible way to try to fix the problem, this may also lead to does not solve this problem raises another question.
(4) eliminate the impact
First of all, looking to see the operation and maintenance logs.
Second, save the scene and recover the scene. Snapshots preserve the scene in the JVM usually include saving a snapshot of the current running thread stack snapshot and save JVM memory.
12. SPI（Service Provider Interface）
It is a service provided by loading Java, written in Java code to avoid dead service provider, but registration services and discovery services through SPI loading mechanism, the decoupling of multiple modules.
Chapter 9 high-performance network middleware
1. In OSI, each layer using the next layer protocols and services. Each layer packet contains a header of its own, this layer defines the header contains information such as the exemplary IP address, TCP port numbers, and the present service needs information such as IP layer fragmentation information, the TCP flow control information and the like.
Because the TCP packet layer is to increase its own TCP header in the IP packet layer, so it seems at the IP layer, any one of the TCP packet is a normal IP packet.
The core 2. TCP sequence number and the acknowledgment number is received, in the middle through which there may be a case where data loss, retransmission by returning ACK sequence number.
3. failover at the application layer, downgrade, backup programs, a retry operation.
----------------