The difference between Kafka session.timeout.ms heartbeat.interval.ms parameters as well as some thoughts on data storage
Often we need to deal with the data in the computer world, and this is one of the reasons we joked CURD engineers. Write the code for two years, contact with a lot of storage systems, Redis, MySQL, Kafka, Elasticsearch ... slowly find some common design philosophy behind always so familiar, and then get to the bottom, you'll find some hidden behind these systems the mathematical theory.
Large amounts of data generated in life need to be handed over to computer processing, OLTP and OLAP divided into two categories based on the application of different treatments. Some data such as login flow, system log information, a steady stream, thrown down to collect the messaging middleware (Kafka); some data, such as a description of recording user characteristics, it is suitable to store MySQL, by date to build an index. In other words: the face of large amounts of data, the data is stored only a small step, it is important how these data together , to reflect the storage system is: Is there a convenient query interface to be able to easily and quickly found we want the data. If the data is placed on Kafka, and it is up to how the investigation? If the data on the MySQL, ideally suited for high cardinality column B + tree index query construction, and for text data type, placed on the ES, the inverted index based on this data structure, according to the tf-idf, bm25 etc. the measure of document similarity algorithm to quickly obtain the desired data.
From this you can see the different storage systems, in order to meet the "Query" storage principle behind them (employed data structure) are different. And for these storage systems are faced with two problems: high reliability and high availability. Reliability, in my opinion, is standing on the storage system itself, usually a single instance to discuss how to ensure reliable data. For example, MySQL instance running it according checkpoint mechanism through redo log files to ensure durability, in addition to double write buffer, data pages are written to ensure reliable. Similarly, there is also Elasticsearch translog mechanism used to ensure data reliability. So, I want to learn more storage systems, may wish to compare the various checkpoint mechanism between them.
Why have reliability data need it? Root cause memory is a volatile memory, according to von Neumann architecture urgency, the program will fetch data from memory to the CPU to do operations. If the data is not fsync to disk, system downtime if that data will not be lost?
For availability, from the point of view of the Client. That I do not care you are behind a redis redis instance or a cluster, you normally just give me a good reader service just fine. Here in order to avoid SPOF, distributed cluster comes in handy, a machine hung up, another machine on top. In a distributed system, the need to manage each storage instance, character then we need to divide the nodes, such as call master node, controller node and the like. After all, management is to have a level Well, we roles are the same, how to manage it? At this point, Redis cluster or clusters and Kafka Elasticsearch clusters are very different, embodied in the Redis cluster is essentially a P2P structure, while the main Elasticsearch and Kafka adopted from the model, why do you say? Although there Redis node points Master and Slave nodes, but equality between its respective Master node, the data is distributed manner Redis hash16384 slots to each master node, each master node is responsible for handling the tank falls data, which is from data distributed master node angle is defined, and the Controller node in Kafka, elasticsearch the master node is not defined in terms of the distribution of the data, but to preserve the cluster meta information from the perspective of the definition of cluster management. and about the specific differences between them in my article also had some description. In addition, MySQL as a relational database, subject to data integrity constraints, transaction support restrictions on the ability of a distributed cluster weaker.
Recently I came across a problem, more service to send a message to a Kafka topic, consumption of some large business, small business news some amount. News because Kafka has not been well supported by priority messages to the consumer, resulting in the delay of some business spending problem. A simple solution is to add several Topic, legacy systems face some problems, increase brought Topic consumers and producers of processing logic complexity. One method is to use Kafka Standalone consumer, to use consumer.partitionFor("TOPIC_NAME")get all the partition information under the topic, then use the consumer.assign(partitions)explicitly specified consumption partition consumer. Another method is based on the partition allocation strategy consumer group custom Kafka consumer, and that this time you have an understanding of Kafka already existing partition allocation strategy, and understand what triggers rebalance time, what scene?
Kafka consumer to consume messages, which partitions which consumer news consumer to do? This is the consumer's partition assignment policy, by default there are three: range, round-robin, sticky. Speaking of round-robin algorithm, really everywhere, it is often used in some scenes require load balancing. When such Elasticsearch client sends a search request to the ES Server, because each node can be used as ES coordinator node receives a user query request by default, and in the coordinator node needs Summarizing query results all fragments, which consumes a lot of memory and the CPU, ES Client is round-robin selection algorithm sends the query to nodes on which ES. If you look carefully, you will find there will be a shadow of the algorithm in Redis inside. Another example: Redis LRU Cache Key on the access pattern, generally follows a power distribution means (power-law distribution): Key access frequency with a small portion of a feature is far greater than other Key, such as access characteristic , LRU cache to achieve good results. In addition, Redis sorted set type is skiplist Based on the new skipNode node belongs to which layer it? This is also a power-law distribution problems, its source code annotations:
Returns a random level for the new skiplist node we are going to create. The return value of this function is between 1 and ZSKIPLIST_MAXLEVEL (both inclusive), with a powerlaw-alike distribution where higher levels are less likely to be returned.
In fact, I want to express some idea or that is the solution, it is versatile, used in various different storage systems, will compare them together, to better understand the principles behind the system.
Each time recently to write some notes, some of the other sorts of thoughts in my mind is always present. This would have been mainly wanted to write kafka these two configuration parameters: the difference between session.timeout.ms and heartbeat.interval.ms of the results on the first pull a pass something related to data storage.
Let's continue:
Because a topic often have multiple partitions, and we will create more consumer spending this topic in a consumer group which, therefore: there is a question: Which consumer consumption which partitions a message to it? This involves three concepts: consumer group, consumer group inside the consumer, a consumer group, and each group coordinator. conusmer partition is assigned by the group management protocol is implemented: as follows:
Each consumer consumer group which are sent to the group coordinator JoinGroup request, there will be a member of this group coordinator of all consumer information, so it is to choose the Leader consumer as a consumer, and told the Leader consumer said: You take the information and those members I'll give you a topic to arrange the partition information which is responsible for consumer consumption which partitions it
Next, Leader consumer allocation strategy according to our configuration (specified by the parameters partition.assignment.strategy) calculate the respective partition to be consumed for each consumer. Thus, each consumer group coordinator SyncGroup sends a request, but only the consumer has requested Leader partition allocation policy, after receipt of partition group coordinator Leader consumer and distributed to the consumer in this scheme. Draw a map, this is the following:
In normal circumstances, when a consumer out of the consumer group will trigger rebalance, rebalance it is called to re-formulate an idea of partition. And develop a good idea of partition, you must promptly inform each consumer, which related to the parameters and heartbeat.interval.ms. Specifically, it is: Each consumer will periodically transmitted according to a time parameter specifies heartbeat.interval.ms group coordinator hearbeat, group coordinator will each consumer response, if the rebalance occurs, each consumer response received contains REBALANCE_IN_PROGRESS identity, so that each consumer knows already happened rebalance, while group coordinator also know each consumer's survival .
Why should heartbeat.interval.ms and session.timeout.ms compare it? session.timeout.ms means: group coordinator detection time required for consumer crash occurs. A consumer group hang inside a consumer, second take up session.timeout.ms detected. For example session.timeout.ms = 10, heartbeat.interval.ms = 3
session.timeout.ms is a "logical" index, which specifies a threshold value --- 10 seconds within this threshold coordinator if any message is not received consumer, it is considered consumer coordinator hung up. The heartbeat.interval.ms is a "physical" indicator, which tells the consumer every three seconds to send a heartbeat packet to the coordinator, the smaller heartbeat.interval.ms, the more prone heartbeat packet, it will affect the number of TCP packets sent and it had a real impact , which is why I called it the "physical" indicators reasons.
If the group coordinator within a period heartbeat.interval.ms consumer does not receive a heartbeat, Group put out of the consumer, it justified. As if the consumer made a small mistake on a stick killed it. In fact, it is possible to network delay, there may occur a long time consumer GC, it affects the heartbeat packets arrive, maybe next heartbeat to normal.
The heartbeat.interval.ms certainly be less than session.timeout.ms if consumer group occurred rebalance, by heartbeat packets inside REBALANCE_IN_PROGRESS, consumer will be able to timely know happened rebalance, thereby updating the consumer consumable partition. If more than session.timeout.ms, group coordinator believe that consumer hung up, that's certainly not the rebalance tell the consumer the information.
Original: https://www.cnblogs.com/hapjin/p/10926882.html
Recently met with some normalization, word accuracy of the results of the Chinese word (word to generate a custom word), customized to meet the specific ES Analyzer plug-in symbol search, search term problem of Chinese industry needs time to write a ... .