Simple white kafka can understand the principle of analytic

Learning process kafka recommended in the following article can master the foundation. highly recommended!

Introduction

Kafka distributed messaging system provides a producer, buffer, consumer models

• broker: middle kafka cluster, the message is stored, is composed of a plurality of server clusters
• topic: kafka classification to the message provided. message broker for storing data of different topic
• producer: to broker a topic in which production data
• consumer: get the data to the broker in a topic

Design ideas

topic with the message

kafka all messages organized into a plurality of stored topic, and each topic in turn can be split into a plurality of partition, each partition has a message composed by one. Each is identified with a message sequence number incremented on behalf of their incoming order, and sequentially stored in the partition.

In this way, the message id to one way to organize.

• producer choose a topic, news production, distribution strategy append message through to the end of a partition
• consumer choose a topic, the id specified location from which to start consuming messages. After the completion of consumer retention id, next time you can continue to consume start from this position, you can start spending from any other location

The id, in kafka is called offset

This organization and processing strategy provides the following benefits:

• Consumers can demand, flexible specified offset consumption
• To ensure that the message invariance, provides a thread-safe guarantee for the concurrent consumption . Each consumer will retain its own offset, do not interfere with each other, there is no thread safety issues
• Efficient Parallel access message . Each topic messages are organized into a plurality of partition, partition evenly distributed to the server in the cluster. Production and consumption of news, will be routed to the designated partition, reduce competition, increase the parallelism of the program
• Increasing the scalability of the system message . Each topic in the message retention may be very large, the message cut through the partition into multiple sub-messages, and is responsible for balancing strategy by assigning partition to a different server. Thus when the machine load is full, the message can be re-expansion by uniform distribution
• Ensure the reliability of the message . After the completion of the message does not delete consumption, consumption by re-reset offset, to ensure that messages are not lost
• Flexible persistence strategy . Message may be saved by a specified time period (e.g., latest one day), to save storage space broker

Backup

Message units assigned to a plurality of partition server, and to partition backup units. Backup strategy: a leader and N followers, leader accepts read and write requests, followers passive replication leader. leader and followers will break up in the cluster to ensure high availability partition

producer

message producer production requires the following parameters:

• topic: Which topic to news production
• partition: partition to which the production of news
• key: the key messages according to a different partition partitions
• message: Message

According kafka source, flexibility to adjust production and partitioning strategy based on different parameters

iftopic isNonethrowErrorp=NoneifpartitionNotNoneifpartition< 0Orpartition>= numPartitions throwErrorp=partitionelif key NotNonep=hash(key) % numPartitionselsep=round-robin() % numPartitionssend message to the partitionp

The above is my translation of the pseudo-code, which is a simple round-robin polling, hash uses murmurhash

consumer

Traditional message system has two modes:

• queue
• Publish and subscribe

kafka consumer group by the two modes unitary

Each consumer will mark its own consumer group name, after consumer group system will be grouped by name, the message will be copied and distributed to all packets, each only a consumer can consume the message.

So reasoning two extreme cases:

• When the consumer group simultaneously with all consumer, the system becomes a mode queue
• When each consumer's consumer group is not the same, the system becomes a publish and subscribe

Concurrent multi-consumer news consumption, easily lead to messages out of order

By limiting consumer synchronization, you can ensure that the message and orderly, but it greatly reduces the concurrency of the program.

kafka through the concept of partition, to ensure the orderly messages within the partition it, easing the above problems. Message in the partition copy distributed to all packets, each only a consumer can consume the message. This ensures that the semantics of a message packet consumption of a partition, is the synchronization rather than concurrent. If a topic is only one partition, then this topic concurrent consumption orderly, or just a single partition and orderly.

General news message system, consumer consumption there are two models:

• push: Message advantage of real-time high. The disadvantage is that not considered consumer spending power and saturation, can easily lead producer overwhelm consumer
• pull: the advantage in speed can be controlled consumption and consumption to ensure the consumer will not be saturated. The disadvantage is that when there is no data, there will be empty polling, cpu consumption

kafka pull employed, and the use of the configurable parameters to ensure that there is data and when the data amount reaches a certain amount, only for Consumer end pull operation, or has been in a state block

kakfa integer consumer consumption status using position to record a single partition and a single partition can only be a single message within the consumer Group consumer consumption, simple to maintain a small overhead. Consumer complete, broker acknowledgment is received, position the next point of consumption offset. Because messages are not deleted after the completion of consumption, position update, consumer still can reset the offset to re-consumption message history

Messaging semantics

producer perspective

• At most once message: producer asynchronous send message or a synchronous message, but the number of retries to zero
• Message is sent at least once: producer synchronization message transmission failed, will retry timeout
• Message sent and only send one: later supports

consumer perspective

• Up to a news consumption: consumer first read the news, and then confirm that position, the final disposition of the message
• Consumer news at least once: consumer first read the message, and then processes the message, the last confirmed position
• Consumer news consumption and only once:
1. If the output of the message processing (e.g., db) can guarantee the update message idempotent, then the consumer can ensure that multiple exactly once semantics
2. If the output terminal can support two-phase commit protocol, is able to confirm the position guarantee and processing output messages while at the same time fail or succeed
3. In position after the output of the memory update process message, and to ensure that the position confirmation process output message atomic (simple, generic)

Availability

In kafka, the node under normal circumstances, all in sync in the state, when a node in a non-synchronization state, it means that the entire system problems, fault tolerance needs to be done

Synchronization represents:

• The node can communicate with zookeeper
• If the node is a follower, then the consumer position and the leader can not be a large gap (the difference between configurable)

A partition synchronization of node consisting of a collection, namely the partition of ISR

kafka by means of two fault-tolerant:

• Data Backup: The backup partition units, the number of copies can be set. When the copy number is N, on behalf of a leader, an N-followers, followers can be regarded as the Consumer leader, pulling the leader message, the append themselves to the system
• failover：
1. When the leader in the non-synchronization , the system in the election of a new leader from followers
2. When a follower state to the non-synchronization when, leader of this follower would eliminate ISR, when this follower resume and complete data synchronization after entering the ISR again

In addition, kafka there is a guarantee: when the producer produce messages only when the message is acknowledged all ISR, it indicates that the message is submitted successfully. Only submit success message to be consumed consumer

To sum up: When there are N copies, N copies are in the ISR, N-1 copies are abnormal, the system can still provide services

Let N copies of the trailer, will face the process of synchronizing data after node recovery, ISR is no node during this period, will lead to the partition service is unavailable. kafka adopt a downgrade measures to deal with: the election of a recovery as the leader node to provide services to its data as a reference, this measure is known as Dirty leader election .

As the leader is the main provider of services, kafka broker's leader sharing multiple partition on a different server to be shared equally risk

Each has parition leader, if you are running the election process within each primary partition, then the election will result in very large main process. kakfa using a lightweight way: elected as a controller, the controller monitors hang from the broker's broker cluster, as above batch selected primary partition

consistency

The above solution ensures data availability, high availability is sometimes reflected in the expense of consistency. If you want to achieve strong consistency, you can take the following measures:

• Disabling dirty leader election , when the ISR is no node, node rather not provide services do not is not fully synchronized
• Set the minimum number of ISR min_isr, to ensure that the message at least one node to be confirmed in order to submit min_isr

Endurance of

The following points based on the fact, kafka heavily dependent on disk rather than memory to store messages

• Cheap hard drive, your memory
• + Pre-read sequential read operation, the cache hit rate can be improved
• The use of surplus operating system memory as pagecache, with pre-reading (read-ahead) + write-back (write-back) technology, data is read from the cache, cache writes returns (operating system back flush), to improve the response speed of the user process
• java object actual size larger than the desired size, so that the high cost of the message stored in the memory
• When the heap memory usage is increasing, gc larger shake
• Based on design ideas to read and write files in the order, simply write code

In the choice of persistent data structure, kafka queue rather than using Btree

• The offset kafka only simple read and append operations, the queue based on the operation time complexity is O (1), the time complexity of operation based Btree O (logN)
• When a large number of file read and write, read and append the queue based on only one disk needs addressing, but Btree will involve multiple times. Addressing process greatly reduces the disk read and write performance

performance

kafka in the following four points optimized:

• The large number of small io io transformed into a few large
• Reducing data copying using sendfile
• Support snappy, gzip, lz4 three algorithms compress bulk messages, reduce the consumption of network transmission
• Nio using network model, and 1 acceptor thread + reactor model N processor threads of thread

Small amounts of a large number of read and write messages result in poor performance, by polymerization message, read and write times can be reduced (to reduce random IO), increasing the amount of data read and write single (increasing order IO)

Under ordinary circumstances, the data from the disk to the network goes through the following steps:

1. Disk -> kernel page cache
2. Kernel page cache-> User buffer
3. 用户buffer->socket buffer
4. socket buffer-> NIC buffer (NIC: network interface)

Use sendfile system call, can be simplified to:

1. Disk -> kernel page cache
2. Kernel page cache-> NIC buffer

Two copies of the reducing step. When a large number of data transfer operations, would significantly improve performance

When a large number of file read and write, read and append the queue based on only one disk needs addressing, but Btree will involve multiple times. Addressing process greatly reduces the disk read and write performance

kafka server-side using the same network, threading model. NiO-based server side, using a tcp acceptor accepts thread connection and the connection is assigned to the N proccessor threads, a thread of execution for a specific IO proccessor write logic processing operation. (Note: compared to this model, netty of N boss + N worker more flexible model)

External dependencies

zookeeper

broker node using a unique id (integer) identifies the zookeeper

/ Brokers / ids / [N] -> host: port instantaneous node

• [N] represents the number of partitions

This znode broker node of the storage port ip

/ Brokers / topics / [topic] / partitions / [N] / state -> leader, isr instantaneous node

• [Topic] on behalf of a topic name
• [N] represents the number of partitions

This znode stored Leader isr id list and the partition (composed by id)

/ Consumers / [group_id] / ids / [customer_id] -> { "topic1": #streams, ..., "topicN": #streams} instantaneous node

• [Group_id] consumer-owned groupid
• [Customer_id] consumer id, the structure is host + uuid
• topicN Subscribe topic name
• #streams number of consumer threads

This znode stores the specified number of threads used in consumer consumption topic

/ Consumers / [group_id] / offsets / [topic] / [N] -> offset persistent node

• [Group_id] consumer-owned groupid
• [Topic] Subscribe topic name
• [N] Number of Partitions

consumer can manage the offset in three ways:

• Manual management. The use of low levels of consumer api, flexible, more trouble
• To the zookeeper management. Use of high-level consumer api, set offsets.storage = zookeeper, convenient, less performance. 0.8.2 default configuration
• To kafka management. Use of high-level consumer api, provided offsets.storage = kafka, convenient, original excellent performance. The principle is to select a broker kafka as offset manager, create a topic called __consumer_offsets will offset In this topic, recommended storage

This znode stored in a designated consumer topic in the latest consumer offset

/ Consumers / [group_id] / owners / [topic] / [N] -> consumer_id instantaneous node

Specify the partition can only be one consumer consumption of all consumer group at a time, by consumer_id exist under the specified partition, then the partition will be able to ensure that this can only be consumer spending.