【Kafka】Super detailed introduction

concept

Kafka is a messaging engine (Messaging System), which is a set of specifications. Enterprises use this set of specifications to transfer semantically accurate messages between different systems to achieve loosely coupled asynchronous data transfer.

The goal is that system A sends a message to the message engine, and system B reads the message sent by A from the message engine.

Kafka stores data in binary. It supports both point-to-point model and publish/subscribe model.

• Point-to-point model: also called message queue model. If we take the above definition of "private version" as an example, then the message sent by system A can only be received by system B, and no other system can read the message sent by A. Examples of daily life such as telephone customer service belong to this model: the same customer's incoming call can only be handled by one customer service staff, and the second customer service staff cannot serve the customer.
• Publish/subscribe model: Different from the above, it has a topic (Topic) concept, which you can understand as a message container with similar logical semantics. The model also has a sender and a receiver, but in a different way. The sender is also called the Publisher, and the receiver is called the Subscriber. Different from the point-to-point model, this model may have multiple publishers sending messages to the same topic, and there may also be multiple subscribers, all of whom can receive messages on the same topic. Newspaper subscription in daily life is a typical publish/subscribe model.

The role of the message engine:

• Peak shaving and valley filling: Buffer the upstream and downstream instantaneous burst traffic to make it smoother. Especially for upstream systems with strong sending capabilities, without the protection of the message engine, the "fragile" downstream systems may be directly overwhelmed, resulting in an "avalanche" of full-link services. However, once there is a message engine, it can effectively resist the impact of upstream traffic, and truly fill the upstream "peaks" into "valleys", avoiding traffic shocks.
• The loose coupling between the sender and the receiver also simplifies application development to a certain extent and reduces unnecessary interactions between systems.

The noun terms are as follows:

• Message: Record. Kafka is a message engine. The message here refers to the main object processed by Kafka.
• Subject: Topic. A topic is a logical container for carrying messages, and it is often used to distinguish specific services in actual use.
• Partition: Partition. An ordered and immutable sequence of messages. There can be multiple partitions under each topic.
• Message displacement: Offset. Indicates the location information of each message in the partition, which is a monotonically increasing and constant value.
• Replica: Replica. The same message in Kafka can be copied to multiple places to provide data redundancy. These places are called replicas. Dungeons are also divided into leader dungeons and follower dungeons, each with a different role division. The copy is at the partition level, that is, each partition can be configured with multiple copies to achieve high availability.
• Producer: Producer. An application that publishes new messages to a topic.
• Consumer: Consumer. An application that subscribes to new messages from a topic.
• Consumer displacement: Consumer Offset. To represent the consumption progress of consumers, each consumer has its own consumer displacement.
• Consumer Group: Consumer Group. A group of multiple consumer instances that consume multiple partitions simultaneously to achieve high throughput.
• Rebalance: Rebalance. After a consumer instance in the consumer group hangs up, other consumer instances automatically reallocate the process of subscribing to topic partitions. Rebalance is an important means for Kafka consumers to achieve high availability.

Why is the copy of Kafka not allowed to provide external services?

• If the follower copy is allowed to provide external read services (master-write-slave read), there will first be data consistency problems. It takes time to synchronize messages from the master node to the slave node, which may cause data inconsistency between the master and slave nodes. Master writing and slave reading are nothing more than to reduce the pressure on the leader node and balance the load of read requests to the follower nodes. If Kafka's partitions are relatively evenly distributed to each broker, the effect of load balancing can also be achieved. There is no need to deliberately implement the master write Increased code implementation complexity from reading

Consumer Group：

• Under a consumer group, a partition can only be consumed by one consumer, but a consumer may be assigned multiple partitions, so the displacement of multiple partitions can also be submitted when the displacement is submitted.
• If the number of consumers in the Consumer Group > the number of partitions, a consumer cannot be assigned to any partition and is in an idle state.

Producer：

• If the producer specifies the target partition to be sent, the message will naturally go to that partition; otherwise, it will be determined according to the partition strategy specified by the producer-side parameter partitioner.class; if you have not specified partitioner.class, then the default rule is: see Whether the message has a key, if so, calculate the murmur2 hash value of the key % topic partition number; if there is no key, determine the partition according to the polling method.

monitor:

• JMXTrans + InfluxDB + Grafana (recommended)
• Kafka manager
• kafka eagle

deployment plan

disk

Estimated disk usage based on the number of retained messages:

• Number of new messages
• message retention time
• average message size
• Number of backups
• Whether to enable compression

假设你所在公司有个业务每天需要向 Kafka 集群发送 1 亿条消息，每条消息保存两份以防止数据丢失，另外消息默认保存两周时间。现在假设消息的平均大小是 1KB，那么你能说出你的 Kafka 集群需要为这个业务预留多少磁盘空间吗？

我们来计算一下：每天 1 亿条 1KB 大小的消息，保存两份且留存两周的时间，那么总的空间大小就等于 1 亿 * 1KB * 2 / 1000 / 1000 = 200GB。一般情况下 Kafka 集群除了消息数据还有其他类型的数据，比如索引数据等，故我们再为这些数据预留出 10% 的磁盘空间，因此总的存储容量就是 220GB。既然要保存两周，那么整体容量即为 220GB * 14，大约 3TB 左右。Kafka 支持数据的压缩，假设压缩比是 0.75，那么最后你需要规划的存储空间就是 0.75 * 3 = 2.25TB。

network

Estimate the number of servers based on QPS and bandwidth: (Note: Bandwidth resources in the industry are generally measured in Mbps instead of MBps)

假设你公司的机房环境是千兆网络，即 1Gbps，现在你有个业务，其业务目标或 SLA 是在 1 小时内处理 1TB 的业务数据。那么问题来了，你到底需要多少台 Kafka 服务器来完成这个业务呢？

让我们来计算一下，由于带宽是 1Gbps，即每秒处理 1Gb 的数据，假设每台 Kafka 服务器都是安装在专属的机器上，也就是说每台 Kafka 机器上没有混布其他服务，毕竟真实环境中不建议这么做。通常情况下你只能假设 Kafka 会用到 70% 的带宽资源，因为总要为其他应用或进程留一些资源。

根据实际使用经验，超过 70% 的阈值就有网络丢包的可能性了，故 70% 的设定是一个比较合理的值，也就是说单台 Kafka 服务器最多也就能使用大约 700Mb 的带宽资源。

稍等，这只是它能使用的最大带宽资源，你不能让 Kafka 服务器常规性使用这么多资源，故通常要再额外预留出 2/3 的资源，即单台服务器使用带宽 700Mb / 3 ≈ 240Mbps。需要提示的是，这里的 2/3 其实是相当保守的，你可以结合你自己机器的使用情况酌情减少此值。

好了，有了 240Mbps，我们就可以计算 1 小时内处理 1TB 数据所需的服务器数量了。根据这个目标，我们每秒需要处理 2336Mb 的数据，除以 240，约等于 10 台服务器。如果消息还需要额外复制两份，那么总的服务器台数还要乘以 3，即 30 台。

CPU

Usually Kafka is not very CPU-intensive, so there is no best practice in this regard. But in some cases Kafka broker consumes a lot of CPU:

1. The server and client use different compression algorithms;
2. Inconsistency between server and client versions causes message format conversion; 3
3. Broker-side decompression check

However, compared to bandwidth resources, CPU is usually not the bottleneck

number of partitions

There are some suggestions for partitioning on the Internet. I think this coarse-grained method is very good and worth a try:

1. First of all, you need to determine the SLA of your business. For example, you want your producer TPS to be 100,000 messages/second, assuming it is T1
2. Create a single-partition topic in your real environment to test TPS, assuming T2
3. The number of partitions you need can be roughly equal to T1 / T2

Order

Use –bootstrap-server above 2.2, use –zookeeper below 2.2

view version

cd kafka/libs
其中有kafka_2.12-2.8.0.jar，则版本为2.8.0

Find the ip/port of kafka and zookeeper

root@master~# kubectl get svc -n kafka -o wide
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE   SELECTOR
bootstrap   ClusterIP   10.109.83.55    <none>        9092/TCP            15d   app=kafka
broker      ClusterIP   None            <none>        9092/TCP            15d   app=kafka
outside-0   NodePort    10.108.34.0     <none>        32400:32400/TCP     15d   app=kafka,kafka-broker-id=0
outside-1   NodePort    10.104.65.215   <none>        32401:32401/TCP     15d   app=kafka,kafka-broker-id=1
outside-2   NodePort    10.99.118.241   <none>        32402:32402/TCP     15d   app=kafka,kafka-broker-id=2
pzoo        ClusterIP   None            <none>        2888/TCP,3888/TCP   15d   app=zookeeper,storage=persistent
zoo         ClusterIP   None            <none>        2888/TCP,3888/TCP   15d   app=zookeeper,storage=persistent-regional
zookeeper   ClusterIP   10.103.22.130   <none>        2181/TCP            15d   app=zookeeper

topic

• If the client reports an error to a certain topic to create a producer kafka: client has run out of available brokers to talk to (Is your cluster reachable?), you can manually delete the topic and then manually rebuild the topic

create topic

./kafka-topics.sh --zookeeper 10.103.22.130:2181 --create --topic ttt1 --partitions 1 --replication-factor 1

Check

get topic list

./kafka-topics.sh  --list --bootstrap-server 192.168.2.165:9092

./kafka-topics.sh --zookeeper 10.103.22.130:2181/kafka --list

# out
__consumer_offsets
topic1
kafka-image-topic2

get topic details

./kafka-topics.sh --zookeeper 10.103.22.130:2181 --describe --topic topic-a

modify topic

Modify partition level parameters (such as adding partition)

./kafka-topics.sh --zookeeper 10.103.22.130:2181 --alter --topic ttt1 --partitions 2

## out
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded

delete topic

• The premise is to set the kafka配置文件server.propertiesin delete.topic.enableto true

./kafka-topics.sh --zookeeper 192.168.2.165:2181 --delete  --topic <topic_name>

Set message size limit

• Use –zookeeper for topic-level static parameters, and –bootstrap-server for dynamic parameters

./kafka-configs.sh --zookeeper 10.103.22.130:2181 --entity-type topics --entity-name ttt1 --alter --add-config max.message.bytes=10485760

## out
Completed Updating config for entity: topic 'ttt1'.

Production

View production

./kafka-console-producer.sh --broker-list 192.168.2.158:9092 --topic topic-a

production news

./kafka-console-producer.sh --topic topic-a --bootstrap-server broker:9092
> 进站去重车数据如下：
> {
    
    "Ts":1677507305663,"Data":"99990000","Info":1080}}

reference

view consumption

server

# 输入
   ./kafka-console-consumer.sh --bootstrap-server 192.168.2.111:9092   --topic topic-a
或 ./kafka-console-consumer.sh --zookeeper        127.0.0.1:2181/kafka --topic topic-a


./kafka-console-consumer.sh --bootstrap-server 192.168.2.142:32400   --topic ttt

```bash
zk启动 brew services start zookeeper或zkServer start
/usr/local/Cellar/kafka/2.4.0/libexec/bin

view offset

If you want to find the offset according to the time, you can have the following methods:

• Find the *.indexand .log, and the file name is offset.

-rw-r--r--  1 root root       43 Nov 28  2021 partition.metadata
-rw-r--r--  1 root root       10 Nov  8 12:30 00000000000000380329.snapshot
-rw-r--r--  1 root root 10485756 Nov  8 12:31 00000000000000380329.timeindex
-rw-r--r--  1 root root 10485760 Nov  8 12:31 00000000000000380329.index
drwxr-xr-x  2 root root     4096 Nov  8 12:31 ./
-rw-r--r--  1 root root     3622 Nov  8 12:33 00000000000000380329.log
drwxr-xr-x 77 root root     4096 Nov  8 12:34 ../

view backlog

server

# 输入
cd ~/deep/kafka/kafka/bin
watch -n 1 ./kafka-consumer-groups.sh --bootstrap-server 192.168.2.111:9092 --describe --group topic-a

# 输出：其中lag有值表示积压。
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
topic-a                 0          159955          159955          0          UBUNTU.local-61e8b3d0-1456-49ce-8656-fe18cab4026a    

client has run out of available brokers to talk to (Is your cluster reachable?)报错的调试

• Kafka performs consistency verification based on zookeeper. A topic must correspond to a broker. If the importer reports an [ERROR] kafka %v create producer failed：client has run out of available brokers to talk to (Is your cluster reachable?)error , it may be that there is a problem with Kafka. It may be that the offset is inconsistent and the disorder is caused. The more violent way is to delete the logs of zookeeper and Kafka .

principle

• producer = "divided into multiple partitions (for example, divided into p1, p2, p3...p10) = "each consumer reads from a different partition (if consumer1 reads p1, consumer2 cannot read p1).

UI tools

Kafka Offset Explorer Supports Mac, Win, Linux

The go sarama library uses

Shopify/sarama library

consumer

It can be created with NewConsumerGroup(), or it can be created with NewClient() first and then NewConsumerGroupFromClient().

We need to implement three callback functions: Setup(), ConsumeClaim(), and CleanUp(), and the sarama library will schedule the above functions.

If you need to reset the Offset, you can implement it through ResetOffset() in Setup().

The complete code is as follows:

package kafka

import (
	"context"
	"github.com/Shopify/sarama"
	log "github.com/siruspen/logrus"
	"lonk/configs"
	"strconv"
	"time"
)

func StartConsumerGroup(ctx context.Context, conf *configs.KafkaInputConfig, msgHandler KafkaMsgHandler) {
    
    
	cli, err := newConsumerGroup(conf) // 新建一个 client 实例
	if err != nil {
    
    
		log.Fatalf("[newConsumerGroup] conf: %v, err: %v", conf, err)
	}
	k := kafkaConsumerGroup{
    
    
		msgHandler:                msgHandler,
		ready:                     make(chan bool, 0), // 标识 consumer 是否 ready
		partitionInitialOffsetMap: conf.PartitionInitialOffsetMap,
	}
	go func() {
    
    
		defer cli.Close()
		for {
    
    
			// Consume().newSession().newConsumerGroupSession() 先调用 Setup(); 再开多个协程(每个协程都用for循环持续调用consume().ConsumeClaim()来处理消息); Consume() 内部的 <-sess.ctx.Done() 会阻塞
			if err := cli.Consume(ctx, []string{
    
    conf.Topic}, &k); err != nil {
    
    
				log.Errorln("Error from consumer", err)
			}
			if ctx.Err() != nil {
    
     // 若 ctx.cancel() 而会引发 cli.Consume() 内部对 ctx.Done() 的监听，从而结束 cli.Consume() 的阻塞, 并
				return
			}
			k.ready = make(chan bool, 0) // 当 rebalanced 时 cli.Consume() 会退出且 ctx.Err() == nil, 则重置此 channel, 继续在下一轮 for 循环调用 Consume()
		}
	}()
	<-k.ready // 直到 close(consumer.ready) 解除阻塞
	log.Infoln("Sarama consumer up and running!...")
}

func newConsumerGroup(conf *configs.KafkaInputConfig) (sarama.ConsumerGroup, error) {
    
    
	sConf := sarama.NewConfig()
	sConf.Version = sarama.V2_8_0_0
	sConf.Consumer.Offsets.Initial = sarama.OffsetOldest
	sConf.Consumer.Offsets.Retention = 7 * 24 * time.Hour
	cli, err := sarama.NewClient(conf.Brokers, sConf)
	if err != nil {
    
    
		log.Fatalf("[NewClient] conf: %v, err: %v", sConf, err)
	}
	consumerGroup, err := sarama.NewConsumerGroupFromClient(conf.Group, cli)
	if err != nil {
    
    
		log.Fatalf("[NewConsumerGroupFromClient] conf: %v, err: %v", sConf, err)
	}
	return consumerGroup, nil
}

// Consumer represents a Sarama consumer group consumer
type kafkaConsumerGroup struct {
    
    
	msgHandler                func(message *sarama.ConsumerMessage)
	ready                     chan bool
	partitionInitialOffsetMap map[string]int64
}

// Setup 回调函数 is run at the beginning of a new session, before ConsumeClaim
func (k *kafkaConsumerGroup) Setup(session sarama.ConsumerGroupSession) error {
    
    
	for topic, partitions := range session.Claims() {
    
    
		for _, partition := range partitions {
    
    
			initialOffset, ok := k.partitionInitialOffsetMap[strconv.Itoa(int(partition))]
			if !ok {
    
    
				log.Fatalf("invalid topic:%v, partition: %v", topic, partition)
			}
			log.Infof("Sarama Consumer is resetting offset to %v:%v:%v", topic, partition, initialOffset)
			session.ResetOffset(topic, partition, initialOffset, "")
		}
	}
	close(k.ready) // 启动后此处会标记 ready， 使 StartKafkaConsumer() 不再阻塞
	return nil
}

// Cleanup 回调函数 is run at the end of a session, once all ConsumeClaim goroutines have exited
func (k *kafkaConsumerGroup) Cleanup(sarama.ConsumerGroupSession) error {
    
    
	log.Infoln("Sarama Consumer is cleaning up!...")
	return nil
}

// ConsumeClaim 回调函数 must start a consumer loop of ConsumerGroupClaim's Messages().
func (k *kafkaConsumerGroup) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    
    
	// NOTE: Do not move the code below to a goroutine. The `ConsumeClaim` itself is called within a goroutine, see: https://github.com/Shopify/sarama/blob/master/consumer_group.go#L27-L29
	for {
    
    
		select {
    
    
		case message := <-claim.Messages():
			k.msgHandler(message)
			session.MarkMessage(message, "")
		// Should return when `session.Context()` is done. If not, will raise `ErrRebalanceInProgress` or `read tcp <ip>:<port>: i/o timeout` when kafka rebalance. see: https://github.com/Shopify/sarama/issues/1192
		case <-session.Context().Done():
			return nil
		}
	}
}

Reference: Example of official consumerGroup
Reference: Use of sarama consumer group
Reference: sarama partition consumer specify offset according to time