Kafka clustered production environment set up Notes

Kafka Taiwan official description is not detailed, but there are many online reference, is or will set up their own note-taking under the production environment (Update) on CentOS7.6.

Tuning

# vi /etc/sysctl.d/99-kafka.conf

# swap
vm.swappiness = 1
# dirty page
vm.dirty_background_ratio = 5
vm.dirty_ratio = 70
# tcp socket
net.core.rmem_default = 4194304
net.core.wmem_default = 4194304
# tcp socket
net.ipv4_tcp_rmem = 4096 65536 4194304
net.ipv4_tcp_wmem = 4096 65536 4194304
# other network
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 5120
net.core.netdev_max_backlog = 100000

topic management

By kafka comes with tools to achieve topic of management, the main tool for achieving rely kafka-topic.sh

List Topics

> kafka-topics.sh --bootstrap-server <kafka_host1>:9092,<kafka_host2>:9092,... --list

Delete topics

> kafka-topics.sh --bootstrap-server <kafka_host1>:9092,<kafka_host2>:9092,... --delete --topic <topic_name>

Note:
1. If Kafka service is configured delete.topic.enable = true, directly from the command line deletion, failure to delete Topic can be deleted by zookeeper-client. The following illustrates the use of a separate binary zookeeper packages, and has entered the bin directory.

# 进入zookeeper命令行客户端
> zkCli.sh -server <zkhost1:2181>[,<zkhost:2181>,...]
# 执行删除topic的操作
] deleteall /brokers/topics/<topic_name> 

2. If Kafka service delete.topic.enable not configured = false, directly by the delete command to delete topic, the topic will only be marked as "marked for deletion" is deleted, then deleted by zookeeper-client will not delete the topic of data , you need the appropriate topic directory log directory on the server to delete the appropriate broker .log data directory.