Performance testing and monitoring tools Clusters
Kafka provides a lot of useful tools, such as Kafka Design Analysis (three) - Kafka High Availability (lower) mentioned in the operation and maintenance Tools --Partition Reassign Tool, Preferred Replica Leader Election Tool, Replica Verification Tool, State Change Log Merge Tool. This chapter describes Kafka provide performance testing tool, Metrics reporting tools and Yahoo open source Kafka Manager.
Kafka performance test scripts
$KAFKA_HOME/bin/kafka-producer-perf-test.shThe script is designed for performance testing Kafka Producer, the main indicators output 4, a total amount of message transmission (in MB), sends a message per second (MB / second), total number of messages sent, message sent per second ( records / second). Except that the test results are output to the standard output, the script also provides CSV Reporter, soon results stored in the form of a CSV file, to facilitate use of the test result analysis tool in other$KAFKA_HOME/bin/kafka-consumer-perf-test.shThe script is used to test Kafka Consumer performance testing metrics and performance test scripts as Producer.
Kafka Metrics
Kafka use Yammer Metrics reporting server and Metric information about the client. Yammer Metrics 3.1.0 Metrics provide six types of collection --Meters, Gauges, Counters, Histograms, Timers, Health Checks. At the same time, Yammer Metrics will Metric collection and reporting (or release) separation, can be freely combined according to need. It currently supports Reporter has Console Reporter, JMX Reporter, HTTP Reporter , CSV Reporter, SLF4J Reporter, Ganglia Reporter, Graphite Reporter. Thus, Kafka which also supports output over several Metrics information Reporter.
Using JConsole to view a single server Metrics
One view Kafka Server Metrics case by using JConsole JMX, is not to install other tools (since the Kafka has been installed, it will certainly Java installed, and JConsole is a Java tool that comes with) is the simplest and most convenient way.
First you must enable Kafka's JMX Reporter JMX_PORT by setting a valid value for an environment variable. As export JMX_PORT=19797. JConsole can then be used to access a server via a port table Kafka above to view its Metrics information, as shown in FIG.
One benefit of using JConsole is no need to install additional tools, obvious shortcomings, not intuitive data presentation, data organization unfriendly, more importantly Metrics can not monitor the entire cluster simultaneously. In the figure, at kafka.cluster-> Partition-> UnderReplicated-> topic4, only two nodes 2 and 5, this is not only because these two topic4 Partition data is in state replication. In fact, topic4 on the Broker only these two Partition, other Partition on other Broker, so the server through JMX Reporter saw only two Partition.
View the entire cluster by Kafka Manager Metrics
Related Sponsors
QCon Beijing 2017, April 16-18, Beijing · National Convention Center, exciting content Behind the Scenes
Kafka Manager is open source Kafka Yahoo management tool. It supports the following functions:
- Manage multiple clusters
- Easy viewing cluster status
- 执行preferred replica election
- Partition batch generates and executes a plurality of programs allocated Topic
- Creating Topic
- Delete Topic (only supports 0.8.2 and above, also asked in the Broker
delete.topic.enableis set to true) - Add to an existing Topic Partition
- Configuring update Topic
- Under the premise of Broker JMX Reporter turned on, polling Broker level and Topic-level Metrics
- Consumer Group and consumption status monitoring
- Support add and view LogKafka
After installing Kafka Manager, Cluster very convenient to add only need to specify the list Zookeeper Cluster and used to indicate the version Kafka, as shown in FIG.
Here we must note, add Cluster here refers to add an existing cluster Kafka access control list, rather than deploying a new Cluster of Kafka by Kafka Manager, this is different with Cloudera Manager.
Kafka Benchmark
Kafka is a core feature of high throughput, so the test paper focuses on Kafka throughput.
Test herein using a total of six virtual machine installed Red Hat 6.6, 3 as Broker, as a further three or Producer Consumer. Each virtual machine configuration is as follows:
- CPU：8 vCPU， Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz，2 Sockets，4 Cores per socket，1 Thread per core
- Memory: 16 GB
- Disk: 500 GB
Open Kafka JMX Reporter using port 19797, using JMX polling function monitoring performance testing Kafka-Manager in throughput.
This paper tests the following four scenarios, the main index test is how many megabytes of data per second, the number of messages per second.
Producer Only
This set of tests does not use any Consumer, just start Broker and Producer.
Producer Number VS. Throughput
Experimental conditions: 3 Broker, 1 th Topic, 6 th the Partition, without the Replication, asynchronous mode, Payload message of 100 bytes.
Test items: two, three, were tested throughput of Producer.
测试目标：如Kafka设计解析（一）- Kafka背景及架构介绍所介绍，多个Producer可同时向同一个Topic发送数据，在Broker负载饱和前，理论上Producer数量越多，集群每秒收到的消息量越大，并且呈线性增涨。本实验主要验证该特性。同时作为性能测试，本实验还将监控测试过程中单个Broker的CPU和内存使用情况
测试结果：使用不同个数Producer时的总吞吐率如下图所示
由上图可看出，单个Producer每秒可成功发送约128万条Payload为100字节的消息，并且随着Producer个数的提升，每秒总共发送的消息量线性提升，符合之前的分析。
性能测试过程中，Broker的CPU和内存使用情况如下图所示。
(点击放大图像)
由上图可知，在每秒接收约117万条消息（3个Producer总共每秒发送350万条消息，平均每个Broker每秒接收约117万条）的情况下，一个Broker的CPU使用量约为248%，内存使用量为601 MB。
Message Size VS. Throughput
实验条件：3个Broker，1个Topic，6个Partition，无Replication，异步模式，3个Producer。
测试项目：分别测试消息长度为10，20，40，60，80，100，150，200，400，800，1000，2000，5000，10000字节时的集群总吞吐量。
测试结果：不同消息长度时的集群总吞吐率如下图所示:
由上图可知，消息越长，每秒所能发送的消息数越少，而每秒所能发送的消息的量（MB）越大。另外，每条消息除了Payload外，还包含其它Metadata，所以每秒所发送的消息量比每秒发送的消息数乘以100字节大，而Payload越大，这些Metadata占比越小，同时发送时的批量发送的消息体积越大，越容易得到更高的每秒消息量（MB/s）。其它测试中使用的Payload为100字节，之所以使用这种短消息（相对短）只是为了测试相对比较差的情况下的Kafka吞吐率。
Partition Number VS. Throughput
实验条件：3个Broker，1个Topic，无Replication，异步模式，3个Producer，消息Payload为100字节。
测试项目：分别测试1到9个Partition时的吞吐量。
测试结果：不同Partition数量时的集群总吞吐率如下图所示：
由上图可知，当Partition数量小于Broker个数（3个）时，Partition数量越大，吞吐率越高，且呈线性提升。本文所有实验中，只启动3个Broker，而一个Partition只能存在于1个Broker上（不考虑Replication。即使有Replication，也只有其Leader接受读写请求），故当某个Topic只包含1个Partition时，实际只有1个Broker在为该Topic工作。如之前文章所讲，Kafka会将所有Partition均匀分布到所有Broker上，所以当只有2个Partition时，会有2个Broker为该Topic服务。3个Partition时同理会有3个Broker为该Topic服务。换言之，Partition数量小于等于3个时，越多的Partition代表越多的Broker为该Topic服务。如前几篇文章所述，不同Broker上的数据并行插入，这就解释了当Partition数量小于等于3个时，吞吐率随Partition数量的增加线性提升。
当Partition数量多于Broker个数时，总吞吐量并未有所提升，甚至还有所下降。可能的原因是，当Partition数量为4和5时，不同Broker上的Partition数量不同，而Producer会将数据均匀发送到各Partition上，这就造成各Broker的负载不同，不能最大化集群吞吐量。而上图中当Partition数量为Broker数量整数倍时吞吐量明显比其它情况高，也证实了这一点。
Replica Number VS. Throughput
实验条件：3个Broker，1个Topic，6个Partition，异步模式，3个Producer，消息Payload为100字节。
测试项目：分别测试1到3个Replica时的吞吐率。
测试结果：如下图所示:
由上图可知，随着Replica数量的增加，吞吐率随之下降。但吞吐率的下降并非线性下降，因为多个Follower的数据复制是并行进行的，而非串行进行。
Consumer Only
实验条件：3个Broker，1个Topic，6个Partition，无Replication，异步模式，消息Payload为100字节。
测试项目：分别测试1到3个Consumer时的集群总吞吐率。
测试结果：在集群中已有大量消息的情况下，使用1到3个Consumer时的集群总吞吐量如下图所示：
由上图可知，单个Consumer每秒可消费306万条消息，该数量远大于单个Producer每秒可消费的消息数量，这保证了在合理的配置下，消息可被及时处理。并且随着Consumer数量的增加，集群总吞吐量线性增加。
根据Kafka设计解析（四）- Kafka Consumer设计解析所述，多Consumer消费消息时以Partition为分配单位，当只有1个Consumer时，该Consumer需要同时从6个Partition拉取消息，该Consumer所在机器的I/O成为整个消费过程的瓶颈，而当Consumer个数增加至2个至3个时，多个Consumer同时从集群拉取消息，充分利用了集群的吞吐率。
Producer Consumer pair
实验条件：3个Broker，1个Topic，6个Partition，无Replication，异步模式，消息Payload为100字节。
测试项目：测试1个Producer和1个Consumer同时工作时Consumer所能消费到的消息量。
测试结果：1,215,613 records/second。
转载于:https://www.cnblogs.com/licheng/p/6443609.html