12, kafka how to ensure no loss of data
12.1 How to ensure no loss of data producers
kafka the ack mechanism: when kafka transmission data, each transmission confirmation message will have a feedback mechanism to ensure that the message can be properly received
If synchronous mode: ack mechanism to ensure that data is not lost if the ack is set to 0, a big risk, is generally not recommended set to 0
producer.type=sync
request.required.acks=1
If asynchronous mode: send to control data through the buffer, there are two values to control the time threshold quantity threshold value of the message, if the buffer is full of data has not been sent, if the set is cleaned up immediately mode, the risk is large, must be set to blocking mode
Conclusion: producer likely to lose data, but can be configured to ensure that the message is not lost
producer.type=async
request.required.acks=1
queue.buffering.max.ms=5000
queue.buffering.max.messages=10000
queue.enqueue.timeout.ms = -1
batch.num.messages=200
12.2 How to ensure that consumers do not lose data
By offset commit to ensure that data is not lost, kafka own record every time the offset value of consumption, consumption continues to the next time, then the last offset to consume
12.3 broker how to ensure that data is not lost:
partition copy mechanism
13, installation and use of monitoring tools kafkaManager
Step One: Download kafkaManager
Source Download:
https://github.com/yahoo/kafka-manager/
Download the source code, and then upload decompression ready to compile
cd /export/servers/kafka-manager-1.3.3.15
unzip kafka-manager-1.3.3.15.zip -d ../servers/
./sbt clean dist
After the completion of the translation, we need to install this package just below the path
/export/servers/kafka-manager-1.3.3.15/target/universal
Need to download the source code to conduct their own compiler, too much trouble, do not compile your own, have compiled version can be used directly to take over
Step Two: Upload the compiled archive and extract
We will be compiled kafkamanager archive uploaded to the server and unzip
cd /export/softwares
unzip kafka-manager-1.3.3.15.zip -d /export/servers/
The third step: modify the configuration file
cd /export/servers/kafka-manager-1.3.3.15/
vim conf/application.conf
kafka-manager.zkhosts="node01:2181,node02:2181,node03:2181"
Step four: add execute permissions to the startup script kafkamanager
cd /export/servers/kafka-manager-1.3.3.15/bin
chmod u+x ./*
Step Five: Start the process kafkamanager
cd /export/servers/kafka-manager-1.3.3.15
nohup bin/kafka-manager -Dconfig.file=/export/servers/kafka-manager-1.3.3.15/conf/application.conf -Dhttp.port=8070 2>&1 &
Step Six: browser page views
==================================================
Lessons Learned:
1, kafka basic introduction
2, kafka comparison with traditional message queue
Guarantee transaction-based: Why business systems activeMQ
Why Big Data platform with kafka: throughput, speed
3, the introduction kafka component model: get
producer: Producers message
A kind of collection message: topic
broker: server
consumer: the consumer, the consumer topic among data
partition: the partition, a partition composed of a plurality of topic
segement: a partition composed of a plurality segement
.log: Data File
.index: Index of data
4, kafka installation: buttoned
5, use the command line to get kakfa
Creating topic Producer analog simulation consumers
6, kafka the java API to run through the line
7, data partitioning strategy kafka: know
The first: Specifies the partition number
The second: no partition number, with the key value data, to partition the key taken directly hashCode
Third: No partition number, there is no key value, round-robin transmission
Fourth: Custom Partitioning
8. notes for kafka profile
9, flume and get kafka integration
10, kakfa use of monitoring tools as possible to the success