Start kafka service
- It is first necessary to enable zookeeper before starting the kafka service.
apache-zookeeper-3.6.0-bin\bin\zkserver
- After starting the kafaka service is the startup configuration file.
kafka-server-start.bat ./config/server.properties
window uses kafka
- After downloading kafka, if you use the windows version to enter the bin \ window directory, use the command here. If it is a .sh file in the bin directory under the linux system.
Create theme
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
View topic
kafka-topics.bat --list --zookeeper localhost:2181
Test producer
kafka-console-producer.bat --broker-list localhost:9092 --topic test
Test consumers
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Delete topic
Add settings in server.properties,
delete.topic.enable = true is not enabled by default
kafka-topics --delete --topic test --zookeeper localhost:2181
delete data
If you want to keep the theme, only delete the existing data (log) of the theme. This can be achieved by modifying the data retention time:
kafka-configs --zookeeper localhost:2181 --entity-type topics --entity-name test --alter --add-config retention.ms=3000
//修改保留时间为三秒，但不是修改后三秒就马上删掉，kafka是采用轮训的方式，轮训到这个主题发现三秒前的数据都是删掉。时间由自己在server.properties里面设置，设置见下面。
Execution output: Completed Updating config for entity: topic 'test'.
Note: Modify the retention time to 10 seconds, not delete it immediately. Kafka is a polling method. When polling to this topic, the data 10 seconds ago is deleted.
The time is dominated by the log.retention.check.interval.ms option in server.properties
Assuming that the value of log.retention.check.interval.ms is 1 minute, then after waiting for 70 seconds, the data of this topic will be automatically deleted
View topic again
kafka-configs.bat --zookeeper zookeeper-1.default.svc.cluster.localhost:2181 --describe --entity-type topics --entity-name test
kafka placement
The global strategy is configured in the server.properties file for each topic
server.properties
Segmentation strategy attributes
| Attribute name | meaning | Defaults |
|---|---|---|
| log.roll.{hours,ms} | The cycle time of the log rolling, when a specified cycle time is reached, a new segment is forced to be generated | 168 （7day） |
| log.segment.bytes | The maximum capacity of each segment. When the specified capacity is reached, a new segment will be forced to be generated | 1G (-1 is unlimited) |
| log.retention.check.interval.ms | Cycle time for checking log fragment files | 60000 |
Log refresh strategy
Kafka's logs are actually in the cache at first, and then written to the log files batch by batch according to the strategy to improve the throughput rate.
| Attributes | Explanation | Defaults |
|---|---|---|
| log.flush.interval.messages | How many messages are written to the log file | 10000 |
| advertised.listeners | The solution to the problem that the Kafka cluster cannot access the external network, fill in your local intranet IP here | PLAINTEXT://your.host.name:9092 |
| log.flush.interval.ms | When this time is reached, a flush is forced | null |
| log.flush.scheduler.interval.ms | Check periodically whether the information needs to be flushed | Great value |
Log preservation and cleaning strategy
| Attribute name | meaning | Defaults |
|---|---|---|
| log.cleanup.polict | There are only two strategies for log cleaning and saving: delete and compact | delete |
| log.retention.hours | Log storage time, you can choose hours, minutes and ms | 168 (7day) |
| log.retention.bytes | The maximum allowable log file before deletion | -1 |
| log.segment.delete.delay.ms | The retention time before the log file is actually deleted | 60000 |
| log.cleanup.interval.mins | How often to call the cleanup step | 10 |
| log.retention.check.interval.ms | Check periodically whether any logs meet the conditions for deletion (used in new versions) | 300000 |
In particular, here is the real time of the log. When the conditions for deletion are met, the log will be "deleted", but the deletion here is only to "delete" the log, and the file just cannot be indexed. But the file itself still exists. Only after log.segment.delete.delay.ms has passed this time will the file be deleted from the file system.
