[01] The basic concept of Kafka

1. Base Noun

• Producer
• Consumer
• Broker-- Server -> Clusters
• Themes and zoning
  theme is a logical concept, partitions can be added to the log file.
  offset is the offset for each file partition, offset does not cross the partition -> partitions ensure orderly message.
  Partition resolved to write papers, a single machine IO bottleneck problem.
• Multi-copy
  copy is to enhance disaster recovery capabilities.
  leader responsible for a copy of read and write requests, follower and leader is responsible for synchronizing
  • AR（所有副本）=ISR（In-Sync Replicas） + OSR（Out-of-Sync Replicas）
  • HW (High Watermark)
    water line, after the data is written to all copies, offset next position. Consumers can only pull data before HW.
  • LEO (Log End Offset)
    at a written message to be offset.
    ISR is set smallest LEO HW partition.

1. Server-related parameters
   log.dirs for the log file storage directory. log.dirs higher than log.dir priority. Only log.dir the default is / tmp / kafka-logs
2. A simple example

• Command line
  to create a theme

# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --create --topic topic-demo --replicationo-factor 3 --partitions 4

Send a message

# bin/kafka-console-consumer.sh --bootstrapserver localhost:9092 --topic topic-demo

Consumer news

# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-demo