Kafka 01——Kafka installation and simple entry-level use

1. Download and install

1.1 JDK installation

• Because Kafka's own development language is developed based Scala on jdk, you must first install jdk. Regarding the installation of jdk, you can read the following two articles, as follows:Scala
  • Linux installation JDK1.8 .
  • Ordinary users (non-root users) under Linux install JDK8 .

1.2 Installation of Zookeeper

1.2.1 Selection of Zookeeper version

• You can select the corresponding Zookeeper version based on the kafaka version downloaded below. How to choose is as follows:
  • Method 1: Directly look at the jar package under libs in Kafka, as follows:
    
  • Method 2: You can also check the source code to see the version information, as follows:
    

1.2.2 Download and install Zookeeper

• Go to the official website and download the corresponding version as needed, as follows:
  
• Regarding the installation of Zookeeper, you can read the following article
  Zookeeper installation and use (win+linux) .

1.3 Kafka installation

1.3.1 Download

• Go to the official website to download the required version, official website address:
  https://kafka.apache.org/downloads .
  

1.3.2 Unzip

• Then unzip it as follows:

  tar -zxvf kafka_2.12-2.8.2.tgz
  

  

1.3.3 Modify configuration file

• Modify the configuration file server.propertiesas follows:
  

• The configuration information is as follows:
  

  

  # The id of the broker. This must be set to a unique integer for each broker.
  broker.id=0
  
  listeners=PLAINTEXT://内网IP:9092
  advertised.listeners=PLAINTEXT://公网IP:9092
  
  zookeeper.connect=zk的公网IP:2181
  
  # …… 其他配置，先默认即可
  

2. Start kafka

2.1 Kafka startup

• The startup command is as follows:

  ./kafka-server-start.sh ../config/server.properties &
  

  

• After successful startup, it displays:
  

• View process:

  ps -ef | grep kafka
  

  

2.2 Problems encountered when starting kafka

2.2.1 Question 1

• Problem Description,as follows:

  org.apache.kafka.common.KafkaException: Socket server failed to bind to XXX:9092: Cannot assign requested address.
  

  
• problem causes
  • The public IP of the server (the IP exposed to the outside world) and the real IP (the IP displayed by ifconfig) may be just a mapping relationship. When a user accesses the external IP, OpenStack will forward it to the corresponding real IP for access.
  • But at this time, if the server.propertiesconfiguration in the configuration file is listeners=PLAINTEXT://公网IP:9092, it cannot be started because the socket cannot be bound to listen, and the above error will be reported.
  • The solution is also very simple. Just change the above configuration. For listeners=PLAINTEXT://内网IP:9092other uses, just use the public IP as normal. It has nothing to do with the real intranet IP.
• Solve the problem

  • View firstIntranet IP

    ifconfig
    

    

  • Then modify the configuration file server.propertiesas follows:

    listeners=PLAINTEXT://内网IP:9092
    advertised.listeners=PLAINTEXT://公网IP:9092
    

    

• Then just restart.
  

2.2.2 Question 2

3. Simple and practical

3.1 Create topic

• as follows:

  ./kafka-topics.sh --create --zookeeper zookeeper服务的IP:2181 --replication-factor 1 --partitions 1 --topic susu-topic
  

  

3.2 View created topics

• as follows:

  ./kafka-topics.sh -list --zookeeper zookeeper服务的IP:2181
  

  

3.3 Send messages

3.3.1 Send message command

• as follows:

  ./kafka-console-producer.sh --broker-list 内网IP:9092 --topic susu-topic
  
  或者
  
  ./kafka-console-producer.sh --broker-list 公网IP:9092 --topic susu-topic
  

  

3.3.2 Problems encountered

3.3.2.1 Question 1

• The problem is described as follows:

  [2023-08-10 09:01:57,758] WARN [Producer clientId=console-producer] Bootstrap broker 43.143.190.116:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
  [2023-08-10 09:02:15,979] WARN [Producer clientId=console-producer] Bootstrap broker 43.143.190.116:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
  [2023-08-10 09:02:49,538] WARN [Producer clientId=console-producer] Bootstrap broker 43.143.190.116:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
  [2023-08-10 09:02:50,562] ERROR Error when sending message to topic susu-topic with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
  
  org.apache.kafka.common.errors.TimeoutException: Topic susu-topic not present in metadata after 60000 ms.
  
  >[2023-08-10 09:03:18,069] WARN [Producer clientId=console-producer] Bootstrap broker 43.143.190.116:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
  [2023-08-10 09:03:47,001] WARN [Producer clientId=console-producer] Bootstrap broker 43.143.190.116:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
  

  
• Solving the problem:
  My solution here is to open port 9092. Regarding open ports and firewall issues, you can read the following articles:
  Check the firewall status, close the firewall, open and close ports, etc. under Linux .

3.3.2.2 Question 2

• The problems after opening the port are as follows, problem description:

   WARN [Producer clientId=console-producer] Connection to node -1 (/XXX:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
  

  
• Solve the problem:
  Here is to restart the service, restart kafka, and the problem will be solved. I don’t know what the problem is. In short, restarting can solve 99% of the problems! ! !

3.4 Receive messages

3.4.1 Receive message demonstration

• as follows:

  ./kafka-console-consumer.sh --bootstrap-server 外网IP或内网IP:9092 --topic susu-topic --from-beginning
  

  

3.4.2 Tips on receiving messages

• as follows:
  • Consumption method 1:Spend from scratch

    ./kafka-console-consumer.sh --bootstrap-server IP:9092 --topic susu-topic --from-beginning
    

  • Consumption method 2: Start consuming from the offset + 1 of the last message

    ./kafka-console-consumer.sh --bootstrap-server IP:9092 --topic susu-topic 
    

• details as follows:
  

3.5 View zk

• as follows:
  

4. Summary

4.1 Basic concepts of Kafka

• Kafka is a message queue that is mainly used to process message queues in a large amount of data. It is generally used for log processing. Since it is a message queue, Kafka also has the corresponding characteristics of the message queue.
• Kafka, like other MQ, also has its own infrastructure, which mainly consists of producer Producer, Kafka cluster Broker, consumer, and registered message Zookeeper.
  • Topic : Topic, a virtual concept, consists of 1 to multiple Partitions. It can be understood as a queue. Both producers and consumers are oriented to a Topic.
  • Partition : partition, the actual message storage unit. In order to achieve scalability, a very large Topic can be distributed to multiple Brokers. A Topic can be divided into multiple Partitions. Each Partition is an ordered queue (the partitions are ordered, but global ordering cannot be guaranteed).
  • Producer : Message producer, the role that publishes messages to Kafka.
  • Consumer : Message consumer, a client that pulls messages from Kafka for consumption.
  • Broker : Broker, a Kafka server is a Broker, a cluster consists of multiple Brokers, and one Broker can accommodate multiple Topics.

4.2 Common commands

4.2.1 Commonly used basic commands

• Start command:

  ./kafka-server-start.sh ../config/server.properties &
  

• Stop command:

   ./kafka-server-stop.sh
  

• Place

4.2.2 Simple and practical commands

• Create topic

  # 创建topic
  ./kafka-topics.sh --create --zookeeper zookeeper的IP:2181 --replication-factor 1 --partitions 1 --topic susu-topic
  

• View created topic information

  # 查看已经创建的topic信息
  ./kafka-topics.sh -list --zookeeper zookeeper的IP:2181
  

• Send a message

  ./kafka-console-producer.sh --broker-list 内网IP:9092 --topic susu-topic
  
  或者
  
  ./kafka-console-producer.sh --broker-list 公网IP:9092 --topic susu-topic
  

• receive messages

  ./kafka-console-consumer.sh --bootstrap-server 外网IP或内网IP:9092 --topic susu-topic --from-beginning
  

4.3 Data log

• Check the data log, storage path, and see which path is configured in the configuration file log.dirs, as follows:
  

4.4 Structure diagram

• as follows: