Basic introduction and practical examples of Kafka message queue

1、Kafka

1.1 Kafka deployment configuration

1.1.1 Download Kafka

Download Kafka https://kafka.apache.org/downloads.html

https://archive.apache.org/dist/kafka/2.4.1/kafka_2.11-2.4.1.tgz

Download Scala-2.11 version

Scala-2.11经典版本

decompress

Unzip directly to a certain directory, which can be placed together in a Java-related directory without additional installation

3.1.2 Modify configuration file

Enter the Config directory and modify the configuration file

Modify zookeeper.properties

==> zookeeper启动后会自动创建zk-data文件夹，保存所需数据。

clientPort=2181

zookeeper的端口号配置

Modify the server.properties file

advertised.listeners=PLAINTEXT://localhost:9092

zookeeper.connect=localhost:2181

run Kafka

Enter the /bin/windows directory

start zookeeper

zookeeper-server-start.bat ../../config/zookeeper.properties

Start Kafka

kafka-server-start.bat ../../config/server.properties

Subscribe to Kafka messages

kafka-console-consumer --bootstrap-server localhost:9092 --topic TEST-TOPIC

Write a Kafka producer in Python to test

pip install kafka-python

the code

# -*- coding:utf-8 -*-
import json
from kafka import KafkaProducer

if __name__ == '__main__':
    print("Kafka生产消息")
    producer = KafkaProducer(
        value_serializer=lambda v: json.dumps(v).encode('utf-8'),
        bootstrap_servers=['localhost:9092']
    )

    msgDict = {
    
    
        "id": 1,
        "name": "Kafka Producer",
        "msg": "Test Kafka Producer",
        "xList": [1, 2, 3]
    }

    producer.send("TEST-TOPIC", msgDict)
    producer.close()

Console running screenshot

Kafka生产消息

Process finished with exit code 0