Go operation major message queue tutorials (RabbitMQ, Kafka)

Go operation major message queue tutorial

1 RabbitMQ

1.1 Concept

①Basic noun

There are currently many mq products on the market, such as RabbitMQ, Kafka, ActiveMQ, ZeroMQ, and RocketMQ donated by Alibaba to Apache. Even NoSQL such as redis supports MQ functions.

1. Broker: Represents the message queue service entity
2. Virtual Host: virtual host. Identify a collection of exchanges, message queues, and related objects. The vhost is the basis of the AMQP concept and must be specified at link time. The default vhost of RabbitMQ is /.
   • AMQP (Advanced Message Queuing Protocol) Advanced Message Queuing Protocol
3. Exchange: The exchange is used to receive the messages sent by the producer and route these messages to the queue in the server.
4. Queue: Message queue, used to save messages until sent to consumers. It is the container of the message and the destination of the message. A message can be put on one or more queues. The message is always in the queue, waiting for the consumer to connect to the queue to take it away.

②Common mode

1. simple simple mode

The consumer of the message (consumer) monitors (while) the message queue. If there is a message in the queue, it will be consumed. After the message is taken away, it will be automatically deleted from the queue (the hidden message may not be processed correctly by the consumer and has been removed from the queue. disappeared, resulting in the loss of the message)

2. Worker working mode

Multiple consumers competing for messages from a queue

• (Hidden danger, in the case of high concurrency, a certain message will be shared by multiple consumers by default. You can set a switch (syncronize, which is different from the performance of synchronization lock) to ensure that a message can only be used by one consumer)
• Application scenarios: red envelopes; resource scheduling in large projects (the task allocation system does not need to know which task execution system is idle, it directly throws the task into the message queue, and the idle system automatically competes)

3. publish/subscribe publish subscription (shared resources)

Consumers subscribe to messages, and then get messages from subscribed queues for consumption.

• X represents the internal components of switch rabbitMQ. The erlang message generator is the code completion, and the execution efficiency of the code is not high. The message generator puts the message into the switch, and the switch publishes and subscribes to send the message to all message queues. The consumers of the corresponding message queue take to consume news
• Related scenarios: mass mailing, group chat, broadcasting (advertising)

4. routing routing mode

• The switch routes messages to different queues according to the routing rules
• The message producer sends the message to the switch to judge according to the route. The route is a string (info). The currently generated message carries the route character (method of the object). The switch can only match the message queue corresponding to the route key according to the route key. Consumers can consume messages;

5. topic topic mode (a kind of routing mode)

• The asterisk pound sign represents a wildcard
• Asterisks represent multiple words, hash marks represent one word
• Routing function adds fuzzy matching
• The message producer generates a message and sends the message to the switch
• The switch fuzzily matches the corresponding queue according to the rules of the key, and the listening consumer of the queue receives the message consumption

1.2 Build (docker mode)

①Pull the image

# 拉取镜像
docker pull rabbitmq:3.7-management

②Create and start the container

# 创建并运行容器
docker run -d --name myrabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.7-management
#5672是项目中连接rabbitmq的端口（我这里映射的是5672），15672是rabbitmq的web管理界面端口（我映射为15672）

# 输入网址http://ip:15672即可进入rabbitmq的web管理页面,账户密码：guest / guest

③The web interface creates users and virtual hosts

Next, for our follow-up operations, first we create a new Virtual Host and assign him a user name to isolate data and create it according to our own needs.

1. Add virtual host
   
2. New users
   
3. Click on the newly created user to set its host
   
   
4. final effect
   

1.3 Code operation

①RabbitMQ struct: contains creation, consumption, and production messages

package RabbitMQ

import (
	"fmt"
	"github.com/streadway/amqp"
	"log"
)

//amqp:// 账号 密码@地址:端口号/vhost
const MQURL = "amqp://ziyi:[email protected]:5672/ziyi"

type RabbitMQ struct {
    
    
	//连接
	conn *amqp.Connection
	//管道
	channel *amqp.Channel
	//队列名称
	QueueName string
	//交换机
	Exchange string
	//key Simple模式 几乎用不到
	Key string
	//连接信息
	Mqurl string
}

//创建RabbitMQ结构体实例
func NewRabbitMQ(queuename string, exchange string, key string) *RabbitMQ {
    
    
	rabbitmq := &RabbitMQ{
    
    QueueName: queuename, Exchange: exchange, Key: key, Mqurl: MQURL}
	var err error
	//创建rabbitmq连接
	rabbitmq.conn, err = amqp.Dial(rabbitmq.Mqurl)
	rabbitmq.failOnErr(err, "创建连接错误！")
	rabbitmq.channel, err = rabbitmq.conn.Channel()
	rabbitmq.failOnErr(err, "获取channel失败")
	return rabbitmq
}

//断开channel和connection
func (r *RabbitMQ) Destory() {
    
    
	r.channel.Close()
	r.conn.Close()
}

//错误处理函数
func (r *RabbitMQ) failOnErr(err error, message string) {
    
    
	if err != nil {
    
    
		log.Fatalf("%s:%s", message, err)
		panic(fmt.Sprintf("%s:%s", message, err))
	}
}

//简单模式step：1。创建简单模式下RabbitMQ实例
func NewRabbitMQSimple(queueName string) *RabbitMQ {
    
    
	return NewRabbitMQ(queueName, "", "")
}

//订阅模式创建rabbitmq实例
func NewRabbitMQPubSub(exchangeName string) *RabbitMQ {
    
    
	//创建rabbitmq实例
	rabbitmq := NewRabbitMQ("", exchangeName, "")
	var err error
	//获取connection
	rabbitmq.conn, err = amqp.Dial(rabbitmq.Mqurl)
	rabbitmq.failOnErr(err, "failed to connecct rabbitmq!")
	//获取channel
	rabbitmq.channel, err = rabbitmq.conn.Channel()
	rabbitmq.failOnErr(err, "failed to open a channel!")
	return rabbitmq
}

//订阅模式生成
func (r *RabbitMQ) PublishPub(message string) {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 广播类型
		"fanout",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "failed to declare an excha"+"nge")

	//2 发送消息
	err = r.channel.Publish(
		r.Exchange,
		"",
		false,
		false,
		amqp.Publishing{
    
    
			//类型
			ContentType: "text/plain",
			//消息
			Body: []byte(message),
		})
}

//订阅模式消费端代码
func (r *RabbitMQ) RecieveSub() {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 广播类型
		"fanout",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "failed to declare an excha"+"nge")
	//2试探性创建队列，创建队列
	q, err := r.channel.QueueDeclare(
		"", //随机生产队列名称
		false,
		false,
		true,
		false,
		nil,
	)
	r.failOnErr(err, "Failed to declare a queue")
	//绑定队列到exchange中
	err = r.channel.QueueBind(
		q.Name,
		//在pub/sub模式下，这里的key要为空
		"",
		r.Exchange,
		false,
		nil,
	)
	//消费消息
	message, err := r.channel.Consume(
		q.Name,
		"",
		true,
		false,
		false,
		false,
		nil,
	)
	forever := make(chan bool)
	go func() {
    
    
		for d := range message {
    
    
			log.Printf("Received a message:%s,", d.Body)
		}
	}()
	fmt.Println("退出请按 Ctrl+C")
	<-forever
}

//话题模式 创建RabbitMQ实例
func NewRabbitMQTopic(exchagne string, routingKey string) *RabbitMQ {
    
    
	//创建rabbitmq实例
	rabbitmq := NewRabbitMQ("", exchagne, routingKey)
	var err error
	rabbitmq.conn, err = amqp.Dial(rabbitmq.Mqurl)
	rabbitmq.failOnErr(err, "failed     to connect rabbingmq!")
	rabbitmq.channel, err = rabbitmq.conn.Channel()
	rabbitmq.failOnErr(err, "failed to open a channel")
	return rabbitmq
}

//话题模式发送信息
func (r *RabbitMQ) PublishTopic(message string) {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 话题模式
		"topic",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "topic failed to declare an excha"+"nge")
	//2发送信息
	err = r.channel.Publish(
		r.Exchange,
		//要设置
		r.Key,
		false,
		false,
		amqp.Publishing{
    
    
			//类型
			ContentType: "text/plain",
			//消息
			Body: []byte(message),
		})
}

//话题模式接收信息
//要注意key
//其中* 用于匹配一个单词，#用于匹配多个单词（可以是零个）
//匹配 表示匹配imooc.* 表示匹配imooc.hello,但是imooc.hello.one需要用imooc.#才能匹配到
func (r *RabbitMQ) RecieveTopic() {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 话题模式
		"topic",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "failed to declare an exchange")
	//2试探性创建队列，创建队列
	q, err := r.channel.QueueDeclare(
		"", //随机生产队列名称
		false,
		false,
		true,
		false,
		nil,
	)
	r.failOnErr(err, "Failed to declare a queue")
	//绑定队列到exchange中
	err = r.channel.QueueBind(
		q.Name,
		//在pub/sub模式下，这里的key要为空
		r.Key,
		r.Exchange,
		false,
		nil,
	)
	//消费消息
	message, err := r.channel.Consume(
		q.Name,
		"",
		true,
		false,
		false,
		false,
		nil,
	)
	forever := make(chan bool)
	go func() {
    
    
		for d := range message {
    
    
			log.Printf("Received a message:%s,", d.Body)
		}
	}()
	fmt.Println("退出请按 Ctrl+C")
	<-forever
}

//路由模式 创建RabbitMQ实例
func NewRabbitMQRouting(exchagne string, routingKey string) *RabbitMQ {
    
    
	//创建rabbitmq实例
	rabbitmq := NewRabbitMQ("", exchagne, routingKey)
	var err error
	rabbitmq.conn, err = amqp.Dial(rabbitmq.Mqurl)
	rabbitmq.failOnErr(err, "failed     to connect rabbingmq!")
	rabbitmq.channel, err = rabbitmq.conn.Channel()
	rabbitmq.failOnErr(err, "failed to open a channel")
	return rabbitmq
}

//路由模式发送信息
func (r *RabbitMQ) PublishRouting(message string) {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 广播类型
		"direct",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "failed to declare an excha"+"nge")
	//发送信息
	err = r.channel.Publish(
		r.Exchange,
		//要设置
		r.Key,
		false,
		false,
		amqp.Publishing{
    
    
			//类型
			ContentType: "text/plain",
			//消息
			Body: []byte(message),
		})
}

//路由模式接收信息
func (r *RabbitMQ) RecieveRouting() {
    
    
	//尝试创建交换机，不存在创建
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.Exchange,
		//交换机类型 广播类型
		"direct",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "failed to declare an excha"+"nge")
	//2试探性创建队列，创建队列
	q, err := r.channel.QueueDeclare(
		"", //随机生产队列名称
		false,
		false,
		true,
		false,
		nil,
	)
	r.failOnErr(err, "Failed to declare a queue")
	//绑定队列到exchange中
	err = r.channel.QueueBind(
		q.Name,
		//在pub/sub模式下，这里的key要为空
		r.Key,
		r.Exchange,
		false,
		nil,
	)
	//消费消息
	message, err := r.channel.Consume(
		q.Name,
		"",
		true,
		false,
		false,
		false,
		nil,
	)
	forever := make(chan bool)
	go func() {
    
    
		for d := range message {
    
    
			log.Printf("Received a message:%s,", d.Body)
		}
	}()
	fmt.Println("退出请按 Ctrl+C")
	<-forever
}

//简单模式Step:2、简单模式下生产代码
func (r *RabbitMQ) PublishSimple(message string) {
    
    
	//1、申请队列，如果队列存在就跳过，不存在创建
	//优点：保证队列存在，消息能发送到队列中
	_, err := r.channel.QueueDeclare(
		//队列名称
		r.QueueName,
		//是否持久化
		false,
		//是否为自动删除 当最后一个消费者断开连接之后，是否把消息从队列中删除
		false,
		//是否具有排他性 true表示自己可见 其他用户不能访问
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		//额外数据
		nil,
	)
	if err != nil {
    
    
		fmt.Println(err)
	}

	//2.发送消息到队列中
	r.channel.Publish(
		//默认的Exchange交换机是default,类型是direct直接类型
		r.Exchange,
		//要赋值的队列名称
		r.QueueName,
		//如果为true，根据exchange类型和routkey规则，如果无法找到符合条件的队列那么会把发送的消息返回给发送者
		false,
		//如果为true,当exchange发送消息到队列后发现队列上没有绑定消费者，则会把消息还给发送者
		false,
		//消息
		amqp.Publishing{
    
    
			//类型
			ContentType: "text/plain",
			//消息
			Body: []byte(message),
		})
}

func (r *RabbitMQ) ConsumeSimple() {
    
    
	//1、申请队列，如果队列存在就跳过，不存在创建
	//优点：保证队列存在，消息能发送到队列中
	_, err := r.channel.QueueDeclare(
		//队列名称
		r.QueueName,
		//是否持久化
		false,
		//是否为自动删除 当最后一个消费者断开连接之后，是否把消息从队列中删除
		false,
		//是否具有排他性
		false,
		//是否阻塞
		false,
		//额外数据
		nil,
	)
	if err != nil {
    
    
		fmt.Println(err)
	}
	//接收消息
	msgs, err := r.channel.Consume(
		r.QueueName,
		//用来区分多个消费者
		"",
		//是否自动应答
		true,
		//是否具有排他性
		false,
		//如果设置为true,表示不能同一个connection中发送的消息传递给这个connection中的消费者
		false,
		//队列是否阻塞
		false,
		nil,
	)
	if err != nil {
    
    
		fmt.Println(err)
	}
	forever := make(chan bool)

	//启用协程处理
	go func() {
    
    
		for d := range msgs {
    
    
			//实现我们要处理的逻辑函数
			log.Printf("Received a message:%s", d.Body)
			//fmt.Println(d.Body)
		}
	}()

	log.Printf("【*】warting for messages, To exit press CCTRAL+C")
	<-forever
}

func (r *RabbitMQ) ConsumeWorker(consumerName string) {
    
    
	//1、申请队列，如果队列存在就跳过，不存在创建
	//优点：保证队列存在，消息能发送到队列中
	_, err := r.channel.QueueDeclare(
		//队列名称
		r.QueueName,
		//是否持久化
		false,
		//是否为自动删除 当最后一个消费者断开连接之后，是否把消息从队列中删除
		false,
		//是否具有排他性
		false,
		//是否阻塞
		false,
		//额外数据
		nil,
	)
	if err != nil {
    
    
		fmt.Println(err)
	}
	//接收消息
	msgs, err := r.channel.Consume(
		r.QueueName,
		//用来区分多个消费者
		consumerName,
		//是否自动应答
		true,
		//是否具有排他性
		false,
		//如果设置为true,表示不能同一个connection中发送的消息传递给这个connection中的消费者
		false,
		//队列是否阻塞
		false,
		nil,
	)
	if err != nil {
    
    
		fmt.Println(err)
	}
	forever := make(chan bool)

	//启用协程处理
	go func() {
    
    
		for d := range msgs {
    
    
			//实现我们要处理的逻辑函数
			log.Printf("%s Received a message:%s", consumerName, d.Body)
			//fmt.Println(d.Body)
		}
	}()

	log.Printf("【*】warting for messages, To exit press CCTRAL+C")
	<-forever
}

② Test code

1. simple simple mode

consumer.go

func main() {
    
    
	//消费者
	rabbitmq := RabbitMQ.NewRabbitMQSimple("ziyiSimple")
	rabbitmq.ConsumeSimple()
}

producer.go

func main() {
    
    
	//Simple模式 生产者
	rabbitmq := RabbitMQ.NewRabbitMQSimple("ziyiSimple")
	for i := 0; i < 5; i++ {
    
    
		time.Sleep(time.Second * 2)
		rabbitmq.PublishSimple(fmt.Sprintf("%s %d", "hello", i))
	}
}

2. worker mode

consumer.go

func main() {
    
    
	/*
		worker模式无非就是多个消费者去同一个队列中消费消息
	*/
	//消费者1
	rabbitmq1 := RabbitMQ.NewRabbitMQSimple("ziyiWorker")
	go rabbitmq1.ConsumeWorker("consumer1")
	//消费者2
	rabbitmq2 := RabbitMQ.NewRabbitMQSimple("ziyiWorker")
	rabbitmq2.ConsumeWorker("consumer2")
}

producer.go

func main() {
    
    
	//Worker模式 生产者
	rabbitmq := RabbitMQ.NewRabbitMQSimple("ziyiWorker")
	for i := 0; i < 100; i++ {
    
    
		//time.Sleep(time.Second * 2)
		rabbitmq.PublishSimple(fmt.Sprintf("%s %d", "hello", i))
	}
}

3. publish/subscribe mode

consumer.go:

func main() {
    
    
	//消费者
	rabbitmq := RabbitMQ.NewRabbitMQPubSub("" + "newProduct")
	rabbitmq.RecieveSub()
}

producer.go

func main() {
    
    
	//订阅模式发送者
	rabbitmq := RabbitMQ.NewRabbitMQPubSub("" + "newProduct")
	for i := 0; i <= 20; i++ {
    
    
		rabbitmq.PublishPub("订阅模式生产第" + strconv.Itoa(i) + "条数据")
		fmt.Println(i)
		time.Sleep(1 * time.Second)
	}
}

4. router mode

consumer.go

func main() {
    
    
	//消费者
	rabbitmq := RabbitMQ.NewRabbitMQRouting("exZi", "imooc_one")
	rabbitmq.RecieveRouting()
}

producer.go

func main() {
    
    
	//路由模式生产者
	imoocOne := RabbitMQ.NewRabbitMQRouting("exZi", "imooc_one")
	imoocTwo := RabbitMQ.NewRabbitMQRouting("exZi", "imooc_two")

	for i := 0; i <= 10; i++ {
    
    
		imoocOne.PublishRouting("hello imooc one!" + strconv.Itoa(i))
		imoocTwo.PublishRouting("hello imooc two!" + strconv.Itoa(i))
		time.Sleep(1 * time.Second)
		fmt.Println(i)
	}
}

5. topic mode

consumer.go

func main() {
    
    
	/*
		星号井号代表通配符
		星号代表多个单词,井号代表一个单词
		路由功能添加模糊匹配
		消息产生者产生消息,把消息交给交换机
		交换机根据key的规则模糊匹配到对应的队列,由队列的监听消费者接收消息消费
	*/
	//Topic消费者
	//rabbitmq := RabbitMQ.NewRabbitMQTopic("exImoocTopic", "#") //匹配所有的key:topic88和topic99
	rabbitmq := RabbitMQ.NewRabbitMQTopic("exImoocTopic", "imooc.topic88.three") //只匹配topic88的
	rabbitmq.RecieveTopic()
}

producer.go

func main() {
    
    
	//Topic模式生产者
	imoocOne := RabbitMQ.NewRabbitMQTopic("exImoocTopic", "imooc.topic88.three")
	imoocTwo := RabbitMQ.NewRabbitMQTopic("exImoocTopic", "imooc.topic99.four")

	for i := 0; i <= 10; i++ {
    
    
		imoocOne.PublishTopic("hello imooc topic three!" + strconv.Itoa(i))
		imoocTwo.PublishTopic("hello imooc topic four!" + strconv.Itoa(i))
		time.Sleep(1 * time.Second)
		fmt.Println(i)
	}
}

2 Kafka

2.1 Basic concepts

Kafka is distributed, and all its components borker (server server cluster), producer (message production), and consumer (message consumer) can be distributed.
The producer sends data to the broker, and these messages are stored in the Kafka server, and then the consumer initiates a request to the Kafka server to consume the data.
In this process, the kafka server is like a middleman who helps you keep the data. So the kafka server can also be called broker (direct translation of broker can mean middleman or broker).

In the production of messages, an identification topic can be used to distinguish, and partitions can be made; each partition is a sequential, immutable message queue, and can be continuously added.
Provides high throughput for both publish and subscribe. It is understood that Kafka can produce about 250,000 messages (50 MB) per second and process 550,000 messages (110 MB) per second.
The status of the message being processed is maintained on the consumer side, not on the server side. Can automatically balance when it fails

Reference: https://blog.csdn.net/lingfy1234/article/details/122900348

• Application Scenario
  • monitor
  • message queue
  • stream processing
  • log aggregation
  • persistent log
• basic concept
  • topic: topic
  • broker: kafka service cluster, the published messages are stored in a group of servers, called kafka cluster. Each server in the cluster is a broker
  • partition: partition, topic physical grouping
  • message: message, each producer can publish some messages to a topic topic

1. The producer obtains the partition leader information from the Kafka cluster
2. The producer sends the message to the leader
3. The leader writes the message to the local disk
4. The follower pulls the message data from the leader
5. After the follower writes the message to the local disk Send ACK to the leader
6. The leader sends an ACK to the producer after receiving all the follower's ACKs

2.2 Common patterns

①Peer-to-point mode: Train station taxis snatch passengers

The sender sends the message to the message queue, and the consumer consumes it. If there are multiple consumers, they will consume competitively. That is to say, for a certain message, only one consumer can "grab" it. It is similar to the scene of a taxi snatching passengers at the gate of a train station.

②Publish and subscribe mode: no competition between groups, but competition within groups

Consumers subscribe to the corresponding topic (topic), and only those who subscribe to the corresponding topic consumer will receive the message.

For example:

• There are many kinds of milk, such as Bright Milk, Hope Milk, etc. Only if you subscribe to Bright Milk, the milkman will deliver Bright Milk to the corresponding location, and you will have the opportunity to consume this milk

注意: In order to improve the consumption capacity of consumers, the concept of consumer group is introduced in Kafka. It is equivalent to: there will be no competition between different consumer groups because they subscribe to different topics. But there is competition within the consumer group.

For example:

• Taxi drivers in Chengdu and Xiamen form their respective consumer groups.
• Taxi drivers in Chengdu only take people from Chengdu, and taxi drivers in Xiamen only take people from Xiamen. (So ​​their two consumer groups are not in competition)
• There is competition among taxi drivers in Chengdu. (The consumer group is a competitive relationship)

2.3 docker-compose deployment

 vim docker-compose.yml

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:6.2.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      #KAFKA_ADVERTISED_LISTENERS后面改为自己本地宿主机的ip，例如我本地mac的ip为192.168.0.101
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.101:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

# 进入到docker-compose.yml所在目录，执行下面命令
docker-compose up -d
# 查看部署结果,状态为up表明部署成功
docker-compose ps 

2.4 Code operation

# 1. 创建对应topic
docker-compose exec kafka kafka-topics --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server 192.168.0.101:9092

# 2. 查看topic列表
docker-compose exec kafka kafka-topics --list --zookeeper zookeeper:2181

①producer.go

package main

import (
	"fmt"

	"github.com/IBM/sarama"
)

// 基于sarama第三方库开发的kafka client

func main() {
    
    
	config := sarama.NewConfig()
	config.Producer.RequiredAcks = sarama.WaitForAll          // 发送完数据需要leader和follow都确认
	config.Producer.Partitioner = sarama.NewRandomPartitioner // 新选出一个partition
	config.Producer.Return.Successes = true                   // 成功交付的消息将在success channel返回

	// 构造一个消息
	msg := &sarama.ProducerMessage{
    
    }
	msg.Topic = "web_log"
	msg.Value = sarama.StringEncoder("this is a test log")
	// 连接kafka
	client, err := sarama.NewSyncProducer([]string{
    
    "localhost:9092"}, config)
	if err != nil {
    
    
		fmt.Println("producer closed, err:", err)
		return
	}
	defer client.Close()
	// 发送消息
	pid, offset, err := client.SendMessage(msg)
	if err != nil {
    
    
		fmt.Println("send msg failed, err:", err)
		return
	}
	fmt.Printf("pid:%v offset:%v\n", pid, offset)
}

②consumer.go

package main

import (
	"fmt"

	"github.com/IBM/sarama"
)

// kafka consumer

func main() {
    
    
	consumer, err := sarama.NewConsumer([]string{
    
    "localhost:9092"}, nil)
	if err != nil {
    
    
		fmt.Printf("fail to start consumer, err:%v\n", err)
		return
	}
	partitionList, err := consumer.Partitions("web_log") // 根据topic取到所有的分区
	if err != nil {
    
    
		fmt.Printf("fail to get list of partition:err%v\n", err)
		return
	}
	fmt.Println(partitionList)
	for partition := range partitionList {
    
     // 遍历所有的分区
		// 针对每个分区创建一个对应的分区消费者
		pc, err := consumer.ConsumePartition("web_log", int32(partition), sarama.OffsetNewest)
		if err != nil {
    
    
			fmt.Printf("failed to start consumer for partition %d,err:%v\n", partition, err)
			return
		}
		defer pc.AsyncClose()
		// 异步从每个分区消费信息
		go func(sarama.PartitionConsumer) {
    
    
			for msg := range pc.Messages() {
    
    
				fmt.Printf("Partition:%d Offset:%d Key:%v Value:%v", msg.Partition, msg.Offset, msg.Key, string(msg.Value))
			}
		}(pc)
	}
	//演示时使用
	select {
    
    }
}

③Operation effect