Golang: use sarama to deliver tasks to kafka

Get into the habit of writing together! This is the 11th day of my participation in the "Nuggets Daily New Plan · April Update Challenge",Click to view event details。

[使用sarama向kafka投递任务]()
    [前言]()
    [sarama的下载]()
    [启动kafka环境]()
    [启动一个kafka消费者接受任务]()
    [go代码编写]()
    [步骤总览]()
    [完整代码]()
    [运行代码，查看效果]()
复制代码

Use sarama to deliver tasks to kafka

foreword

When I understand my zookeeper and kafka, and start them successfully. You can try to deliver tasks to the kafka process. The programming language used here is golang, and the golang library involved is: github.com/Shopify/sarama. saramais a golang client library that listens to apache kafka.

download of sarama

download commandgo get github.com/Shopify/sarama

If there is an error, you can try to change the go agent, in the command line:

go env -W GOPROXY=https://goproxy.cn

If it is other errors, try to paste the error message Baidu.

Start the kafka environment

0. zookeeper

   Enter the zookeeper decompression directory, run the corresponding system, run the zkServerwin system, and run bin/zkServer.cmdthe linux system zkServer.sh. Kafka needs to rely on zookeeper to start, so zookeeper must be started first

1. kafka

Also enter the decompression directory of kafka and run the corresponding service to specify the service configuration file. Run under win (under the root directory of the decompression): bin/windows/kafka-server-start.bat config/server.properties, the files under linux are inbin/zookeeper-server-start.sh config/server.properties

Start a kafka consumer to accept tasks

Enter the kafka bindirectory, run the corresponding kafka-console-consumerfiles according to the system, win is in bin/windows/, linux is in bin/. The operation needs to bring the 连接地址sum topic号, the following example, the topic isshopping

It will block after startup, and then we can deliver the task.

go code writing

Overview of steps:

0. Define kafka configuration
1. Build the message structure
2. connect to kafka
3. Send a message

Define kafka configuration

The configuration class can be obtained via sarama.NewConfig(). There are two main things for initialization, 1. Select the partition, 2. Set the return of the channel after successful delivery. Remember to import sarama.

/*记得引入sarama。 import("github.com/Shopify/sarama") */
config := sarama.NewConfig()
// 1. 初始化生产者配置
config.Producer.RequiredAcks = sarama.WaitForAll
// 选择分区
config.Producer.Partitioner = sarama.NewRandomPartitioner
// 成功交付的信息
config.Producer.Return.Successes = true
复制代码

Build the message structure

The same sarama has defined the structure, and the Topic and Value (specific values) need to be assigned, as follows:

// 2. 构造一个消息，结构体类型
msg := &sarama.ProducerMessage{
    Topic: "shopping",
    Value: sarama.StringEncoder("20220411happy03"),
}
复制代码

connect to kafka

Use sarama.NewSyncProducerto pass in the configuration file. Open connection and close connection must be paired

// 3. 连接kafka
client, err := sarama.NewSyncProducer([]string{"127.0.0.1:9092"}, config)
if err != nil {
    fmt.Println(err)
}
defer client.Close()
复制代码

Send a message (delivery task)

client.SendMessage(msg)There are 3 return values, the first is the partition where the data is located, the second is the offset of the data, and the third is the error message, no error is returnednil

// 4. 发送消息
partition, offset, err := client.SendMessage(msg)
fmt.Println("所在分区:")
fmt.Println(partition)
fmt.Println("数据偏移量:")
fmt.Println(offset)
if err != nil {
    fmt.Println(err)
}
复制代码

full code

package main
​
import (
    "fmt"
    "github.com/Shopify/sarama"
)
​
func main() {
    config := sarama.NewConfig()
    // 1. 初始化生产者配置
    config.Producer.RequiredAcks = sarama.WaitForAll
    // 选择分区
    config.Producer.Partitioner = sarama.NewRandomPartitioner
    // 成功交付的信息
    config.Producer.Return.Successes = true
​
    // 2. 构造一个消息，结构体类型
    msg := &sarama.ProducerMessage{
        Topic: "shopping",
        Value: sarama.StringEncoder("20220411happy02"),
    }
​
    // 3. 连接kafka
    client, err := sarama.NewSyncProducer([]string{"127.0.0.1:9092"}, config)
    if err != nil {
        fmt.Println(err)
    }
    defer client.Close()
​
    // 4. 发送消息
    partition, offset, err := client.SendMessage(msg)
    fmt.Println("partition:")
    fmt.Println(partition)
    fmt.Println("offset:")
    fmt.Println(offset)
    if err != nil {
        fmt.Println(err)
    }
}
​
复制代码

Run the code to see the effect

end