sarama introduction
As of the current time, there are two main packages for golang to operate kafka on github:
Shopify/sarama starts
confluentinc/confluent-kafka-go starts
saram is written in pure go language, confluent-kafka-go is an api that wraps c
rhythm
To connect to Kafka in Go language, use a third-party library: github.com/Shopify/sarama.
download and install
go get github.com/Shopify/sarama
Matters needing attention
The zstd compression algorithm has been added to the version after sarama v1.20, which requires the use of cgo. When compiling on the Windows platform, it will prompt an error similar to the following:
github.com/DataDog/zstd
exec: "gcc":executable file not found in %PATH%
So please use v1.19 version of sarama on Windows platform.
Kafka sends and receives messages
package main
import (
"fmt"
"github.com/Shopify/sarama"
)
// 基于sarama第三方库开发的kafka client
func main() {
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll // 发送完数据需要leader和follow都确认
config.Producer.Partitioner = sarama.NewRandomPartitioner // 新选出一个partition
config.Producer.Return.Successes = true // 成功交付的消息将在success channel返回
// 构造一个消息
msg := &sarama.ProducerMessage{
}
msg.Topic = "web_log"
msg.Value = sarama.StringEncoder("this is a test log")
// 连接kafka
client, err := sarama.NewSyncProducer([]string{
"192.168.1.7:9092"}, config)
if err != nil {
fmt.Println("producer closed, err:", err)
return
}
defer client.Close()
// 发送消息
pid, offset, err := client.SendMessage(msg)
if err != nil {
fmt.Println("send msg failed, err:", err)
return
}
fmt.Printf("pid:%v offset:%v\n", pid, offset)
}
consumer
package main
import (
"fmt"
"github.com/Shopify/sarama"
)
// kafka consumer
func main() {
consumer, err := sarama.NewConsumer([]string{
"127.0.0.1:9092"}, nil)
if err != nil {
fmt.Printf("fail to start consumer, err:%v\n", err)
return
}
partitionList, err := consumer.Partitions("web_log") // 根据topic取到所有的分区
if err != nil {
fmt.Printf("fail to get list of partition:err%v\n", err)
return
}
fmt.Println(partitionList)
for partition := range partitionList {
// 遍历所有的分区
// 针对每个分区创建一个对应的分区消费者
pc, err := consumer.ConsumePartition("web_log", int32(partition), sarama.OffsetNewest)
if err != nil {
fmt.Printf("failed to start consumer for partition %d,err:%v\n", partition, err)
return
}
defer pc.AsyncClose()
// 异步从每个分区消费信息
go func(sarama.PartitionConsumer) {
for msg := range pc.Messages() {
fmt.Printf("Partition:%d Offset:%d Key:%v Value:%v", msg.Partition, msg.Offset, msg.Key, msg.Value)
}
}(pc)
}
}
Configuration file read
Now, we edit the my.ini file and enter the following ( partially from Grafana ).
# possible values : production, development
app_mode = development
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
data = /home/git/grafana
[server]
# Protocol (http or https)
protocol = http
# The http port to use
http_port = 9999
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = true
Great, next we need to write the main.go file to manipulate the configuration file we just created.
package main
import (
"fmt"
"os"
"gopkg.in/ini.v1"
)
func main() {
cfg, err := ini.Load("my.ini")
if err != nil {
fmt.Printf("Fail to read file: %v", err)
os.Exit(1)
}
// 典型读取操作，默认分区可以使用空字符串表示
fmt.Println("App Mode:", cfg.Section("").Key("app_mode").String())
fmt.Println("Data Path:", cfg.Section("paths").Key("data").String())
// 我们可以做一些候选值限制的操作
fmt.Println("Server Protocol:",
cfg.Section("server").Key("protocol").In("http", []string{
"http", "https"}))
// 如果读取的值不在候选列表内，则会回退使用提供的默认值
fmt.Println("Email Protocol:",
cfg.Section("server").Key("protocol").In("smtp", []string{
"imap", "smtp"}))
// 试一试自动类型转换
fmt.Printf("Port Number: (%[1]T) %[1]d\n", cfg.Section("server").Key("http_port").MustInt(9999))
fmt.Printf("Enforce Domain: (%[1]T) %[1]v\n", cfg.Section("server").Key("enforce_domain").MustBool(false))
// 差不多了，修改某个值然后进行保存
cfg.Section("").Key("app_mode").SetValue("production")
cfg.SaveTo("my.ini.local")
}
Running the program, we can see the following output:
$ go run main.go
App Mode: development
Data Path: /home/git/grafana
Server Protocol: http
Email Protocol: smtp
Port Number: (int) 9999
Enforce Domain: (bool) true
$ cat my.ini.local
# possible values : production, development
app_mode = production
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
data = /home/git/grafana
...