I. Introduction
I wrote an article a few days ago, which is to sync.Mapget goroutinethe returned results and then process them, but I always feel that the solution is average and not very good. After all, channelhe is the appointed prince, so it is better to use it channelbetter.
golang controls the number of goroutines and obtains processing results
2. Misunderstandings and actual combat code
1. Misunderstandings
Bloggers use concurrency channelthat is generally used for control , so the structure is relatively simple, and they take it for granted that it is only suitable for storing simple structures. It is not convenient to process complex function processing results , which is a thousand miles away .goroutinechannelchannelchannel
2. Actual combat code
注：以下为脱敏的伪代码
代码含义概览：
（1）通过channel控制goroutine数量
（2）定义channel结构体，结构体里面可以根据需求嵌套其他结构体，实现我们想要的复杂结构
（3）每次循环获取go函数处理结果，处理每次循环结果
//EsBulkDataRes 可以定义复杂的结构，或者嵌套结构体
type EsBulkDataRes struct {
SucceededNums int `json:"succeeded_nums"`
FailedNums int `json:"failed_nums"`
ErrData string `json:"err_data"`
//TestDoc *TestDoc
}
type TestDoc struct {
ID string `json:"_id,omitempty"`
Source map[string]interface{
} `json:"_source,omitempty"` //map里面可以存储复杂的结构
}
func testChannel() {
//定义要获取的返回值，成功数量，失败数量，失败id集合
succeededNums, failedNums := 0, 0
var errData string
DealRes := make(chan EsBulkDataRes)
defer func() {
close(DealRes)
}()
wg := sync.WaitGroup{
}
//控制goroutine数量，保证同时只有10个goroutine
chan1 := make(chan struct{
}, 10)
ctx := context.Background()
for {
//业务逻辑
// ....
//goroutine加速，chan1写满，则阻塞。等待之前的goroutine释放才能继续循环
chan1 <- struct{
}{
}
wg.Add(1)
go func(ctx context.Context, targetIndexName string, esClient *elastic.Client) {
defer func() {
if err1 := recover(); err1 != nil {
//产生了panic异常
log.Errorf(ctx, "%s go panic! err:(+%v)", logPreFix, err1)
}
//执行完毕再释放channel
<-chan1
return
}()
bulkRequest := esClient.Bulk()
//ES使用bulk方法批量刷新数据
bulkResByAssetInfo := new(EsBulkDataRes)
bulkResByAssetInfo, err = BulkEsDataByAssetInfo(ctx, bulkRequest, targetIndexName)
if err != nil {
log.Errorf(ctx, "%s BulkEsDataByAssetInfo error (%+v) ,test:(+%v)", logPreFix, err, string_util.TransferToString(fileUnitList))
return
}
//执行结果写入channel
DealRes <- *bulkResByAssetInfo
//执行完毕再释放channel
<-chan1
}(ctx, targetIndexName, esClient)
//goroutine 执行结束，读取channel累加结果
//读取channel，也是为了方便下一个 goroutine 的写入。没读的话，会阻塞
select {
case d, ok := <-DealRes:
if !ok {
continue
}
//累加结果
succeededNums += d.SucceededNums
failedNums += d.FailedNums
errData += d.ErrData
case <-ctx.Done():
return
}
}
wg.Wait()
//打印结果
fmt.Println("成功数量:", succeededNums)
fmt.Println("失败数量:", failedNums)
fmt.Println("失败id:", errData)
}
3. Channel controls multiple goroutine serial
This part is an interview question encountered by the blogger before, talking about multiple gofunctions, each gofunction depends on the processing result of the previous step, how to achieve serialization. The pseudo code is given here, just refer to it below.
//定义channel结构
type chanStruct struct {
Res1 int64
}
//每个函数中重新给channel赋值
func test1(chan1 chan chanStruct) {
res1 := new(chanStruct)
fmt.Println("test1")
res1.Res1 = 2
chan1 <- *res1
return
}
func test2(chan2 chan chanStruct) {
res2 := new(chanStruct)
fmt.Println("test2")
res2.Res1 = 3
chan2 <- *res2
return
}
func test3(chan3 chan chanStruct) {
fmt.Printf("test3,chanStruct:(%+v)", chan3)
return
}
//https://segmentfault.com/q/1010000041024462/
//无缓冲通道读写都必须在协程里，否则会阻塞。有缓冲通道则可以不需要都准备好，读或者写可以写在当前线程里而不会阻塞。
func main() {
chan0 := make(chan chanStruct, 1)
//这里使用了"golang.org/x/sync/errgroup" 这个包，博主个人实验，在此处非必需
g, ctx := errgroup.WithContext(context.Background())
chan0 <- chanStruct{
Res1: 1,
}
fmt.Println("write chan success!")
//errgroup控制并发，并获取goroutine返回的错误(此处没用到)
g.Go(func() error {
for {
select {
//注意这里，由于每次我们都读出来了channel，因此需要在函数中给channel赋值
//保证能触发下一个函数
case d, ok := <-chan0:
fmt.Println("d:", d)
if ok {
if d.Res1 == 1 {
go test1(chan0)
} else if d.Res1 == 2 {
go test2(chan0)
} else if d.Res1 == 3 {
go test3(chan0)
fmt.Println("end")
return nil
}
}
case <-ctx.Done():
return ctx.Err()
}
}
})
//errgroup的Wait类似于sync.withGroup的Wait()方法，等待goroutine执行结束
if err := g.Wait(); err != nil {
log.Fatal(err)
}
}
Four. Postscript
How do we make sure that the code we write is better? How to define this good? Is it just for functionality or for elegance? The above are the questions asked by the big guy when the blogger chats with him.
For us developers, it is absolutely okay to take the realization of requirements as the first goal, but the code quality also needs to be continuously improved. How to improve it, of course, is to look at the code of the boss! Where does the boss have the most code, of course github!
The blogger recently read https://github.com/olivere/esdiffthe code of the big guy, only to realize that he was narrow-minded before, and marveled at the wonderful writing style of the big guy. This is just an unknown open source project. I don’t know how magnificent the well-known projects such as k8s and etcd will be! come on!
end