Use go-zero to quickly build microservices that support high concurrency

0. Why is it difficult to do a good job of microservices?

To do a good job of microservices, we need to understand and master a lot of knowledge points, from several dimensions:

Basic functional level
1. Concurrency control & current limiting to avoid service being overwhelmed by burst traffic
2. Service registration and service discovery to ensure dynamic detection of nodes that increase or decrease
3. Load balancing needs to distribute traffic according to the node's capacity
4. Timeout control to avoid useless work on requests that have timed out
5. Fuse design, fast failure, guarantee the recovery ability of failed nodes
High-level functional level
1. Request authentication to ensure that each user can only access his own data
2. Link tracking, used to understand the entire system and quickly locate problems with specific requests
3. Logs for data collection and problem location
4. Observability, no optimization without measurement

For each of these points, we need to use a long space to describe its principles and implementation, so it is very difficult for our back-end developers to master and implement these knowledge points into the business system. , But we can rely on the framework system that has been verified by large traffic. The go-zero microservice framework was born for this.

In addition, we always adhere to the philosophy that tools are greater than conventions and documents . We want to reduce the mental burden of developers as much as possible, devote our energy to the code that generates business value, and reduce the writing of repetitive code, so we developed goctltools.

Below I use short-chain microservices to demonstrate the process of quickly creating microservices through go-zero . After walking through it, you will find that writing microservices is so easy!

1. What is short-chain service?

The short-chain service is to convert a long URL URL into a short URL string through program calculations and other methods.

This short chain service is written to demonstrate the process of go-zero building a complete microservice as a whole. The algorithm and implementation details are simplified as much as possible, so this is not a high-level short chain service.

2. Short-chain microservice architecture diagram

Here we separate shorten and expand into two microservices. It does not mean that a remote call needs to be split into one microservice, but just for the simplest demonstration of multiple microservices.
The latter redis and mysql are also shared, but in the real project, each microservice should use its own database as much as possible, and the data boundary should be clear

3. Preparation

Install etcd, mysql, redis
Prepare goctl tool
https://github.com/tal-tech/go-zero/releasesDownload the latest version directly , and automatic updates will be added later
- You can also compile from the source code, in any directory, the purpose is to compile the goctl tool

1. git clone https://github.com/tal-tech/go-zero 2. tools/goctlCompile the goctl tool in the directory go build goctl.go 3. Put the generated goctl $PATHunder to ensure that the goctl command can be run

Create working directoryshorturl
shorturlPerform go mod init shorturlinitialization in the directorygo.mod

4. Write API Gateway code

Generated shorturl.apiand edited by goctl . For brevity, the beginning of the file is removed info. The code is as follows:

type (
  shortenReq struct {
      url string `form:"url"`
  }

  shortenResp struct {
      shortUrl string `json:"shortUrl"`
  }
)

type (
  expandReq struct {
      key string `form:"key"`
  }

  expandResp struct {
      url string `json:"url"`
  }
)

service shorturl-api {
  @server(
      handler: ShortenHandler
  )
  get /shorten(shortenReq) returns(shortenResp)

  @server(
      handler: ExpandHandler
  )
  get /expand(expandReq) returns(expandResp)
}

The usage of type is the same as that of go. Service is used to define api requests such as get/post/head/delete. The explanation is as follows:

service shorturl-api {This line defines the service name
@serverPart of it is used to define the attributes used on the server side
handlerThe name of the server handler is defined
get /shorten(shortenReq) returns(shortenResp)Defines the routing, request parameters, return parameters, etc. of the get method
- Use goctl to generate API Gateway code

goctl api go -api shorturl.api -dir api

The generated file structure is as follows:

.
├── api
│   ├── etc
│   │   └── shorturl-api.yaml         // 配置文件
│   ├── internal
│   │   ├── config
│   │   │   └── config.go             // 定义配置
│   │   ├── handler
│   │   │   ├── expandhandler.go      // 实现expandHandler
│   │   │   ├── routes.go             // 定义路由处理
│   │   │   └── shortenhandler.go     // 实现shortenHandler
│   │   ├── logic
│   │   │   ├── expandlogic.go        // 实现ExpandLogic
│   │   │   └── shortenlogic.go       // 实现ShortenLogic
│   │   ├── svc
│   │   │   └── servicecontext.go     // 定义ServiceContext
│   │   └── types
│   │       └── types.go              // 定义请求、返回结构体
│   └── shorturl.go                   // main入口定义
├── go.mod
├── go.sum
└── shorturl.api

Start the API Gateway service, listening on port 8888 by default

go run api/shorturl.go -f api/etc/shorturl-api.yaml

Test the API Gateway service

curl -i "http://localhost:8888/shorten?url=http://www.xiaoheiban.cn"

The return is as follows:

HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 27 Aug 2020 14:31:39 GMT
Content-Length: 15

{"shortUrl":""}

You can see that our API Gateway did nothing but returned a null value. Next, we will implement business logic in the rpc service

Can be modified internal/svc/servicecontext.goto pass service dependencies (if needed)
internal/logicThe corresponding file under the implementation logic can be modified
The goctlcode can be called by generating APIs in various client languages
At this point, you can already generate client code through goctl for parallel development of client classmates, supporting multiple languages, see the documentation for details

5. Write a shorter rpc service

rpc/shortenWrite shorten.protofiles in the directory

Proto file template can be generated by command

goctl rpc template -o shorten.proto

The contents of the modified file are as follows:

syntax = "proto3";

package shorten;

message shortenReq {
    string url = 1;
}

message shortenResp {
    string key = 1;
}

service shortener {
    rpc shorten(shortenReq) returns(shortenResp);
}

By goctlgenerating the code rpc, the rpc/shortencommand in the directory

goctl rpc proto -src shorten.proto

The file structure is as follows:

rpc/shorten
├── etc
│   └── shorten.yaml               // 配置文件
├── internal
│   ├── config
│   │   └── config.go              // 配置定义
│   ├── logic
│   │   └── shortenlogic.go        // rpc业务逻辑在这里实现
│   ├── server
│   │   └── shortenerserver.go     // 调用入口, 不需要修改
│   └── svc
│       └── servicecontext.go      // 定义ServiceContext，传递依赖
├── pb
│   └── shorten.pb.go
├── shorten.go                     // rpc服务main函数
├── shorten.proto
└── shortener
    ├── shortener.go               // 提供了外部调用方法，无需修改
    ├── shortener_mock.go          // mock方法，测试用
    └── types.go                   // request/response结构体定义

It can be run directly as follows:

$ go run shorten.go -f etc/shorten.yaml
Starting rpc server at 127.0.0.1:8080...

etc/shorten.yamlThe configuration of the listening port can be modified in the file

6. Write expand rpc service

rpc/expandWrite expand.protofiles in the directory

Proto file template can be generated by command

goctl rpc template -o expand.proto

The contents of the modified file are as follows:

syntax = "proto3";

package expand;

message expandReq {
    string key = 1;
}

message expandResp {
    string url = 1;
}

service expander {
    rpc expand(expandReq) returns(expandResp);
}

By goctlgenerating the code rpc, the rpc/expandcommand in the directory

goctl rpc proto -src expand.proto

The file structure is as follows:

rpc/expand
├── etc
│   └── expand.yaml                // 配置文件
├── expand.go                      // rpc服务main函数
├── expand.proto
├── expander
│   ├── expander.go                // 提供了外部调用方法，无需修改
│   ├── expander_mock.go           // mock方法，测试用
│   └── types.go                   // request/response结构体定义
├── internal
│   ├── config
│   │   └── config.go              // 配置定义
│   ├── logic
│   │   └── expandlogic.go         // rpc业务逻辑在这里实现
│   ├── server
│   │   └── expanderserver.go      // 调用入口, 不需要修改
│   └── svc
│       └── servicecontext.go      // 定义ServiceContext，传递依赖
└── pb
    └── expand.pb.go

Modify the port etc/expand.yamlinside ListenOnas 8081it is 8080already shortenoccupied by the service

Run after modification, as follows:

$ go run expand.go -f etc/expand.yaml
Starting rpc server at 127.0.0.1:8081...

etc/expand.yamlThe configuration of the listening port can be modified in the file

7. Modify the API Gateway code to call the shorten/expand rpc service

Modify the configuration file shorter-api.yamland add the following content

Shortener:
  Etcd:
    Hosts:
      - localhost:2379
    Key: shorten.rpc
Expander:
  Etcd:
    Hosts:
      - localhost:2379
    Key: expand.rpc

Automatically discover available shorten/expand services through etcd

Amend internal/config/config.goas follows, add shorten/expand service dependency

type Config struct {
  rest.RestConf
  Shortener rpcx.RpcClientConf     // 手动代码
  Expander  rpcx.RpcClientConf     // 手动代码
}

Amend internal/svc/servicecontext.goas follows:

type ServiceContext struct {
  Config    config.Config
  Shortener rpcx.Client                                 // 手动代码
  Expander  rpcx.Client                                 // 手动代码
}

func NewServiceContext(config config.Config) *ServiceContext {
  return &ServiceContext{
      Config:    config,
      Shortener: rpcx.MustNewClient(config.Shortener),    // 手动代码
      Expander:  rpcx.MustNewClient(config.Expander),     // 手动代码
  }
}

Passing dependencies between different business logics through ServiceContext

Amend internal/logic/expandlogic.goas follows:

type ExpandLogic struct {
  ctx context.Context
  logx.Logger
  expander rpcx.Client            // 手动代码
}

func NewExpandLogic(ctx context.Context, svcCtx *svc.ServiceContext) ExpandLogic {
  return ExpandLogic{
      ctx:    ctx,
      Logger: logx.WithContext(ctx),
      expander: svcCtx.Expander,    // 手动代码
  }
}

func (l *ExpandLogic) Expand(req types.ExpandReq) (*types.ExpandResp, error) {
  // 手动代码开始
  resp, err := expander.NewExpander(l.expander).Expand(l.ctx, &expander.ExpandReq{
      Key: req.Key,
  })
  if err != nil {
      return nil, err
  }

  return &types.ExpandResp{
      Url: resp.Url,
  }, nil
  // 手动代码结束
}

Increase the expanderdependency on the service, and realize the short chain recovery to the url by calling expanderthe Expandmethod

Amend internal/logic/shortenlogic.goas follows:

type ShortenLogic struct {
  ctx context.Context
  logx.Logger
  shortener rpcx.Client             // 手动代码
}

func NewShortenLogic(ctx context.Context, svcCtx *svc.ServiceContext) ShortenLogic {
  return ShortenLogic{
      ctx:    ctx,
      Logger: logx.WithContext(ctx),
      shortener: svcCtx.Shortener,    // 手动代码
  }
}

func (l *ShortenLogic) Shorten(req types.ShortenReq) (*types.ShortenResp, error) {
  // 手动代码开始
  resp, err := shortener.NewShortener(l.shortener).Shorten(l.ctx, &shortener.ShortenReq{
      Url: req.Url,
  })
  if err != nil {
      return nil, err
  }

  return &types.ShortenResp{
      ShortUrl: resp.Key,
  }, nil
  // 手动代码结束
}

Added shortenerdependent services, and by calling shortenerthe Shortenrealization url to transform short-chain approach

At this point, API Gateway has been modified. Although there are many codes posted, only a small part of the mid-term changes are made. In order to facilitate understanding of the context, I posted the complete code, and then deal with CRUD+cache

8. Define the database table structure and generate CRUD+cache code

Create the rpc/model directory under shorturl:mkdir -p rpc/model
Write the sql file to create the shorturl table in the rpc/model directory shorturl.sql, as follows:

CREATE TABLE `shorturl`
(
  `shorten` varchar(255) NOT NULL COMMENT 'shorten key',
  `url` varchar(255) NOT NULL COMMENT 'original url',
  PRIMARY KEY(`shorten`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Create DB and table

create database gozero;

source shorturl.sql;

rpc/modelExecute the following command in the directory to generate CRUD+cache code, which -cmeans to useredis cache

goctl model mysql ddl -c -src shorturl.sql -dir .

You can also use datasourcecommands instead ddlto specify that the database link is generated directly from the schema

The generated file structure is as follows:

rpc/model
├── shorturl.sql
├── shorturlmodel.go              // CRUD+cache代码
└── vars.go                       // 定义常量和变量

9. Modify the shorten/expand rpc code to call the crud+cache code

Modify rpc/expand/etc/expand.yaml, add the following content:

DataSource: root:@tcp(localhost:3306)/gozero
Table: shorturl
Cache:
  - Host: localhost:6379

You can use multiple redis as cache, support redis single point or redis cluster

Amend rpc/expand/internal/config.goas follows:

type Config struct {
  rpcx.RpcServerConf
  DataSource string             // 手动代码
  Table      string             // 手动代码
  Cache      cache.CacheConf    // 手动代码
}

Added mysql and redis cache configuration

Amend rpc/expand/internal/svc/servicecontext.goas follows:

type ServiceContext struct {
  c     config.Config
  Model *model.ShorturlModel   // 手动代码
}

func NewServiceContext(c config.Config) *ServiceContext {
  return &ServiceContext{
      c:     c,
      Model: model.NewShorturlModel(sqlx.NewMysql(c.DataSource), c.Cache, c.Table), // 手动代码
  }
}

Amend rpc/expand/internal/logic/expandlogic.goas follows:

type ExpandLogic struct {
  ctx context.Context
  logx.Logger
  model *model.ShorturlModel          // 手动代码
}

func NewExpandLogic(ctx context.Context, svcCtx *svc.ServiceContext) *ExpandLogic {
  return &ExpandLogic{
      ctx:    ctx,
      Logger: logx.WithContext(ctx),
      model:  svcCtx.Model,             // 手动代码
  }
}

func (l *ExpandLogic) Expand(in *expand.ExpandReq) (*expand.ExpandResp, error) {
  // 手动代码开始
  res, err := l.model.FindOne(in.Key)
  if err != nil {
      return nil, err
  }

  return &expand.ExpandResp{
      Url: res.Url,
  }, nil
  // 手动代码结束
}

Modify rpc/shorten/etc/shorten.yaml, add the following content:

DataSource: root:@tcp(localhost:3306)/gozero
Table: shorturl
Cache:
  - Host: localhost:6379

You can use multiple redis as cache, support redis single point or redis cluster

Amend rpc/shorten/internal/config.goas follows:

type Config struct {
  rpcx.RpcServerConf
  DataSource string            // 手动代码
  Table      string            // 手动代码
  Cache      cache.CacheConf   // 手动代码
}

Added mysql and redis cache configuration

Amend rpc/shorten/internal/svc/servicecontext.goas follows:

type ServiceContext struct {
  c     config.Config
  Model *model.ShorturlModel   // 手动代码
}

func NewServiceContext(c config.Config) *ServiceContext {
  return &ServiceContext{
      c:     c,
      Model: model.NewShorturlModel(sqlx.NewMysql(c.DataSource), c.Cache, c.Table), // 手动代码
  }
}

Amend rpc/shorten/internal/logic/shortenlogic.goas follows:

const keyLen = 6

type ShortenLogic struct {
  ctx context.Context
  logx.Logger
  model *model.ShorturlModel          // 手动代码
}

func NewShortenLogic(ctx context.Context, svcCtx *svc.ServiceContext) *ShortenLogic {
  return &ShortenLogic{
      ctx:    ctx,
      Logger: logx.WithContext(ctx),
      model:  svcCtx.Model,             // 手动代码
  }
}

func (l *ShortenLogic) Shorten(in *shorten.ShortenReq) (*shorten.ShortenResp, error) {
  // 手动代码开始，生成短链接
  key := hash.Md5Hex([]byte(in.Url))[:keyLen]
  _, err := l.model.Insert(model.Shorturl{
      Shorten: key,
      Url:     in.Url,
  })
  if err != nil {
      return nil, err
  }

  return &shorten.ShortenResp{
      Key: key,
  }, nil
  // 手动代码结束
}

At this point, the code modification is complete, and I have marked the code that was manually modified.

10. Full call demo

shorten api call

~ curl -i "http://localhost:8888/shorten?url=http://www.xiaoheiban.cn"

The return is as follows:

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 29 Aug 2020 10:49:49 GMT
Content-Length: 21

{"shortUrl":"f35b2a"}

expand api call

curl -i "http://localhost:8888/expand?key=f35b2a"

The return is as follows:

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 29 Aug 2020 10:51:53 GMT
Content-Length: 34

{"url":"http://www.xiaoheiban.cn"}

11. Benchmark

Because writing depends on the writing speed of mysql, it is equivalent to pressing mysql, so the pressure test only tested the expand interface, which is equivalent to reading from mysql and using the cache. In shorten.lua, 100 randomly obtained from db Hot key to generate pressure test request

It can be seen that the qps of 30,000+ can be reached on my MacBook Pro.

12. Summary

We have always emphasized that tools are greater than conventions and documents .

go-zero is not just a framework, but a technical system based on framework + tools that simplifies and standardizes the entire microservice construction.

While keeping it simple, we also encapsulated the complexity of microservice governance into the framework as much as possible, which greatly reduced the mental burden of developers and enabled rapid business development.

The code generated by go-zero+goctl contains various components of microservice governance, including: concurrency control, adaptive fuse, adaptive load shedding, automatic cache control, etc., which can be easily deployed to carry huge traffic.