An article that takes you to uncover the veil of go mod

go module

After the release of go1.16, the go module autochanged onto , which means that in subsequent development, it is recommended to use the go module mode instead of the gopath mode for development.

In the past, I also mostly developed golang in the go module mode, but I am not familiar with it so far 别人是这样做的，我跟着做就是了. There will be these questions:

• What do the contents defined in the go mod file represent;
• In addition to common requireand occasional replacekeywords , exclude, retract(1.16) What are these keywords and how are they used;
• What is the grammatical format of the go mod file? I don’t seem to understand the grammatical format besides following other people’s writing.
• github.com/tal-tech/go-zero v1.1.5What do , github.com/antlr/antlr4 v0.0.0-20210105212045-464bcbc32de2
  , google.golang.org/protobuf v1.25.0 // indirectand other formats represent, and why some are
  // indirectmodified with ;
• Why is there a go.sum under go.mod, and what is its function;
• …

I don't know how many people, like me, have a negligible understanding of go modules.

Recently, with these doubts, I went to study the official reference manual, and these doubts were solved.

project

Before officially entering the module introduction, it is necessary to first understand the relationship between project and module. I believe that students who have developed Android or Java have a very good understanding of module. Generally speaking, a project can be composed of multiple modules, and a module can be used as The independent
project is referenced by other projects as dependencies. The following golang demoproject contains fooand bartwo modules

demo
├── bar
│   └── go.mod
└── foo
    └── go.mod

module introduction

go module (hereinafter referred to as: module, module, project module)
is a collection of released versions of packages in golang. It is a way for Go to manage dependencies, similar to Gradle in Android and Maven in Java. Of course, their management form It must be very different, but the purpose is the same, to manage dependencies.

In go.mod, it contains the module path of the main module, module dependencies and its associated information (version, etc.). If a project module needs to be
go module modedeveloped in (module mode), the root directory of the project module must contain go.modthe file .

module path (module path)

The module path is the name in a project module, declared with modulethe command , and it is also the prefix of the package import in the project module. Let's take a look at demo/foothe module path below:

$ cat demo/foo/go.mod

module github.com/foo

go 1.16

require github.com/tal-tech/go-zero

github.com/foofooIt is the module path of the main module module, github.com/tal-tech/go-zerowhich is also the module path, and they are also
the prefix of the package import in foo. I added a Echofunction , and then main.gocalled it in . Let's observe the prefix of its package import

directory tree

foo
├── base
│   └── base.go
├── go.mod
└── main.go

main.go

package main

import "github.com/foo/base" // github.com/foo 为 module path
import "fmt"

func main() {
	msg := base.Echo("go-zero")
	fmt.Println(msg)
}

The main function of the module path is to describe what the function of a project module is and where it can be found. Therefore, the components of the module path include

• repo path
• repo folder
• Version

Its manifestations are as follows: { {.repo_url}}/{ {.nameOfDir}}/{ {.version}}, example: github.com/tal-tech/go-zero/v2, github.com/tal-tech
which clearly informs the path repoof , go-zerowhich is the folder of repo, v2which is the version number. Generally, the v1version is generally not written by default, and only v1when it is greater than , it needs to be used to distinguish.

version number

The version number here refers to the version of the dependent module in the go.mod file, which { {.version}}is . The in the following example v1.1.5is the version number of the module dependency mentioned this time.

module github.com/foo

go 1.16

require github.com/tal-tech/go-zero v1.1.5

Version number composition and rules

The version number consists of major version(major version), minor version(minor version), patch version(revision version);

• major version: After the content in the module has been changed backwards, the version will be upgraded. When the version number is upgraded, minor versionand patch versionmust be reset to zero;
• minor version: This version number will be upgraded after new features are released (features) or a backward compatible content change is made. When this version number is upgraded, it must be reset to zero patch version;
• patch version: This version number can be upgraded when there is a bug fix or function optimization, and this version number can also be changed when there is a pre-release release requirement

Examples: v0.0.0, v1.2.3,v1.2.10-pre

If a version major versionis 0or patch-versionhas a version suffix (such as: pre), this version is considered unstable, such as v0.2.0, v1.5.0-pre, v1.1.3-beta
and more about version semantic definitions, please refer to "Semantic Versioning 2.0.0"

In addition to this, Golang can also use some tags and branches to represent a certain version, such as: github.com/tal-tech/go-zero 39540e21d249e91f89d96d015a6e3795cfb2be44
, github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21d249,
github.com/tal-tech/go-zero@master

Among them, github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21d249this version is called (pseudo-version) in golang Pseudo-versions
, which does not fully follow the version rules above. The pseudo-version consists of three parts:

• version base prefix ( vX.Y.Z-0or vX.0.0), such asv1.1.6-0
• Timestamp: the creation timestamp of the revision, such as20210303091609
• revision identifier, such as39540e21d249

Depending on the base prefix, the pseudo-version has three forms:

• vX.0.0-yyyymmddhhmmss-abcdefabcdef: used when there is no releaseversion
• vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef: used when the base version is a pre-release version
• vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdefrelease: vX.Y.Z
  used when the version is similar to , such github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21d249as releasethe version ofv1.1.5

The pseudo-version does not need to be manually entered, and it will automatically convert it to a pseudo-version when executing some go commands to obtain the code of a commit record version (revision) as a dependency

The github.com/tal-tech/go-zero v1.1.6-0.20210303091609-39540e21d249above is go get github.com/tal-tech/go-zero 39540e21d249e91f89d96d015a6e3795cfb2be44the result of automatic conversion after executing

version suffix

For forward compatibility, if is major versionupgraded to 2, the module path must specify a version suffix v2(its value remains consistent with the value major versionin ), and major versionwhen is less than 2, the version suffix is ​​not allowed.

Let's look at an example. I used this library in the demo/foomodule project miniRedis, and the library major versionhas been upgraded to 2. What if I reference the following version in go.mod?

module github.com/foo

go 1.16

// 正确引入
// require github.com/alicebob/miniredis/v2 v2.14.1 

// 错误引入
require github.com/alicebob/miniredis v2.14.1

invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

The above is the scene when the go mod is used. Let’s take a look at v2.x.xwhen (provided that a v1.0.0 version is released first), the module path does not add a version suffix, and it will be used in another module. What effect:
the example module foo
foo project currently has a release version

• v1.0.0
• v2.0.1
• …

Projects using modulesbar

• without version suffix

fooProject directory tree

foo
├── echo
│   └── echo.go
└── go.mod

module path isgithub.com/zsh/foo

module github.com/anqiansong/foo

go 1.16

In barproject go.mod use v2.0.0version

require github.com/anqiansong/foo v2.0.0

You will find that the error content is

 require github.com/anqiansong/foo: reading https://goproxy.cn/github.com/anqiansong/foo/@v/v2.0.0.info: 404 Not Found
	server response: not found: github.com/anqiansong/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

If require github.com/zsh/foo v1.0.0it is possible.

• After adding the version suffix

fooProject directory tree

foo
├── echo
│   └── echo.go
└── go.mod

module path isgithub.com/zsh/foo/v2

module github.com/anqiansong/foo/v2

go 1.16

In barproject go.mod use v2.0.1version

require github.com/anqiansong/foo/v2 v2.0.1

barOperating normally

If major versionyou upgrade v2to , if the version is not intended to be forward compatible, and you don’t want to add a version suffix to the module path, you can +incompatibleend .
Other project reference examples arerequire github.com/zsh/foo v2.0.0+incompatible

How to parse the module in the package

The go command first searches the build list for modules prefixed with the package path. For example, if the package is imported example.com/a/band the module example.com/ais in the build list, the go command will check to see example.com/a
if the package bin . And the directory contains at least one go file, so that it can be considered package. Generation constraints should not be used for this purpose. If only one module in the build list provides the package, that module is used. go
The command reports an error if no module provides the package, or if two or more modules provide the package . mod=modThe flag instructs gothe command to try to find new modules providing missing packages, and to update go.modand go.sum. go getThe and go mod tidycommands do this automatically.

When the go command updates or obtains module dependencies, it will check the GOPROXYenvironment variable, GOPROXYwhose value is a comma-separated list of urls, or keywords direct, off,

• The specific url separated by commas is the proxy address, which will tell gothe command to initiate a connection with this value
• direct: Specify module dependencies to obtain through the version control system
• off: Indicates not trying to connect to get the module

If a specific url is GOPROXYset , assuming that gothe command is looking for one github.com/tal-tech/go-zero/zrpc, packagethe gocommand will search for the module in parallel

• github.com/tal-tech/go-zero/zrpc
• github.com/tal-tech/go-zero
• github.com/tal-tech
• github.com

If one or more of them match the content github.com/tal-tech/go-zero/zrpcthat satisfies , take the longest moduleone as the dependency. After finding the appropriate module and version, gothe command will fill in the go.mod
and go.sumfiles require. If the parsed moduleis not actively introduced by the main module , a comment will be added after the value requireof // direct
, and if none of them match, an error will be reported; if GOPROXYthere are multiple url proxies, if the previous one fails, the above steps will be executed to the subsequent proxies in turn.

go.mod file

The identification of a (module project) is to contain a text file coded as and named modulein its root directory . The content in the file is oriented . Each line contains an instruction, and each line consists of a and , like:UTF-8go.modgo.mod行关键字参数

module github.com/anqiansong/foo

go 1.16

require github.com/tal-tech/go-zero
require github.com/tal-tech/go-queue

replace go.etcd.io/etcd => go.etcd.io/etcd v0.0.0-20200402134248-51bdeb39e698

retract [v1.0.0, v1.0.1]

Of course, content with the same keyword can be separated and composed with 关键字+ block, like:

module github.com/anqiansong/foo

go 1.16

require (
    github.com/tal-tech/go-zero
    github.com/tal-tech/go-queue
)

replace go.etcd.io/etcd => go.etcd.io/etcd v0.0.0-20200402134248-51bdeb39e698

retract [v1.0.0, v1.0.1]

go.modIt is machine-writable, like executing some commands (such as: go get, go mod edit) may automatically update go.modthe file .

module components

When parsing the contents of go.modthe file , it is parsed as

• 空白符: contains spaces (U+0020), tabs (U+0009), carriage returns (U+000D) and line feeds (U+000A)
• 注释: Comments only support single-line comments//
• 标点: The punctuation marks are (, ), ,,=>
• 关键字 ： go 、 require 、 replace 、 exclude 、 retract
• 标识符: A character sequence 空白符consisting , such as module path, semantic version
• 字符串: The interpreted string wrapped in English double quotation marks "(U+0022) or <the original string wrapped in (U+0060). Such as "github/com/tal-tech/go-zero", ``

标识符字符串and go.modcan be replaced in syntax

module syntax lexical

go.modExtended Backus-Naur FormGrammars are defined via (EBNF normal form), like

GoMod = { Directive } .
Directive = ModuleDirective |
            GoDirective |
            RequireDirective |
            ExcludeDirective |
            ReplaceDirective |
            RetractDirective .

moduleinstruction

moduleThe keyword defines the module path of the main module go.mod, and there is only one modulespecified in the file.

Grammar rules:

ModuleDirective = "module" ( ModulePath | "(" newline ModulePath newline ")" newline .

Example:

module github.com/tal-tech/go-zero

goinstruction

goThe keyword moduledefines the Go language version that is expected to be used. The version must be a valid Go version (it can be understood as conforming to versionthe rules, or it can be understood as a version that has been released by Go)

goAfter the version is defined by the keyword , the compiler knows which go version should be used to compile the package. In addition, gothe keyword definition version can also be used to enable some features of gothe command , such as whether to automatically start vendoring in the version 1.14and later .

Grammar rules:

GoDirective = "go" GoVersion newline .
GoVersion = string | ident .  /* valid release version; see above */

Example:

go 1.16

requireinstruction

requireDeclare the minimum version of module dependencies. After requirespecifying the version, gorelated commands will load dependencies based on this value according to MVS rules.

goWhen looking for a dependency, if the dependency is not directly dependent on the main module, // directa comment .

Grammar rules:

RequireDirective = "require" ( RequireSpec | "(" newline { RequireSpec } ")" newline ) .
RequireSpec = ModulePath Version newline .

Example:

module github.com/tal-tech/go-zero

go 1.16

require (
    golang.org/x/crypto v1.4.5 // indirect
    golang.org/x/text v1.6.7
)

exculeinstruction

exculeThe specified version in the content will be ignored. go 1.16After , excludethe specified module will be ignored. go 1.16Before , if requirethe module of is exclude
specified , it will list and obtain the changed excludeversion.

Grammar rules:

ExcludeDirective = "exclude" ( ExcludeSpec | "(" newline { ExcludeSpec } ")" ) .
ExcludeSpec = ModulePath Version newline .

Example:

module github.com/tal-tech/go-zero

go 1.16

exclude golang.org/x/net v1.2.3

excule (
    golang.org/x/crypto v1.4.5
    golang.org/x/text v1.6.7
)

replaceinstruction

replaceThe instruction is used to replace the specified version of the module or the module with other modules or versions. If =>the left quality specifies a version, replace this version to the target content, otherwise replace all versions of the module to the target content

Grammar rules:

ReplaceDirective = "replace" ( ReplaceSpec | "(" newline { ReplaceSpec } ")" newline ")" ) .
ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline
            | ModulePath [ Version ] "=>" ModulePath Version newline .
FilePath = /* platform-specific relative or absolute file path */

Example:

replace golang.org/x/net v1.2.3 => example.com/fork/net v1.4.5

=>The content on the right can be a valid module path, or a relative or absolute path. If it is a relative or absolute path, the root directory of the path must contain go.modthe file

Example:

require github.com/foo v1.0.0

replace github.com/foo v1.0.0 => ../bar

retractCommand (new in 1.16)

retractThe content of the statement is used to mark certain versions or a certain version range (closed interval) as withdrawn. Generally, a comment needs to be written before retractthe statement When executing go getthe command, if the version retractmarked , or
in retractWithin retractofthe marked , go list -m -versionsand the version will also be hidden when getting the version through .

Grammar rules:

RetractDirective = "retract" ( RetractSpec | "(" newline { RetractSpec } ")" ) .
RetractSpec = ( Version | "[" Version "," Version "]" ) newline .

Example:

// someting wrong
retract v1.0.0

// someting wrong in range of versions => v1.0.0~v1.2.0 
retrace [v1.0.0,v1.2.0]

Let's take a look at an example. There are github.com/zsh/retractalready v1.0.0versions such as and so on. Let's add a line retractof command to mark
v1.0.0withdrawal:

// someting wrong
retract v1.0.0

Then release a version as , and then refer to the version inv1.0.1github.com/zsh/barv1.0.0

require github.com/anqiansong/retract v1.0.0

Then execute go get github.com/zsh/[email protected], no surprises, you will get a prompt containing something wrongand prompt to update to v1.0.1the information

$ go get github.com/anqiansong/[email protected]

go: warning: github.com/anqiansong/[email protected]: retracted by module author: someting wrong
go: to switch to the latest unretracted version, run:
        go get github.com/anqiansong/retract@latestgo get: downgraded github.com/anqiansong/retract v1.0.1 => v1.0.0

Get github.com/zsh/retractall module release versions

$ go list -m -versions github.com/anqiansong/retract

github.com/anqiansong/retract v1.0.1

View the results github.com/zsh/[email protected]of :

• go get

  $ go get github.com/anqiansong/[email protected]
  

  go: warning: github.com/anqiansong/[email protected]: retracted by module author: someting wrong
  go: to switch to the latest unretracted version, run:
  go get github.com/anqiansong/retract@latest
  

• go list -m -u

  $ go get github.com/anqiansong/[email protected]
  

  github.com/anqiansong/retract v1.0.0 (retracted) [v1.0.1]
  

• go list -m -versions

    $ go list -m -versions github.com/anqiansong/retract
  

  ```text
  

  github.com/anqiansong/retract v1.0.1
  ```

illustrate:

retractThe version of the main module is controlled, not the version of the dependent module.

retractOther modules of the marked version can still be referenced, but some gocommands will retracthave , as above.

automatic update

Most gocommands . go get, go mod tidycommand can be used to fix most of these problems. Additionally, -mod=mod
flags can be used on most module-aware commands ( go build, , go testetc.) to indicate that gothe command automatically fixes problems ingo.mod and .go.sum

module perception

Most go commands Module-warecan GOPATHbe run in either mode or mode. In Module-waremode , the go command uses the go.mod file to find version dependencies,
it usually loads packages from the module cache, and downloads the module if it is missing. In GOPATHmode , the go command ignores module; it looks for dependencies in vendorthe or GOPATHdirectories .

As of Go 1.16 go.mod, Module-warethe mode is enabled by default regardless of the existence of the file. In earlier versions go.mod, Module-warethe mode was enabled when a file existed in the working directory file or any parent directory.

Module-wareThe mode can be controlled via the GO111MODULE environment variable, which can be set to on, offorauto

• off: goThe associated command ignores go.modthe file and GOPATHruns in mode
• on: onor an empty string, the related command will run Module-warein mode
• auto: If there is go.moda file , it will Module-warerun in mode. In Go 1.15 and lower versions, this value is the default value.

Some go module related commands

This command must be in Module-waremode to be valid

Order	usage	Remark	example
go list -m	go list -m [-u] [-retracted] [-versions] [list flags] [modules]	View module information	go list -m all
go mod init	go mod init [module-path]	Initialize and create a go.mod file in the working directory	go mod init demo
go mod tidy	go mod tidy [-e] [-v]	Organize the go.mod file	go mode tidy
go clean -modcache	go clean [-modcache]	clear module cache	go clean -modcache

Proxy

A module proxy is GETa HTTPserver that supports a response to a request that has no queryparameters and does not even require specific headerinformation, even if the value is a fixed file system site (eg: file:// URL).

HTTPThe response status code of the module proxy must contain 200(OK), 3xx, 4xx, 5xx, 4xx, 5xxare considered to be response errors, 404and 410
indicate that all module requests are unavailable. Note that the contentType of the wrong response should be set to text/plain, and the character set is utf-8or us-ascii.

URLs

goThe command can connect to a proxy server or a version control system by reading GOPROXYthe environment variable configuration, GOPROXYaccepting multiple url values ​​separated by a comma (,) or vertical bar (|), when
separated by an English comma (,), only the response status When the code is 404 or 410, the subsequent proxy address will be tried. If it is separated by a vertical bar (|), any error (including timeout) in http will skip to try the subsequent proxy address. Can also be an director offkeyword.

The following table is a path that a proxy address must implement and has a request response (that is, a proxy server must support the implementation of routing)

• $baseFor the proxy server address, such as: https://goproxy.cn
• $moduleFor module path, such as: github.com/tal-tech/go-zero
• $versionfor the module version

path	describe	example	example result
$base/$module/@v/list	Return a list of known versions of the given module in plain text, one per line. This list should not include fake versions	curl -X GET https://goproxy.cn/github.com/tal-tech/go-zero/@v/list	v1.0.0 v1.0.1 v1.0.2 v1.0.3 v1.0.4 ...

$base/$module/c/$version.info returns metadata about a specific version of a module in json format. The response must be a JSON object corresponding to the following Go data structure:

  type Info struct {
      Version string    // version string
      Time    time.Time // commit time
  }
  

</td>
<td>curl -X GET https://goproxy.cn/github.com/tal-tech/go-zero/@v/v1.1.5.info</td>
<td>
<pre>{

“Version”: “v1.1.5”,
“Time”: “2021-03-02T03:02:57Z”
}

$base/$module/@v/$version.mod returns the information in the specified version of go.mod /td> curl -X GET https://goproxy.cn/github.com/tal-tech/go-zero/ @v/v1.1.5.mod

module github.com/tal-tech/go-zero

go 1.14

require (
github.com/ClickHouse/clickhouse-go v1.4.3
github.com/DATA-DOG/go-sqlmock v1.4.1
github.com/alicebob/miniredis/v2 v2.14.1
github.com/antlr/antlr4 v0.0.0-20210105212045-464bcbc32de2
…
)

$base/$module/@v/$version.zip returns the zip file wget of the specified version of go module https://goproxy.cn/github.com/tal-tech/go-zero/@v/v1.1.5. zip v1.1.5.zip $base/$module/@latest Returns metadata about the latest known version of the module in json format curl -X GET https://goproxy.cn/github.com/tal-tech/go- zero/@latest

{
  "Version": "v1.1.5",
  "Time": "2021-03-02T03:02:57Z"
}

When obtaining the latest version of the module, gothe relevant commands will first request $base/$module/@v/listthe address . If no suitable version is found,
the request $base/$module/@latestwill be obtained and matched to see if it is satisfied. goThe relevant commands will be sorted by releaseversion , pre-releaseversion, and pseudoversion.

The $base/$module/$version.modand $base/$module/$version.zipaddresses must be provided, because these information can be used go.sumfor .

The downloaded content of the go module is generally stored in $GOPATH/pkg/mod/cache/downloadthe path , including those downloaded by the version control system.

direct

GOPROXYIf directthe value is set go, the module resource will be downloaded from the version control system ( git, svnetc.) when the relevant command is executed. Of course, two other environment
variables GOPRIVATEand GONOPROXYcooperation are required. For more information about environment variables, please refer to here

module file size constraints

There are a number of restrictions on the content of module zipfiles . These constraints ensure that compressed files can be extracted safely and consistently on a wide range of platforms.

A module can be up to 500MiB (whether in compressed or uncompressed form)

go.modFile requirements are displayed within 16MiB

go.sum

There may be a file go.sumnamed , which is directly attached to go.mod. When executing gorelated commands to obtain the module,
it will zipcheck go.sumwhether the values ​​in the file and are consistent.

go.sumA value in consists of a module path, versionand a hashvalue, such as:

github.com/anqiansong/retract v1.0.1 h1:jxcsUM/6tvxM7p14/XMeZPFbql5KAAZJfFqiHG+YKxA=

reference documents

• 《Semantic Versioning 2.0.0》
• ["Go Module Official Documentation"](