JavaScript Object Notation (JSON) is a standard protocol for sending and receiving structured information. Among similar protocols, JSON is not the only one standard protocol. XML (§7.14), ASN.1, and Google's Protocol Buffers are all similar protocols and have their own characteristics, but JSON is the most widely used one for reasons of brevity, readability, and popularity.
The Go language has good support for encoding and decoding these standard formats, and is supported by packages such as encoding/json, encoding/xml, encoding/asn1 in the standard library (Annotation: Protocol Buffers are supported by github.com/golang/ protobuf package), and such packages have similar API interfaces. In this section, we will give an overview of the usage of the important encoding/json packages.
JSON is the Unicode text encoding for various types of values in JavaScript—strings, numbers, booleans, and objects. It can represent the basic data types in Chapter 3 and the aggregate data types in this chapter such as arrays, slices, structures, and maps in an efficient and readable way.
The basic JSON types are numbers (decimal or scientific notation), boolean values (true or false), strings, where strings are sequences of Unicode characters enclosed in double quotes, and backslash escapes similar to Go are supported feature, but JSON uses \Uhhhhescaped numbers to represent a UTF-16 encoding. representation; and UTF-16 also has big-endian and little-endian issues), not the rune type of the Go language.
These primitive types can be recursively combined via JSON's array and object types. A JSON array is an ordered sequence of values, written in square brackets and separated by commas; a JSON array can be used to encode Go arrays and slices. A JSON object is a string-to-value map, written as a series of name:value pairs, enclosed in curly braces and separated by commas; the JSON object type can be used to encode the Go language map type (the key type is a string ) and structures. E.g:
boolean true
number -273.15
string "She said \"Hello, BF\""
array ["gold", "silver", "bronze"]
object {"year": 1980,
"event": "archery",
"medals": ["gold", "silver", "bronze"]}
Consider an app that collects various movie reviews and provides feedback features. Its Movie data type and a typical list of values representing movies are shown below. (In the struct declaration, the string literal following the Year and Color members is the struct member Tag; we'll explain what this does later.)
gopl.io/ch4/movie
type Movie struct {
Title string
Year int `json:"released"`
Color bool `json:"color,omitempty"`
Actors []string
}
var movies = []Movie{
{Title: "Casablanca", Year: 1942, Color: false,
Actors: []string{"Humphrey Bogart", "Ingrid Bergman"}},
{Title: "Cool Hand Luke", Year: 1967, Color: true,
Actors: []string{"Paul Newman"}},
{Title: "Bullitt", Year: 1968, Color: true,
Actors: []string{"Steve McQueen", "Jacqueline Bisset"}},
// ...
}
Such data structures are particularly well suited to the JSON format, and it's easy to convert between the two. The process of converting a movie-like structure slice in Go to JSON is called marshaling. Marshaling is done by calling the json.Marshal function:
data, err := json.Marshal(movies)
if err != nil {
log.Fatalf("JSON marshaling failed: %s", err)
}
fmt.Printf("%s\n", data)
The Marshal function returns an encoded bytes slice containing a long string with no whitespace indentation; we wrap it for display:
[{"Title":"Casablanca","released":1942,"Actors":["Humphrey Bogart","Ingr
id Bergman"]},{"Title":"Cool Hand Luke","released":1967,"color":true,"Ac
tors":["Paul Newman"]},{"Title":"Bullitt","released":1968,"color":true,"
Actors":["Steve McQueen","Jacqueline Bisset"]}]
This compact representation, while containing all the information, is difficult to read. To produce a human-readable format, another json.MarshalIndent function will produce neatly indented output. The function has two additional string arguments for the prefix of each line of output and the indentation of each level:
data, err := json.MarshalIndent(movies, "", " ")
if err != nil {
log.Fatalf("JSON marshaling failed: %s", err)
}
fmt.Printf("%s\n", data)
The above code will produce output like this:
[
{
"Title": "Casablanca",
"released": 1942,
"Actors": [
"Humphrey Bogart",
"Ingrid Bergman"
]
},
{
"Title": "Cool Hand Luke",
"released": 1967,
"color": true,
"Actors": [
"Paul Newman"
]
},
{
"Title": "Bullitt",
"released": 1968,
"color": true,
"Actors": [
"Steve McQueen",
"Jacqueline Bisset"
]
}
]
When encoding, by default the member names of Go structs are used as JSON objects (through the reflect technique, which we will discuss in Section 12.6). Only exported struct members are encoded, which is why we chose to start the member names with an uppercase letter.
Careful readers may have noticed that the members of the Year name become released after encoding, and the Color members become color starting with lowercase letters after encoding. This is caused by the structure member Tag. A struct member Tag is a string of meta information associated with the member at compile time:
Year int `json:"released"`
Color bool `json:"color,omitempty"`
The member Tag of the structure can be any string literal value, but it is usually a sequence of key:"value" key-value pairs separated by spaces; because the value contains double quote characters, the member Tag is generally used as a native string literal value. form writing. The value corresponding to the key name at the beginning of json is used to control the encoding and decoding behavior of the encoding/json package, and other packages under encoding/... also follow this convention. The first part of the json corresponding value in the member Tag is used to specify the name of the JSON object, for example, the TotalCount member in the Go language corresponds to the total_count object in JSON. The Tag of the Color member also brings an additional omitempty option, which means that the JSON object will not be generated when the Go language structure member is empty or zero value (here false is zero value). Sure enough, Casablanca was a black-and-white film and didn't output the Color members.
The inverse operation of encoding is decoding, which corresponds to decoding JSON data into the data structure of Go language, which is generally called unmarshaling in Go language, and is completed by the json.Unmarshal function. The following code decodes the movie data in JSON format into a struct slice with only the Title member. By defining appropriate Go language data structures, we can selectively decode the members of interest in JSON. When the Unmarshal function call returns, the slice will be filled with the value containing only the Title information, other JSON members will be ignored.
var titles []struct{ Title string }
if err := json.Unmarshal(data, &titles); err != nil {
log.Fatalf("JSON unmarshaling failed: %s", err)
}
fmt.Println(titles) // "[{Casablanca} {Cool Hand Luke} {Bullitt}]"
Many web services provide JSON interfaces that send JSON-formatted requests and return JSON-formatted information through HTTP interfaces. To illustrate this, we demonstrate a similar usage through Github's issue query service. First, we need to define the appropriate types and constants:
gopl.io/ch4/github
// Package github provides a Go API for the GitHub issue tracker.
// See https://developer.github.com/v3/search/#search-issues.
package github
import "time"
const IssuesURL = "https://api.github.com/search/issues"
type IssuesSearchResult struct {
TotalCount int `json:"total_count"`
Items []*Issue
}
type Issue struct {
Number int
HTMLURL string `json:"html_url"`
Title string
State string
User *User
CreatedAt time.Time `json:"created_at"`
Body string // in Markdown format
}
type User struct {
Login string
HTMLURL string `json:"html_url"`
}
As before, each struct member name is declared to start with an uppercase letter, even if the corresponding JSON object name is lowercase. Because some JSON member names are not the same as Go structure member names, the Go language structure member Tag is required to specify the corresponding JSON name. Similarly, the same processing needs to be done when decoding, the GitHub service returns a lot more information than we define.
The SearchIssues function makes an HTTP request and then decodes the returned JSON-formatted results. Because the query conditions provided by the user may contain special characters like ?and &, in order to avoid conflicts with the URL, we use url.QueryEscape to escape the special characters in the query.
gopl.io/ch4/github
package github
import (
"encoding/json"
"fmt"
"net/http"
"net/url"
"strings"
)
// SearchIssues queries the GitHub issue tracker.
func SearchIssues(terms []string) (*IssuesSearchResult, error) {
q := url.QueryEscape(strings.Join(terms, " "))
resp, err := http.Get(IssuesURL + "?q=" + q)
if err != nil {
return nil, err
}
// We must close resp.Body on all execution paths.
// (Chapter 5 presents 'defer', which makes this simpler.)
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("search query failed: %s", resp.Status)
}
var result IssuesSearchResult
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
resp.Body.Close()
return nil, err
}
resp.Body.Close()
return &result, nil
}
In the earlier examples, we used the json.Unmarshal function to decode a JSON-formatted string into a byte slice. But in this example, we use the stream-based decoder json.Decoder, which can decode JSON data from an input stream, although this is not required. As you might expect, there is also a json.Encoder encoding object for the output stream.
We call the Decode method to populate the variable. There are multiple ways to format the structure here. Below is the simplest one, printing each issue with a fixed width, but in the next section we'll see how templates can be used to output complex formats.
gopl.io/ch4/issues
// Issues prints a table of GitHub issues matching the search terms.
package main
import (
"fmt"
"log"
"os"
"gopl.io/ch4/github"
)
func main() {
result, err := github.SearchIssues(os.Args[1:])
if err != nil {
log.Fatal(err)
}
fmt.Printf("%d issues:\n", result.TotalCount)
for _, item := range result.Items {
fmt.Printf("#%-5d %9.9s %.55s\n",
item.Number, item.User.Login, item.Title)
}
}
Specify search criteria with command line parameters. The following command is to query the problems related to JSON decoding in the Go language project, and the results returned by the query:
$ go build gopl.io/ch4/issues
$ ./issues repo:golang/go is:open json decoder
13 issues:
#5680 eaigner encoding/json: set key converter on en/decoder
#6050 gopherbot encoding/json: provide tokenizer
#8658 gopherbot encoding/json: use bufio
#8462 kortschak encoding/json: UnmarshalText confuses json.Unmarshal
#5901 rsc encoding/json: allow override type marshaling
#9812 klauspost encoding/json: string tag not symmetric
#7872 extempora encoding/json: Encoder internally buffers full output
#9650 cespare encoding/json: Decoding gives errPhase when unmarshalin
#6716 gopherbot encoding/json: include field name in unmarshal error me
#6901 lukescott encoding/json, encoding/xml: option to treat unknown fi
#6384 joeshaw encoding/json: encode precise floating point integers u
#6647 btracey x/tools/cmd/godoc: display type kind of each named type
#4237 gjemiller encoding/base64: URLEncoding padding is optional
GitHub's web service interface https://developer.github.com/v3/ contains more features.
Exercise 4.10: Modify the issues program to classify the issues according to the time, such as less than a month, less than a year, and more than a year.
Exercise 4.11: Write a tool that allows users to create, read, update, and close issues on GitHub at the command line, automatically opening the user's default editor for entering text messages when necessary.
Exercise 4.12: The popular web comic service xkcd also provides a JSON interface. For example, a request to https://xkcd.com/571/info.0.json will return a detailed description of the 571 number that many people love. Download each link (only once) and create an offline index. Write an xkcd tool that uses these offline indexes to print the URLs of comics that match the search terms entered on the command line.
Exercise 4.13: Use the JSON service interface of the Open Movie Database, which allows you to retrieve and download movie titles and corresponding poster images from https://omdbapi.com/ . Write a poster tool to download the corresponding poster through the movie name entered on the command line.