Detailed explanation of Go network programming

An Introduction to the Internet Protocol

1.1 Internet layered model

The logical implementation of the Internet is divided into several layers. Each floor has its own function, and like a building, each floor is supported by the floor below. What the user touches is only the uppermost layer, and the lower layers will not be felt at all. To understand the Internet, you need to understand the functions of each layer from the bottom up.

As shown in the figure above, the Internet will be divided into different layers according to different models, but no matter what model is used to divide, the higher the layer is, the closer it is to the user, and the lower the layer is, the closer it is to the hardware. In software development, what we use most is the model that divides the Internet into five layers in the above figure.

Next, we introduce each layer layer by layer from bottom to top.

physical layer

Our computer needs to communicate with the external Internet, we need to connect the computer to the network first, we can use twisted pair, optical fiber, radio waves and other methods. This is called the "real physical layer," and it's the physical means of connecting computers together. It mainly specifies some electrical characteristics of the network, and is responsible for transmitting electrical signals of 0 and 1.

data link layer

Pure 0 and 1 have no meaning, so our users will give them some specific meanings and stipulate the way of interpreting electrical signals: for example: How many electrical signals count as a group? What is the significance of each signal bit? This is the function of the "data link layer". It is above the "physical layer" and determines the grouping method and representative meaning of 0 and 1 transmitted by the physical layer. In the early days, each company had its own way of grouping electrical signals. Gradually, a protocol called "Ethernet" (Ethernet) took the lead.

Ethernet stipulates that a group of electrical signals constitutes a data packet, called a "frame". Each frame is divided into two parts: header (Head) and data (Data). Among them, "header" contains some description items of the data packet, such as sender, receiver, data type, etc.; "data" is the specific content of the data packet. The length of the "header" is fixed at 18 bytes. The length of "data", the shortest is 46 bytes, the longest is 1500 bytes. Therefore, the entire "frame" is as short as 64 bytes and as long as 1518 bytes. If the data is very long, it must be divided into multiple frames for transmission.

So, how are the sender and receiver identified? Ethernet stipulates that all devices connected to the network must have a "network card" interface. Packets must be sent from one NIC to another. The address of the network card is the sending address and receiving address of the data packet, which is called the MAC address. When each network card leaves the factory, it has a unique MAC address in the world, with a length of 48 binary bits, usually represented by 12 hexadecimal numbers. The first 6 hexadecimal numbers are the manufacturer's serial number, and the last 6 are the serial number of the network card of the manufacturer. With the MAC address, the network card and the path of the data packet can be located.

We will obtain the MAC address of the recipient through the ARP protocol. After having the MAC address, how to accurately send the data to the recipient? In fact, Ethernet here adopts a very "primitive" method. It does not send the data packet to the receiver accurately, but sends it to all computers in the network, so that each computer can read the "header" of the packet. , find the MAC address of the receiver, and then compare it with its own MAC address. If the two are the same, accept the packet for further processing, otherwise discard the packet. This way of sending is called "broadcasting".

Network layer

According to the rules of the Ethernet protocol, we can rely on the MAC address to send data out. In theory, relying on the MAC address, your computer's network card can find the network card of a computer in another corner of the world, but this approach has a major flaw that Ethernet uses broadcasting to send data packets, and all members hand in hand. Packet", not only inefficient, but also the data sent can only be limited to the subnet where the sender is located. That is to say, if the two computers are not in the same subnet, the broadcast cannot be transmitted. This design is reasonable and necessary, because it is unrealistic if every computer on the Internet will receive all data packets sent and received on the Internet.

Therefore, a way must be found to distinguish which MAC addresses belong to the same subnet and which do not. If it is the same subnet, it will be sent by broadcast, otherwise it will be sent by "routing". This led to the birth of the "network layer". Its function is to introduce a new set of addresses, so that we can distinguish whether different computers belong to the same subnet. This set of addresses is called "network address", or "URL" for short.

After the emergence of the "network layer", each computer has two types of addresses, one is the MAC address and the other is the network address. There is no connection between the two addresses. The MAC address is bound to the network card, and the network address is assigned by the network administrator. The network address helps us determine which subnet the computer is on, and the MAC address sends the packet to the target NIC in that subnet. Therefore, it can be inferred logically that the network address must be processed first, and then the MAC address.

The protocol that specifies the network address is called the IP protocol. The address it defines is called an IP address. Currently, the fourth version of the IP protocol, referred to as IPv4, is widely used. This version of IPv4 stipulates that a network address consists of 32 binary digits. We usually use decimal numbers divided into four segments to represent IP addresses, from 0.0.0.0 to 255.255.255.255.

The data sent according to the IP protocol is called an IP packet. The IP packet is also divided into two parts: "header" and "data": the "header" part mainly includes information such as version, length, IP address, etc., and the "data" part is the specific content of the IP data packet. The length of the "header" part of the IP data packet is 20 to 60 bytes, and the total length of the entire data packet is at most 65535 bytes.

transport layer

With the MAC address and IP address, we can already establish communication between any two hosts on the Internet. But the problem is that there will be many programs on the same host that need to use the network to send and receive data. For example, both programs, QQ and browser, need to connect to the Internet and send and receive data. How do we distinguish which program a certain data packet belongs to? In other words, we also need a parameter to indicate which program (process) this data packet is for. This parameter is called "port" (port), which is actually the number of each program that uses the network card. Each data packet is sent to a specific port of the host, so different programs can get the data they need.

"Port" is an integer between 0 and 65535, exactly 16 binary bits. Ports from 0 to 1023 are occupied by the system, and users can only choose ports greater than 1023. With IP and port, we can uniquely determine a program on the Internet, and then realize program communication between networks.

We have to add port information in the packet, which requires a new protocol. The simplest implementation is called the UDP protocol, and its format is almost in front of the data, plus the port number. A UDP packet is also composed of two parts: "header" and "data": the "header" part mainly defines the sending port and the receiving port, and the "data" part is the specific content. The UDP packet is very simple. The "header" part has only 8 bytes in total, and the total length does not exceed 65,535 bytes, which fits into an IP packet.

The advantage of the UDP protocol is that it is relatively simple and easy to implement, but the disadvantage is that the reliability is poor. Once the data packet is sent, it is impossible to know whether the other party has received it. In order to solve this problem and improve network reliability, the TCP protocol was born. The TCP protocol can ensure that data will not be lost. Its disadvantages are complex process, difficult implementation, and high resource consumption. There is no limit to the length of a TCP packet, and it can be infinitely long in theory. However, in order to ensure the efficiency of the network, the length of a TCP packet usually does not exceed the length of an IP packet to ensure that a single TCP packet does not need to be divided.

application layer

The application receives the data from the "transport layer", and the next step is to unpack the data. Since the Internet is an open architecture with various data sources, the data format of the communication must be specified in advance, otherwise the receiver will not be able to obtain the actual data content sent. The role of the "application layer" is to specify the data format used by the application, such as the common Email, HTTP, FTP and other protocols on top of our TCP protocol, these protocols constitute the application layer of the Internet protocol.

As shown in the figure below, the HTTP data of the sender will add the header information of each layer of the protocol in turn during the transmission process of the HTTP data through the Internet, and the receiver will unpack the data according to the protocol after receiving the data packet.

Two socket programming

Socket is the process communication mechanism of BSD UNIX. It is also commonly called "socket". It is used to describe the IP address and port, and is the handle of a communication chain. Socket can be understood as the API of the TCP/IP network, which defines many functions or routines, and programmers can use them to develop applications on the TCP/IP network. Applications running on the computer usually send requests to the network or respond to network requests through "sockets".

socket diagram

Socket is an intermediate software abstraction layer for communication between the application layer and the TCP/IP protocol family. In the design mode, Socket is actually a facade mode, which hides the complex TCP/IP protocol family behind Socket. For users, they only need to call the relevant functions specified by Socket to let Socket organize data that conforms to the specified protocol and then proceed communication.

• Socket is also known as "socket". Applications usually send requests to the network or respond to network requests through "sockets".
• There are two commonly used Socket types: streaming Socket and datagram Socket. Streaming is a connection-oriented Socket for connection-oriented TCP service applications, and datagram Socket is a connectionless Socket for Connectionless UDP service application
• TCP: more reliable, connection-oriented, slower
• UDP: not too reliable, faster

For example: TCP is like a cash-on-delivery express delivery. You must see you when you arrive at home to count as a complete set of procedures. UDP is like a certain courier express cabinet, just throw it away and you will not receive it. Generally, UDP is used for live broadcast.

Three TCP programming

Go language implements TCP communication

TCP protocol

TCP/IP (Transmission Control Protocol/Internet Protocol), that is, Transmission Control Protocol/Internet Protocol, is a connection-oriented (connection-oriented), reliable, byte-stream-based transport layer (Transport layer) communication protocol, because it is Connection-oriented protocol, data is transmitted like a water flow, and there will be sticky packets.

TCP server

A TCP server can connect to many clients at the same time. For example, users from all over the world use their browsers on their computers to visit Taobao.com. Because it is very convenient and efficient to create multiple goroutines in the Go language to achieve concurrency, we can create a goroutine to process each time a link is established.

The processing flow of the TCP server program:

    1.监听端口
    2.接收客户端请求建立链接
    3.创建goroutine处理链接。

The TCP server code implemented by the net package of Go language is as follows:

// tcp/server/main.go

// TCP server端

// 处理函数
func process(conn net.Conn) {
    defer conn.Close() // 关闭连接
    for {
        reader := bufio.NewReader(conn)
        var buf [128]byte
        n, err := reader.Read(buf[:]) // 读取数据
        if err != nil {
            fmt.Println("read from client failed, err:", err)
            break
        }
        recvStr := string(buf[:n])
        fmt.Println("收到client端发来的数据：", recvStr)
        conn.Write([]byte(recvStr)) // 发送数据
    }
}

func main() {
    listen, err := net.Listen("tcp", "127.0.0.1:20000")
    if err != nil {
        fmt.Println("listen failed, err:", err)
        return
    }
    for {
        conn, err := listen.Accept() // 建立连接
        if err != nil {
            fmt.Println("accept failed, err:", err)
            continue
        }
        go process(conn) // 启动一个goroutine处理连接
    }
}

Save the above code and compile it into a server or server.exe executable file.

TCP client

The flow of a TCP client for TCP communication is as follows:

    1.建立与服务端的链接
    2.进行数据收发
    3.关闭链接

The TCP client code implemented using the net package of the Go language is as follows:

// tcp/client/main.go

// 客户端
func main() {
    conn, err := net.Dial("tcp", "127.0.0.1:20000")
    if err != nil {
        fmt.Println("err :", err)
        return
    }
    defer conn.Close() // 关闭连接
    inputReader := bufio.NewReader(os.Stdin)
    for {
        input, _ := inputReader.ReadString('\n') // 读取用户输入
        inputInfo := strings.Trim(input, "\r\n")
        if strings.ToUpper(inputInfo) == "Q" { // 如果输入q就退出
            return
        }
        _, err = conn.Write([]byte(inputInfo)) // 发送数据
        if err != nil {
            return
        }
        buf := [512]byte{}
        n, err := conn.Read(buf[:])
        if err != nil {
            fmt.Println("recv failed, err:", err)
            return
        }
        fmt.Println(string(buf[:n]))
    }
}

Compile the above code into client or client.exe executable file, start the server first and then start the client, enter any content on the client and press Enter, you can see the data sent by the client on the server, so as to realize TCP communication .

Four UDP programming

Go language implements UDP communication

UDP protocol

The Chinese name of the UDP protocol (User Datagram Protocol) is the User Datagram Protocol. It is a connectionless transport layer protocol in the OSI (Open System Interconnection, Open System Interconnection) reference model. It can directly send and receive data without establishing a connection. Reception is an unreliable and out-of-sequence communication, but the UDP protocol has better real-time performance and is usually used in fields related to live video.

UDP server

The UDP server code implemented using the net package of the Go language is as follows:

// UDP/server/main.go

// UDP server端
func main() {
    listen, err := net.ListenUDP("udp", &net.UDPAddr{
        IP:   net.IPv4(0, 0, 0, 0),
        Port: 30000,
    })
    if err != nil {
        fmt.Println("listen failed, err:", err)
        return
    }
    defer listen.Close()
    for {
        var data [1024]byte
        n, addr, err := listen.ReadFromUDP(data[:]) // 接收数据
        if err != nil {
            fmt.Println("read udp failed, err:", err)
            continue
        }
        fmt.Printf("data:%v addr:%v count:%v\n", string(data[:n]), addr, n)
        _, err = listen.WriteToUDP(data[:n], addr) // 发送数据
        if err != nil {
            fmt.Println("write to udp failed, err:", err)
            continue
        }
    }
}

UDP client

The UDP client code implemented using the net package of the Go language is as follows:

// UDP 客户端
func main() {
    socket, err := net.DialUDP("udp", nil, &net.UDPAddr{
        IP:   net.IPv4(0, 0, 0, 0),
        Port: 30000,
    })
    if err != nil {
        fmt.Println("连接服务端失败，err:", err)
        return
    }
    defer socket.Close()
    sendData := []byte("Hello server")
    _, err = socket.Write(sendData) // 发送数据
    if err != nil {
        fmt.Println("发送数据失败，err:", err)
        return
    }
    data := make([]byte, 4096)
    n, remoteAddr, err := socket.ReadFromUDP(data) // 接收数据
    if err != nil {
        fmt.Println("接收数据失败，err:", err)
        return
    }
    fmt.Printf("recv:%v addr:%v count:%v\n", string(data[:n]), remoteAddr, n)
}

Five http programming

web workflow

The working principle of the web server can be simply summarized as:

• The client establishes a TCP connection to the server through the TCP/IP protocol
• The client sends an HTTP protocol request packet to the server, requesting the resource document in the server
• The server sends an HTTP protocol response packet to the client. If the requested resource contains dynamic language content, the server will call the interpretation engine of the dynamic language to process the "dynamic content" and return the processed data to the client.
• Client disconnected from server. The HTML document is interpreted by the client, and the graphical result is rendered on the client screen

HTTP protocol

Hypertext Transfer Protocol (HTTP, HyperText Transfer Protocol) is the most widely used network protocol on the Internet. It specifies the rules for mutual communication between browsers and World Wide Web servers. It is a data transfer protocol HTTP protocol for transmitting World Wide Web documents through the Internet
. Usually carried on top of the TCP protocol

HTTP server

package main

import (
    "fmt"
    "net/http"
)

func main() {
    //http://127.0.0.1:8000/go
    // 单独写回调函数
    http.HandleFunc("/go", myHandler)
    //http.HandleFunc("/ungo",myHandler2 )
    // addr：监听的地址
    // handler：回调函数
    http.ListenAndServe("127.0.0.1:8000", nil)
}

// handler函数
func myHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Println(r.RemoteAddr, "连接成功")
    // 请求方式：GET POST DELETE PUT UPDATE
    fmt.Println("method:", r.Method)
    // /go
    fmt.Println("url:", r.URL.Path)
    fmt.Println("header:", r.Header)
    fmt.Println("body:", r.Body)
    // 回复
    w.Write([]byte("www.5lmh.com"))
}

HTTP server

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    //resp, _ := http.Get("http://www.baidu.com")
    //fmt.Println(resp)
    resp, _ := http.Get("http://127.0.0.1:8000/go")
    defer resp.Body.Close()
    // 200 OK
    fmt.Println(resp.Status)
    fmt.Println(resp.Header)

    buf := make([]byte, 1024)
    for {
        // 接收服务端信息
        n, err := resp.Body.Read(buf)
        if err != nil && err != io.EOF {
            fmt.Println(err)
            return
        } else {
            fmt.Println("读取完毕")
            res := string(buf[:n])
            fmt.Println(res)
            break
        }
    }
}

Six WebSocket Programming

What is webSocket

• WebSocket is a protocol for full-duplex communication over a single TCP connection
• WebSocket makes the data exchange between client and server easier, allowing the server to actively push data to the client
• In the WebSocket API, the browser and the server only need to complete a handshake, and a persistent connection can be created directly between the two, and two-way data transmission can be performed.
• Third-party packages need to be installed:
  In cmd: go get -u -v github.com/gorilla/websocket

We start an http server and specify the root path to route to an html page, which is used to simulate the client of websocket communication. The page will provide a button to trigger a section of js that executes websocket communication. The server receives the websocket request, and then fully responds to the browser with the content of the request.

The server that receives the websocket request:

package main

import (
	"fmt"
	"net/http"

	"github.com/gorilla/websocket"
)

var upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
}

func main() {
	http.HandleFunc("/echo", func(w http.ResponseWriter, r *http.Request) {
		conn, _ := upgrader.Upgrade(w, r, nil) // error ignored for sake of simplicity

		for {
			// Read message from browser
			msgType, msg, err := conn.ReadMessage()
			if err != nil {
				return
			}

			// Print the message to the console
			fmt.Printf("%s sent: %s\n", conn.RemoteAddr(), string(msg))

			// Write message back to browser
			if err = conn.WriteMessage(msgType, msg); err != nil {
				return
			}
		}
	})

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		http.ServeFile(w, r, "H:\\go\\main\\websockets.html")
	})

	http.ListenAndServe(":8080", nil)
}

Client sending websocket request:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>WebSockets</title>
</head>
<body>
<input id="input" type="text" />
<button onclick="send()">Send</button>
<pre id="output"></pre>
<script>
    var input = document.getElementById("input");
    var output = document.getElementById("output");
    var socket = new WebSocket("ws://localhost:8080/echo");

    socket.onopen = function () {
        output.innerHTML += "Status: Connected\n";
    };

    socket.onmessage = function (e) {
        output.innerHTML += "Server: " + e.data + "\n";
    };

    function send() {
        socket.send(input.value);
        input.value = "";
    }
</script>
</body>
</html>

Reposted from https://www.cnblogs.com/qi66/p/16773296.html