When using golang to develop a manual customer service system, I encountered a sticky package problem, so what is a sticky package? For example, we agree with the client that the data interaction format is a string in json format:
{"Id":1,"Name":"golang","Message":"message"}
When the client sends data to the server, if the server does not receive it in time, the client sends another piece of data, and the server receives two consecutive strings at this time, as follows:
{"Id":1,"Name":"golang","Message":"message"}{"Id":1,"Name":"golang","Message":"message"}
If the receiving buffer is full, it is also possible to receive half of the json string. If it is purple, how can it be decoded with json? What a headache. The following uses golang to simulate the generation of this sticky package.
Note: The code posted below can run on golang 1.3.1, if you find any problems, please contact me.
Sticky Package Example
server.go
/ / Sticky package problem demonstration server
package main
import (
"fmt"
"net"
"os"
)
func main() {
netListen, err := net.Listen("tcp", ":9988")
CheckError(err)
defer netListen.Close()
Log("Waiting for clients")
for {
conn, err := netListen.Accept()
if err != nil {
continue
}
Log(conn.RemoteAddr().String(), " tcp connect success")
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn) {
buffer := make([]byte, 1024)
for {
n, err := conn.Read(buffer)
if err != nil {
Log(conn.RemoteAddr().String(), " connection error: ", err)
return
}
Log(conn.RemoteAddr().String(), "receive data length:", n)
Log(conn.RemoteAddr().String(), "receive data:", buffer[:n])
Log(conn.RemoteAddr().String(), "receive data string:", string(buffer[:n]))
}
}
func Log(v ...interface{}) {
fmt.Println (v ...)
}
func CheckError(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
}
client.go
//The sticky package problem demo client
package main
import (
"fmt"
"net"
"os"
"time"
)
func sender(conn net.Conn) {
for i := 0; i < 100; i++ {
words := "{\"Id\":1,\"Name\":\"golang\",\"Message\":\"message\"}"
conn.Write([]byte(words))
}
}
func main() {
server := "127.0.0.1:9988"
tcpAddr, err := net.ResolveTCPAddr("tcp4", server)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
defer conn.Close()
fmt.Println("connect success")
go sender(conn)
for {
time.Sleep(1 * 1e9)
}
}
You can see that the strings in json format are glued together, and there is a slight sadness - the headache is coming again.
Causes of sticky bags
There are many related explanations on the Internet about the causes of sticky packets. The main reason is that the tcp data transfer mode is the stream mode, which can be received and sent multiple times while maintaining a long connection. If you want to know more about it, you can take a look at the content of the tcp protocol. Here I recommend Brother Bird's private kitchen dishes, which are very easy to understand.
sticky package solution
There are two main methods:
1. The client disconnects after sending it once, and connects again when it needs to send data, such as http. Let's use golang to demonstrate this process, and there will be no sticky package problem.
//Client code, demonstrating the disconnection after sending data once
package main
import (
"fmt"
"net"
"os"
"time"
)
func main() {
server := "127.0.0.1:9988"
for i := 0; i < 10000; i++ {
tcpAddr, err := net.ResolveTCPAddr("tcp4", server)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
words := "{\"Id\":1,\"Name\":\"golang\",\"Message\":\"message\"}"
conn.Write([]byte(words))
conn.Close()
}
for {
time.Sleep(1 * 1e9)
}
}
2. In the format of header + data, the data to be analyzed is read according to the header information. The form is as follows: For the server-side code, refer to the server-side code that demonstrates the sticky package generation process above.
golang sticky package problem header definition
When reading data from the data stream, only the required data can be obtained according to the packet header and data length. This is actually the protocol (protocol) that is usually said, but this data transmission protocol is very simple, unlike tcp, ip and other protocols that have more definitions. In the actual process, a protocol class or a protocol file is usually defined to encapsulate the process of encapsulation and unpacking. The following code demonstrates the process of packing and unpacking:
protocol.go
//Communication protocol processing, mainly dealing with the process of encapsulation and unpacking
package protocol
import (
"bytes"
"encoding/binary"
)
const (
ConstHeader = "www.01happy.com"
ConstHeaderLength = 15
ConstSaveDataLength = 4
)
//packet
func Packet(message []byte) []byte {
return append(append([]byte(ConstHeader), IntToBytes(len(message))...), message...)
}
//unpack
func Unpack(buffer []byte, readerChannel chan []byte) []byte {
length := len(buffer)
var i int
for i = 0; i < length; i = i + 1 {
if length < i+ConstHeaderLength+ConstSaveDataLength {
break
}
if string(buffer[i:i+ConstHeaderLength]) == ConstHeader {
messageLength := BytesToInt(buffer[i+ConstHeaderLength : i+ConstHeaderLength+ConstSaveDataLength])
if length < i+ConstHeaderLength+ConstSaveDataLength+messageLength {
break
}
data := buffer[i+ConstHeaderLength+ConstSaveDataLength : i+ConstHeaderLength+ConstSaveDataLength+messageLength]
readerChannel <- data
i += ConstHeaderLength + ConstSaveDataLength + messageLength - 1
}
}
if i == length {
return make([]byte, 0)
}
return buffer[i:]
}
// convert integer to byte
func IntToBytes(n int) []byte {
x := int32(n)
bytesBuffer: = bytes.NewBuffer ([] byte {})
binary.Write(bytesBuffer, binary.BigEndian, x)
return bytesBuffer.Bytes ()
}
// convert byte to integer
func BytesToInt(b []byte) int {
bytesBuffer: = bytes.NewBuffer (b)
var x int32
binary.Read(bytesBuffer, binary.BigEndian, &x)
return int(x)
}
Tips: In the process of unpacking, pay attention to the problem of out-of-bounds array; in addition, pay attention to the uniqueness of the packet header.
server.go
// server unpacking process
package main
import (
"./protocol"
"fmt"
"net"
"os"
)
func main() {
netListen, err := net.Listen("tcp", ":9988")
CheckError(err)
defer netListen.Close()
Log("Waiting for clients")
for {
conn, err := netListen.Accept()
if err != nil {
continue
}
Log(conn.RemoteAddr().String(), " tcp connect success")
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn) {
//declare a temporary buffer to store the truncated data
tmpBuffer := make([]byte, 0)
//declare a pipe to receive unpacked data
readerChannel := make(chan []byte, 16)
go reader(readerChannel)
buffer := make([]byte, 1024)
for {
n, err := conn.Read(buffer)
if err != nil {
Log(conn.RemoteAddr().String(), " connection error: ", err)
return
}
tmpBuffer = protocol.Unpack(append(tmpBuffer, buffer[:n]...), readerChannel)
}
}
func reader(readerChannel chan []byte) {
for {
select {
case data := <-readerChannel:
Log(string(data))
}
}
}
func Log(v ...interface{}) {
fmt.Println (v ...)
}
func CheckError(err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
}
client.go
// client sends packet
package main
import (
"./protocol"
"fmt"
"net"
"os"
"time"
)
func sender(conn net.Conn) {
for i := 0; i < 1000; i++ {
words := "{\"Id\":1,\"Name\":\"golang\",\"Message\":\"message\"}"
conn.Write(protocol.Packet([]byte(words)))
}
fmt.Println("send over")
}
func main() {
server := "127.0.0.1:9988"
tcpAddr, err := net.ResolveTCPAddr("tcp4", server)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
os.Exit(1)
}
defer conn.Close()
fmt.Println("connect success")
go sender(conn)
for {
time.Sleep(1 * 1e9)
}
}
Running this program can see that the server has obtained the expected json format data very well. Complete code demo download: golang sticky package problem solving example
finally
The two methods demonstrated above are suitable for different scenarios. The first method is more suitable for passive scenarios, such as opening a web page and processing the interaction only when the user requests it. The second method is suitable for the type of active push, such as an instant chat system, because it is inevitable to maintain a long connection to push messages to users in real time, so this method is used at this time.