Python-based socket network communication [1]

1. Socket principle

After learning the knowledge of the boss, simply take some notes
https://www.jianshu.com/p/066d99da7cbd
http://c.biancheng.net/view/2351.html

1.1 What is Socket

In the field of computer communication, socket is translated as "socket", which is a convention or a way to communicate between computers. Through the agreement of socket, a computer can receive data
　　from other computers, and can also send data to other computers. > Read and write write/read –> close” mode to operate.
　　My understanding is that Socket is an implementation of this mode: that is, socket is a special file, and some socket functions are operations on it (read/write IO, open, close).
　　The Socket() function returns an integer Socket descriptor, and subsequent operations such as connection establishment and data transmission are all implemented through the Socket.

Baidu Encyclopedia: A socket (socket) is an abstraction layer through which an application can send or receive data, and it can be opened, read, written, and closed like a file. Sockets allow applications to insert I/O into the network and communicate with other applications on the network. A network socket is a combination of an IP address and a port

1.2 sockets in Unix/Lunix

In the UNIX/Linux system, in order to unify the operation of various hardware and simplify the interface, different hardware devices are also regarded as a file. Operations on these files are equivalent to operations on ordinary files on disk.

You may have heard many experts say that everything in UNIX/Linux is a file! That guy was right.

In order to represent and distinguish the opened files, UNIX/Linux will assign an ID to each file. This ID is an integer and is called a file descriptor (File Descriptor). For example:
0 is usually used to indicate the standard input file (stdin), and its corresponding hardware device is the keyboard;
usually 1 is used to indicate the standard output file (stdout), and its corresponding hardware device is the monitor.

When a UNIX/Linux program performs any form of I/O, it reads from or writes to a file descriptor. A file descriptor is just an integer associated with an open file, and behind it may be an ordinary file on the hard disk, a FIFO, a pipe, a terminal, a keyboard, a monitor, or even a network connection.

Note that a network connection is also a file, and it has file descriptors too! You must understand this sentence.

We can use the socket() function to create a network connection, or open a network file, and the return value of socket() is the file descriptor. With the file descriptor, we can use common file operation functions to transfer data, for example:
use read() to read data from a remote computer;
use write() to write data to a remote computer.

You see, as long as you use socket() to create a connection, the rest is file operations. Network programming is so simple!

1.3 sockets in Windows

Windows also has a concept similar to "file descriptors", but they are usually called "file handles". Therefore, this tutorial will use "handle" if it refers to the Windows platform, and "descriptor" if it refers to the Linux platform.

Unlike UNIX/Linux, Windows will distinguish between sockets and files, and Windows treats sockets as a network connection, so it is necessary to call data transfer functions specially designed for sockets, and the input and output functions for ordinary files are invalid

2. How do processes communicate in the network

2.1, local inter-process communication

a. Message passing (pipeline, message queue, FIFO)
　　b. Synchronization (mutex, condition variable, read-write lock, file and write record lock, semaphore)
　　c. Shared memory (anonymous and named, eg: channel )
　　d. Remote Procedure Call (RPC)

2.2 How do processes communicate in the network

To understand how processes communicate in the network, we have to solve two problems:
　　a. How do we identify a host, that is, how to determine which host the process we are going to communicate runs on.
　　b. How do we identify a unique process, locally through pid identification, how should we identify it in the network?
Solution:
　　a. The TCP/IP protocol family has helped us solve this problem. The "ip address" of the network layer can uniquely identify the host in the network b. The "
　　protocol + port" of the transport layer can uniquely identify the application program in the host (process), therefore, we can use the triplet (ip address, protocol, port) to identify the process of the network, and the process communication in the network can use this mark to interact with other processes

3. Socket communication classification

The triplet [ip address, protocol, port] can be used between processes in the network to communicate between networks. Socket is a middleware tool that uses triplets to solve network communication. For now, almost all applications use Socket, there are two commonly used data transmission methods for Socket communication:
a. SOCK_STREAM: corresponds to the TCP protocol, indicating a connection-oriented data transmission method. Data can reach another computer without error, and if damaged or lost, can be resent, but relatively slowly. The common http protocol uses SOCK_STREAM to transmit data, because it is necessary to ensure the correctness of the data, otherwise the webpage cannot be parsed normally.
　　b. SOCK_DGRAM: Corresponds to the UDP protocol, indicating a connectionless data transmission method. The computer only transmits the data and does not check the data. If the data is damaged during transmission or does not reach another computer, there is no way to remedy it. In other words, if the data is wrong, it is wrong and cannot be retransmitted. Because SOCK_DGRAM does less verification work, it is more efficient than SOCK_STREAM.
　　For example: QQ video chat and voice chat use SOCK_DGRAM to transmit data, because the efficiency of communication must be ensured first, and the delay should be minimized, while the correctness of data is secondary. Even if a small part of data is lost, video and audio can still Normal analysis, noise or noise at most, will not have a substantial impact on communication quality
　　
Socket programming is based on TCP and UDP protocols, and their hierarchical relationship is shown in the figure below:

Diagram Socket() function

4. Three-way handshake for Socket (TCP) connection establishment

The functions of Socket are simplified to three: establish connection, send data and receive data, the following link is the process of establishing connection
http://c.biancheng.net/view/2351.html

Socket buffer

After each socket is created, two buffers are allocated, an input buffer and an output buffer.

write()/send() does not transmit data to the network immediately, but writes the data into the buffer first, and then sends the data from the buffer to the target machine by the TCP protocol. Once the data is written to the buffer, the function can return successfully, regardless of whether they reached the target machine or not, and regardless of when they were sent to the network, these are the things that the TCP protocol takes care of.

The TCP protocol is independent of the write()/send() function. The data may be sent to the network as soon as it is written into the buffer, or it may continue to backlog in the buffer, and the data written multiple times will be
sent This depends on many factors such as the network situation at the time, whether the current thread is idle, etc., and is not controlled by the programmer.

The same goes for the read()/recv() functions, which also read data from the input buffer instead of directly from the network

These I/O buffer properties can be organized as follows:

1. The I/O buffer exists separately in each TCP socket;
2. The I/O buffer is automatically generated when the socket is created;
3. Continues to transmit data left in the output buffer even if the socket is closed;
4. Closing the socket will lose the data in the input buffer.

Under normal circumstances, it does not matter the size of the default buffer, but you can also use the following code to view and modify

sock.getsockopt()        # 获取缓冲区大小
sock.setsockopt()         #更改缓冲区大小

Send and receive function characteristics:

recv features:

1. If the other end of the established link is disconnected, recv returns an empty string immediately

2. recv is to take the content from the receiving buffer, and block when the buffer is empty

3. If recv cannot accept the contents of the buffer at one time, it will automatically accept the next execution

send features:

1. If the other end of the sending does not exist, a Pipe Broken exception will be generated

2. send is to send content from the send buffer, and block when the buffer is full

This is the blocking mode of TCP sockets. The so-called blocking means that the previous action is not completed, and the next action will be suspended until the previous action is completed, so as to maintain synchronization. By default, TCP sockets are in blocking mode, which is also the most commonly used.

Sticky packet problem of TCP protocol

In the transfer process of socket buffer and data, it can be seen that the receiving and sending of data are irrelevant. The read()/recv() function will receive as much data as possible no matter how many times the data is sent. That is, the execution times of read()/recv() and write()/send() may be different.
For example, write()/send() is repeated three times, sending the string "abc" each time, then read()/recv() on the target machine may receive "abc" three times, each time receiving "abc"; Receive twice, the first time to receive "abcab", the second time to receive "cabc"; it is also possible to receive the string "abcabcabc" once.

Suppose we want the client to send a student’s student ID every time, and let the server return the student’s name, address, grades and other information. At this time, there may be problems, and the server cannot distinguish the student’s student ID. For example, if 1 is sent for the first time and 3 is sent for the second time, the server may treat it as 13, and the returned information is obviously wrong.

This is the problem of "sticky packets" of data. Multiple data packets sent by the client are received as one data packet. Also known as the unboundedness of data, the read()/recv() function does not know the start or end marker of the data packet (in fact, there is no start or end marker), and only treats them as a continuous data stream.

Really close the socket

The close method can release the resources of a connection, but not immediately. If you want to release immediately, please use the shutdown method before close. The shutdown method
is used to realize the communication mode. There are three modes. SHUT_RD closes the receiving message channel, and SHUT_WR closes Send message channel, SHUT_RDWR Both channels are closed
That is to say, if you want to close a connection, first close all channels, and then connect in release. The above three static variables correspond to digital constants: 0, 1, 2

self.tcpClient.shutdown(2)      #关闭消息发送通道
self.tcpClient.close()            #关闭套接字连接

Python socket programming example 1 - text transmission

client:

#port = str(input('please input sever port:'))
host = '192.168.2.107'    #客户端连接到服务器的ip
port = 5270  #端口
sever_address = (host, port)     #元组定义服务器地址，用于作为socket.connect()函数参数 连接到服务器
text_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)     #创建一个socket对象为text_client

text_client.connect(sever_address)       #连接到服务器
succeed_flag='cok'                       #设置消息发送成功标志
text = 'connect succeed'
while True :
    try:
        text_client.send(text.encode())  # 发送文本数据，用 encode() 方法将str变为byte
        text = input('please input the message')
        receive_text = text_client.recv(1024).decode()
        print(receive_text)

    finally:
        print('send over!')

server:

import socket

#可以手动输入本机ip地址，若有多个网口，服务器想从那个网口接收数据，就输入那个网口的ip
#hostname = socket.gethostname()  #可以用 .gethostname()函数来自动得到主机ip，不用手动输入了
#host = socket.gethostbyname(hostname)
host = '192.168.2.107'    #客户端连接到服务器的ip
port = 5270  #端口
sever_address = (host, port)     #创建元组作为 socket.bind()函数的输入，

text_sever = socket.socket(socket.AF_INET, socket.SOCK_STREAM)     #创建一个socket对象为text_sever 为服务器
text_sever.bind(sever_address)    #.bind() 函数绑定端口，该服务器监听此端口
text_sever.listen(4)         #开启监听，同时接入数量最多为4

succeed_flag = 'sok'
while True :
    try:
        print(host)
        print('waiting connect')
        text_client_socket, text_client_address = text_sever.accept()              #accept() 函数，堵塞等待client的连接，连接到后才会执行下一条语句
        print(text_client_address[0] + 'is connected!')

        while True :
            receive_text = text_client_socket.recv(1024)  .decode()            #接收client发送的数据，数据最大为1024 ；此处可以看出接收用户数据测试
            print(receive_text)
            text_client_socket.send(succeed_flag.encode())                     #发送给client ok ，反馈自己确实接收到数据

    finally:
        print('work over!')

Python socket programming example 2 - video transmission

client:

#-*- coding: UTF-8 -*-
import cv2                    #opencv2库，用于进行各种图像处理
import time
import socket                 #socket库，用于构建tcp/ip  通信

# 服务端ip地址
HOST = '192.168.2.102'      #字符串类型存储 host ip，tcp/ip通信服务器需要固定的ip与port
# 服务端端口号
PORT = 8080
ADDRESS = (HOST, PORT)       #元组方式存储ip与port

# 创建一个套接字，命名为tcpClient
tcpClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)     #socket.socket() 函数：socket.AF_INET：基于IPv4  ，socket.SOCK_STREAM对应TCP
# 连接远程ip
#tcpClient.bind(('192.168.3.122', 8080))
tcpClient.connect(ADDRESS)    #客户端向服务端发起连接。一般address的格式为元组（hostname,port），如果连接出错，返回socket.error错误

#cap = cv2.VideoCapture('test1080.mp4')   #要发送的视频，如果是0为摄像头
cap = cv2.VideoCapture(0)
while True:
    # 计时
    start = time.perf_counter()         #计时器，第一次调用的时间存储在start里
    # 读取图像
    ref, cv_image = cap.read()           #返回第一个ref为true或false，表示是否读到了图像 ，第二个参数表示截取到一帧的图片数据，是一个三维数组
    # 压缩图像
    #cv2.imdecode()函数从指定的内存缓存中读取数据，并把数据转换(解码)成图像格式;主要用于从网络传输数据中恢复出图像。
    #cv2.imencode()函数是将图片格式转换(编码)成流数据，赋值到内存缓存中;主要用于图像数据格式的压缩，方便网络传输。
    img_encode = cv2.imencode('.jpg', cv_image, [cv2.IMWRITE_JPEG_QUALITY, 40])[1]        #第一个参数是压缩为什么格式，第二个参数是要压缩的数据源，最后一个参数是解码压缩参数，数字越大图片质量越好
    # 转换为字节流
    bytedata = img_encode.tostring()                     #将图像转换为字节流
    # 标志数据，包括待发送的字节流长度等数据，用‘,’隔开  ，发送的数据类型是 str
    flag_data = (str(len(bytedata))).encode() + ",".encode() + " ".encode()
    tcpClient.send(flag_data)                                  #客户端发送标志数据，服务器接收后知晓将要发送数据
    # 接收服务端的应答
    data = tcpClient.recv(1024)                #接收数据，数据以bytes类型返回，bufsize指定要接收的最大数据量为1024字节
    if ("ok" == data.decode()):               #接收到服务器返回'OK'后，发送全部图片数据，这里用decode进行了解码，因为socket传输字节流，对收到的字节解码才能得到字符串......
 #   if (data.decode()):
        # 服务端已经收到标志数据，开始发送图像字节流数据
        tcpClient.send(bytedata)
    # 接收服务端的应答
    data = tcpClient.recv(1024)
    if ("ok" == data.decode()):
        # 计算发送完成的延时
        print("延时：" + str(int((time.perf_counter() - start) * 1000)) + "ms")    #再次调用该计时函数，返回与上一次调用的时间间隔

server:

#-*- coding: UTF-8 -*-
import socket
import cv2
import numpy as np

HOST = '192.168.2.102'
PORT = 8080
ADDRESS = (HOST, PORT)
# 创建一个套接字
tcpServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# 绑定本地ip
tcpServer.bind(ADDRESS)
# 开始监听
tcpServer.listen(5)

while True:
    print("等待连接……")
    client_socket, client_address = tcpServer.accept()           #accept() 函数，堵塞等待客户端连接；反回（conn,address）二元元组,其中conn是一个通信对象，可以用来接收和发送数据。address是连接客户端的地址。
    print("连接成功！")
    try:                                                          #使用try   expect  便于处理异常
        while True:
            # 接收标志数据
            data = client_socket.recv(1024)
            if data:                                              #接收到的不为空就进入
                # 通知客户端“已收到标志数据，可以发送图像数据”
                client_socket.send(b"ok")                          #关于b"ok"      看https://www.delftstack.com/zh/howto/python/python-b-in-front-of-string/    ，这里是将ok变为byte
                # 处理标志数据
                flag = data.decode().split(",")                  #  strip() 方法用于移除字符串头尾指定的字符（默认为空格或换行符）或字符序列  ，这里将结尾的 '' 移除
                # 图像字节流数据的总长度
                total = int(flag[0])                         #flag[0]是第一个列表元素，flag是一个列表，返回第一个元素就是数据总长度的字符串形式，再用int（）转为整数
                # 接收到的数据计数
                cnt = 0
                # 存放接收到的数据
                img_bytes = b""

                while cnt < total:
                    # 当接收到的数据少于数据总长度时，则循环接收图像数据，直到接收完毕
                    data = client_socket.recv(256000)                 #一次能接收的最大数据量为256000byte的数据
                    img_bytes += data                                #对总数据量计数
                    cnt += len(data)
                    print("receive:" + str(cnt) + "/" + flag[0])     #打印接收/需接收
                # 通知客户端“已经接收完毕，可以开始下一帧图像的传输”
                client_socket.send(b"ok")

                # 解析接收到的字节流数据，并显示图像
                img = np.asarray(bytearray(img_bytes), dtype="uint8")
                img = cv2.imdecode(img, cv2.IMREAD_COLOR)
                cv2.namedWindow("img", 0)
                cv2.resizeWindow("img", 1280, 720)             #重设置显示界面的大小
                cv2.imshow("img", img)                         #第一个参数是窗口的名字，第二个是图像
                cv2.waitKey(1)

            else:
                print("已断开！")
                break
    finally:
        client_socket.close()

Python socket programming example 3 - tcp proxy proxy server

#https://www.youtube.com/watch?v=iApNzWZG-10

import socket
from threading import Thread
import os
#线程2

class Proxy2Server(Thread):
    #首先设置服务器连接（用_init_方法来构造）
    #参考https://www.cnblogs.com/ant-colonies/p/6718388.html
    def __init__(self, host, port):#如果没有在__init__中初始化对应的实例变量的话，导致后续引用实例变量会出错
        super(Proxy2Server,self).__init__()
        self.game = None #设置为连接用户的套接字，但是该套接字是由Game2Proxy线程创建的
        self.port = port
        self.host = host #连接服务器的ip和端口
        self.server = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        self.server.connect((host,port))

    #在这个线程中执行的函数
    def run(self):
        #创建一个循环来执行数据处理和网络连接
        while True:
            data  = self.server.recv(4096)#最多接收4k的数据
            if data:
                #转发所有数据到用户
                print("[{}] <- {}")  #.format(self.port,data[:100].encode('hex'))#用作测试，可以打印出数据的流向
                self.game.sendall(data)




#线程1(监听用户是否与代理服务器连接)
class Game2Proxy(Thread):

    def __init__(self,host,port):
        super(Game2Proxy,self).__init__()
        self.server = None #设置为连接服务器的套接字，但是该套接字是由线程2创建的
        self.port = port
        self.host = host #连接用户的ip和端口
        sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
        sock.bind((host,port))
        sock.listen(1)#这些都是上面官方文档里面调用的例程实现的
        #等待用户的连接 
        self.game ,addr = sock.accept() #sock.accept接收套接字
        #当客户端连接之后我们将获得代理服务器与客户端通信的套接字，并将其分配给self.game，然后在下面的线程中利用永久循环来接收用户端的数据
    
    def run(self):
        while True: #死循环接收用户的数据
            data = self.game.recv(4096)#最大数据量4k
            if data:    #如果真的接收到了用户发送过来的数据，那麽我们会尝试将此数据转发到服务器的套接字，即另外一个线程的套接字
                #转发给服务器
                print("[{}] -> {}")       #.format(self.port,data[:100].encode('hex'))#用作测试，可以打印出数据的流向
                self.server.sendall(data)



#上面的两个线程创建完毕之后，需要为每一个线程提供对另外一个套接字的引用
#为此，我创建了一个更通用的类，命名为Proxy
class Proxy(Thread):
    def __init__(self,from_host,to_host,port):#如果没有在__init__中初始化对应的实例变量的话，导致后续引用实例变量会出错
        super(Proxy, self).__init__()
        self.from_host = from_host 
        self.to_host = to_host
        self.port = port 

    def run(self):
        while True:
            #print ("[proxy({})] setting up")
            print ("代理服务器设置完毕，等待设备接入...")
            #用户会连接到下面这个
            self.g2p = Game2Proxy(self.from_host, self.port) #运行我们创建的这个线程，它等待用户端连接到指定端口
            #如果代理服务器与用户建立连接之后，另外一个线程将建立到服务器的转发连接
            self.p2s = Proxy2Server(self.to_host, self.port)
            #print ("[proxy({})] connection established")
            print ("代理服务器已和设备连接，正在传输...")
            #现在两个线程都创建了套接字，我们接下来要做的就是交换他们
            self.g2p.server = self.p2s.server   #将与客户端建立的套接字转发给真实服务器
            self.p2s.game = self.g2p.game       #将服务器传回的套接字转发到客户端

            #线程设置完毕，现在我们来真正启动它
            self.g2p.start()
            self.p2s.start()


#写到这里的时候，唯一缺少的就是创建一个或多个代理线程，我们先从主服务器开始
master_server = Proxy('0.0.0.0', '192.168.2.222', 5555)
#监听自己所有本机端口3333，并将它转发到真实的服务器ip 192.168.178.54
master_server.start()   #启动

#_game_server = Proxy('0.0.0.0', '192.168.2.222', 5555)
#_game_server.start()

'''
#除此之外，客户端想要连接多个服务器的时候，我们可以启动多个代理（多分配几个不同端口即可）
for port in range(3000,3006):
    _game_server = Proxy('0.0.0.0','192.168.178.54',port)  
    _game_server.start()
#写到这里就已经可以工作了
'''

some items

Python socket implements FTP client and server to send and receive files and md5 encrypted files

Python implements video transmission based on socket and opencv