zmq message mode

Three message modes of ZMQ

1. What is ZMQ?

Ordinary socket is end-to-end (1:1) relationship, ZMQ is N:M relationship, socket connection needs to explicitly establish connection, destroy connection, select protocol (TCP/UDP) and error handling, ZQM shields These details, like an encapsulated socket library, make network programming easier. ZMQ is not only used for socket communication between hosts, but also for communication between threads and processes.
The socket provided by ZMQ can transmit messages in various protocols, inter-thread, inter-process, TCP, etc. Sockets can be used to create a variety of message modes, such as 'request-reply mode', 'publish-subscribe mode', 'distributed mode', etc.

2. ZMQ Features

Components come and go freely, ZQM will be responsible for automatic reconnection, and the server and client can exit the network at will. For tcp, the server must be started first, and then the client must be started, otherwise an error will be reported.
ZMQ will save the message in the queue if necessary, and start sending it once the connection is established.
ZMQ has a threshold mechanism. When the queue is full, it can automatically block the sender or discard some messages.
ZMQ can use different communication protocols to connect, TCP, inter-process, inter-thread.
ZMQ provides a variety of modes for message routing. Such as request-response mode, publish-subscribe mode, etc., these modes can be used to build network topology.
ZMQ will process I/O operations asynchronously in the background thread, and it uses a data structure that does not deadlock to store messages.

3. Three message modes of ZMQ

1. Reuqest-Reply (request-response mode)

(1). To use the Request-Reply mode, you need to follow certain rules.

(2). The client must first send a message, and then receive the message; the server must first receive the message sent by the client, and then send a response to the client, and so on.

(3). Whichever starts first, the server or the client, will have the same effect.

(4). Before the server receives the message, it will keep blocking, waiting for the client to connect.
insert image description here
Example
Create a client and server, the client sends a message to the server, and the server returns a message to the client, whichever starts
client.py

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:7777")


# 客户端必须要先发送消息，然后在接收消息
if __name__ == '__main__':
    for i in range(1, 10):
        socket.send_string("hello")
        message = socket.recv()
        print(f'received reply message: {
      
      message}')

server.py

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:7777")
count = 0


# 必须要先接收消息，然后在应答
if __name__ == '__main__':
    print('zmq server start....')
    while True:
        message = socket.recv()
        count += 1
        print(f'received request.message: {
      
      message}, count: {
      
      count}')
        time.sleep(1)
        socket.send_string('world!')

Common data request sending API:

#发送数据
socket.send_json(data)   #data 会被json序列化后进行传输 (json.dumps)
socket.send_string(data, encoding="utf-8")   #data为unicode字符串，会进行编码成子节再传输
socket.send_pyobj(obj)    #obj为python对象，采用pickle进行序列化后传输
socket.send_multipart(msg_parts)   # msg_parts, 发送多条消息组成的迭代器序列，每条消息是子节类型，如[b"message1", b"message2", b"message2"]

#接收数据
socket.recv_json()
socket.recv_string()
socket.recv_pyobj()
socket.recv_multipart()

2. Publisher-Subscriber (publish-subscribe mode)

In the Publisher-Subscriber mode, messages flow in one direction. Publishers can only publish messages, but cannot receive messages; subscribers can only receive messages, but cannot send messages.
During the process of publishing messages on the server, if a subscriber exits, it will not affect the publisher to continue publishing messages. When the subscriber connects again, the received message is the message published later
Subscribers who joined late, or subscribers who left midway, will inevitably lose some information
If the publisher stops, all subscribers will block and continue to receive messages when the publisher comes online again.
"Slow connection": We don't know when the subscriber starts to receive messages. Even if the "subscriber" is started first, and then the "publisher" is started, the "subscriber" will still miss some messages, because it takes time to establish a connection Yes, although the time is short, it is not zero. ZMQ performs asynchronous IO transmission in the background, and ZMQ can send a lot of messages within a short period of time when a TCP connection is established.
There is a simple way to synchronize "publisher" and "subscriber", let the publisher delay publishing messages through sleep, and wait for the connection to be established before sending.

Publisher.py

import zmq
import time
import random

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:7777")


if __name__ == '__main__':
   print("发布者启动......")
   time.sleep(2)
   for i in range(1000):
       temp = random.randint(-10, 40)
       message = "我是publisher, 这是我发布给你们的第{}个消息！今日温度{}".format(i+1, temp)
       socket.send_string(message)

Subscriber.py

import zmq

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:7777")

#客户端需要设定一个过滤，否则收不到任何信息
socket.setsockopt_string(zmq.SUBSCRIBE, '')


if __name__ == '__main__':
   print("订阅者一号启动......")
   while True:
       message = socket.recv_string()
       print(f"（订阅者一号）接收到'发布者'发送的消息：{
      
      message}")

3. Push-Pull (parallel pipeline mode/distributed processing)

Ventilator: The task publisher will generate a large number of tasks that can be calculated in parallel.
Worker: There is a group of workers that will handle these tasks
Sink: The result receiver will receive the processing results of all Workers at the end and summarize them
The upstream of the Worker is connected to the "task publisher", and the downstream is connected to the "result receiver"
"Task Publisher" and "Result Receiver" are relatively stable parts of this network structure, and they are bound to endpoints
Worker just connects two endpoints
You need to wait until all the workers are started before distributing tasks. The Socket connection will take a certain amount of time (slow connection). If there is no synchronization, the first Worker will receive many tasks at once when it starts.
"Task Distributor" will evenly distribute tasks to Workers (load balancing mechanism)
"Result Receiver" will evenly collect messages from Worker (fair queue mechanism)

Ventilator.py task distribution

import zmq
import random

raw_input = input
context = zmq.Context()
sender = context.socket(zmq.PUSH)
sender.bind("tcp://*:7777")

sink = context.socket(zmq.PUSH)
sink.connect("tcp://localhost:7778")

if __name__ == '__main__':
   # 同步操作
   print("Press Enter when the workers are ready: ")
   _ = raw_input()
   print("sending tasks to workers...")

   sink.send_string('0')

   # 发送十个任务
   total_time = 0
   for task_one in range(10):
       # 每个任务耗时为N
       workload = random.randint(1, 5)
       total_time += workload

       sender.send_string(str(workload))
   print(f"10个任务的总工作量: {
      
      total_time} 秒")

Worker.py (intermediate processing)

import zmq
import time

context = zmq.Context()

receiver = context.socket(zmq.PULL)
receiver.connect("tcp://localhost:7777")

sender = context.socket(zmq.PUSH)
sender.connect("tcp://localhost:7778")


if __name__ == '__main__':
   while True:
       s = receiver.recv_string()
       print(f'work1 接收到一个任务...需要{
      
      s}秒')

       # Do the work
       time.sleep(int(s))

       # send results to sink
       sender.send_string(f'work1完成一个任务，耗时{
      
      s} 秒')

Sink.py result sink

import zmq
import time

context = zmq.Context()

receiver = context.socket(zmq.PULL)
receiver.bind("tcp://*:7778")


if __name__ == '__main__':
   s = receiver.recv_string()
   print('开始接收处理结果...')

   # 计时，所有任务处理完一共需要多久
   start = time.time()

   # 接收十个任务的处理结果
   for task_one in range(10):
       s = receiver.recv_string()
       print(s)

   print(f"三个worker同时工作，耗时: {
      
      time.time() - start}秒")

zmq.Poller (ZMQ timeout judgment)

socket_req_url = config.zmq_rep_host.format(config.zmq_rep_port)
socket_req = zmq.Context().socket(zmq.REQ)
socket_req.connect(socket_req_url)
poller = zmq.Poller()
poller.register(socket_req, zmq.POLLIN)

Timeout reconnection instance:
Server server:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
count = 0

#必须要先接收消息，然后在应答
if __name__ == '__main__':
    print('zmq server start....')
    while True:
        message = socket.recv()
        count += 1
        print('received request. message:{} count:{}'.format(message, count))
        time.sleep(1)
        socket.send_string("ping test suc")

ClientClient:

import zmq

#超时重连
class PingPort():
    def __init__(self):
        self.port = '5555'
        self.socket_req_url = 'tcp://localhost:{}'.format(self.port)
        self.socket_req = zmq.Context().socket(zmq.REQ)
        self.socket_req.connect(self.socket_req_url)
        self.poller = zmq.Poller()
        self.poller.register(self.socket_req, zmq.POLLIN)

    def ping(self):
        self.socket_req.send_string('ping test')
        if self.poller.poll(3000):
            resp = self.socket_req.recv()
            print(resp)
　　　　　　  return True
        else:
            print('ping {} port fail, no response.'.format(self.port))
            self.socket_req.setsockopt(zmq.LINGER, 0)
            self.socket_req.close()
            self.poller.unregister(self.socket_req)
            print('-------------开始重连--------------------')
            self.socket_req = zmq.Context().socket(zmq.REQ)
            self.socket_req.connect(self.socket_req_url)
            self.poller = zmq.Poller()
            self.poller.register(self.socket_req, zmq.POLLIN)
            self.ping()

if __name__ == '__main__':
    obj = PingPort()
    print(obj.ping)

If the server is not enabled, the display effect is as follows:
insert image description here
if the server is enabled, return True