I have recently encountered ZeroMQ in the project. The internal implementation is quite complicated. I do n’t have time to understand it in depth. Simply record the usage method, and I will have time to fill the hole. Official guidance document http://zguide.zeromq.org/page:all
The project mainly uses ZeroMQ to communicate between services on multiple ip hosts. It can also be achieved directly with scoket, but it is more time-consuming and laborious. ZeroMQ is built on the socket and provides a simpler and more powerful API that can be quickly built. Starting from cross-process, cross-ip and other communication networks. Many articles mentioned that sockets can only achieve one-to-one communication. ZeroMQ can achieve many-to-many connections, and there are three modes to choose from, which can be selected and used according to business needs.
The three communication modes of ZeroMQ are: Request-Reply, Publisher-subscriber, Parallel Pipeline
Python install zmq module: pip install pyzmq
1. Request-Reply (reply mode)
Response mode features:
1. The client makes a request, the server must answer the request, each request is only answered once
2. The client cannot make another request before receiving the reply
3. Multiple clients can make requests, and the server can ensure that each client only receives its own reply
4. What happens if the server is broken or the client is broken?
If the client is broken, there is no impact on the server. If the client is restarted later, the two parties continue to ask one question and answer, but if the server is broken, some problems may occur, which depends on Under what circumstances the server is disconnected, if the server receives is disconnected after answering the question, then it does not affect, after restarting the server, double send continues to question and answer, but if the server is receiving After the question is broken, there is no time to answer the question. This is a problem. The client who asked the question will not wait for the answer and will wait for the answer. Therefore, no new questions will be sent. After the server restarts, The client did not post any questions, so it kept waiting for questions.
Python implements the client and server code as follows:
zmq_server.py
import zmq context = zmq.Context () #create context socket = context.socket (zmq.REP) #create Response server socket socket.bind ( " tcp: // *: 5555 " ) #socket binding, * means this Machine ip, port number is 5555, use tcp protocol communication while True: message = socket.recv () print (type (message)) #The received message will also be bytes type (byte) print ( " Received message: { } " .format (message)) socket.send (b " new message " ) #Send message, byte code message
zmq_client.py
#coding:utf-8 import zmq context = zmq.Context() socket = context.socket(zmq.REQ) socket.connect("tcp://localhost:5555") socket.send(b"A message") response = socket.recv() print(response)
Common data sending APIs are as follows:
# 送 数据 socket.send_json (data) #data will be serialized by json for transmission (json.dumps) socket.send_string (data, encoding = " utf-8 " ) #data is a unicode string, which will be encoded into subsections Then transmit socket.send_pyobj (obj) #obj is a Python object, use pickle to serialize and transmit socket.send_multipart (msg_parts) # msg_parts, send an iterator sequence composed of multiple messages, each message is a subsection type, # such as [b " message1 " , b " message2 " , b " message2 " ] #Receive data socket.recv_json () socket.recv_string () socket.recv_pyobj () socket.recv_multipart ()
2. Publisher-Subscriber (publish-subscribe model)
Publiser broadcasts messages to all clients, and clients filter messages based on the topic of subscription
The Python implementation code is as follows. The publisher publishes two messages. The topic of the first message is client1, which is received by the first subscriber; the topic of the second message is client2, which is received by the second subscriber.
Note that when the subscriber is matched, it is not an exact match. The topic of the message will be matched with the string beginning with client1. If the topic is "client1cient2", it will also be received by the first subscriber
zmq_server.py
#coding:utf-8 import zmq context = zmq.Context() socket = context.socket(zmq.PUB) socket.bind("tcp://*:5555") topic = ["client1", "client2"] while True: for t in topic: data = "message for {}".format(t) msg = [t.encode("utf-8"), data.encode("utf-8")] # The first item in the list is the topic of the message, and the sub filter messages according to the topic print (msg) socket.send_multipart (msg)
zmq_client1.py
#coding:utf-8 import zmq context = zmq.Context() socket = context.socket(zmq.SUB) socket.subscribe("client1") #订阅主题topic为：client1 socket.connect("tcp://localhost:5555") msg = socket.recv_multipart() print(msg)
result:
zmq_client2.py
import zmq
context = zmq.Context ()
socket = context.socket (zmq.SUB)
socket.subscribe ("client2") #Subscription topic topic is: client2
socket.connect("tcp://localhost:5555")
msg = socket.recv_multipart()
print(msg)
result:
3. Parallel Pipeline (parallel pipeline mode)
The pipeline mode consists of three parts. As shown in the figure below, the leftmost producer generates tasks through push. The consumer in the middle receives the tasks and forwards them. Finally, the result collector receives the results of all tasks. Compared with the publisher-subscriber, there is an additional data cache and processing load part. When the connection is disconnected, the data will not be lost. After the reconnection, the data continues to be sent to the client.
Python implements producer, consumer, resultcollector
producer.py
import zmq context = zmq.Context() socket = context.socket(zmq.PUSH) socket.bind("tcp://*:5577") for num in range(2000): work_message = {"num": num} socket.send_json(work_message)
consumer.py
import random import zmq context = zmq.Context() consumer_id = random.randint(1, 1000) #接收工作 consumer_receiver = context.socket(zmq.PULL) consumer_receiver.connect("tcp://localhost:5577") #转发结果 consumer_sender = context.socket(zmq.PUSH) consumer_sender.bind("tcp://*:5578") while True: msg = consumer_receiver.recv_json() data = msg["num"] result = {"consumer_id":consumer_id, "num":data} consumer_sender.send_json(result)
resultcollector.py
#coding:utf-8 import zmq context = zmq.Context() result_receiver = context.socket(zmq.PULL) result_receiver.connect("tcp://localhost:5578") result = result_receiver.recv_json() collecter_data = {} for x in range(1000): if result['consumer_id'] in collecter_data: collecter_data[result['consumer_id']] = collecter_data[result['consumer_id']] + 1 else: collecter_data[result['consumer_id']] = 1 if x == 999: print(collecter_data)
Execution order:
python producer.py
python consumer.py
python resultcollector.py
Reference article:
https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pushpull.html