通常我们写服务器处理模型程序时分三种：
- 创建一个进程处理该请求：但创建一个进程开销比较大，使服务器性能比较差，实现较简单；
- 创建一个线程处理该请求：涉及线程同步，可能导致死锁问题；
- 收到请求放入事件列表，让主进程通过非阻塞IO来处理请求：逻辑较复杂（大多数网络服务器）。
UI编程中，如响应事件onclick：
一、创建一个线程，该线程一直循环事件onclick
- CPU资源浪费；
- 如果此onclick发生阻塞，永远不会进行下一事件；
- 如果一个循环需要扫描的设备非常多，这又会引来响应时间的问题。
二、事件驱动模型
- 有一个事件（消息）队列；
- 鼠标按下时，往这个队列中增加一个点击事件（消息）；
- 有个循环，不断从队列取出事件，根据不同的事件，调用不同的函数；
- 事件（消息）一般都各自保存在各自的处理函数之中，这样，每个消息都有独立的处理函数。
处理情形：
程序中有多种任务，且任务之间高度独立，执行某任务会阻塞。
IO模型：
- 阻塞 I/O（blocking IO）：两个阶段都被阻塞
- 非阻塞 I/O（nonblocking IO）：用户进程需不断的主动询问kernel数据好了没有，第二阶段阻塞
- I/O 多路复用（ IO multiplexing）：可以处理多个socket
- 信号驱动 I/O（ signal driven IO）
- 异步 I/O（asynchronous IO）：不阻塞
I/O 多路复用（ IO multiplexing）：select、poll、epoll
select：于1983年出现，通过select()系统调用来监视多个文件描述符的数组，当select()返回后，该数组中就绪的文件描述符便会被内核修改标志位，使得进程可以获得这些文件描述符从而进行后续的读写操作；可以良好的跨平台支持，但文件描述符存在最大限制1024；select()所维护大量文件描述符的数组，随着文件描述符的大量增加，复制的开销也线性增长，开销大。Linux：ulimit-SHn 65535
poll：于1986年诞生，包含大量文件描述符的数组被整体复制于用户态与内核态之间，而不论这些文件描述符是否就绪，所以开销也随文件描述符增加而增加；另外，select()和poll()将就绪的文件描述符告诉进程后，若进程没有对其进行io操作，那么下次调用时仍会报告这些文件描述符，这种方式成为水平触发（Level Triggered）。
epoll：可以同时支持水平触发和边缘触发（Edge Triggered，将就绪的文件描述符告诉进程，只说一遍，若我们没采取行动，下次不会再告知）
扫描二维码关注公众号，回复：
5158655 查看本文章
1 import select,socket,queue 2 server = socket.socket() 3 server.setblocking(False) #设置成非阻塞模式 4 server_address = (("localhost",9999)) 5 print("starting up %s on %s" % server_address) 6 server.bind(server_address) 7 server.listen(5) 8 inputs = [server] #监测文件描述符 9 outputs = [] 10 message_queue = {} 11 while True: 12 print("waiting for the next event.") 13 readable,writeable,exceptional = select.select(inputs,outputs,inputs)#如果没有任何fd就绪,那程序就会一直阻塞在这里 14 for s in readable: 15 if s is server:#代表来了个新连接 16 connection,client_address = s.accept() 17 print("new connection from",client_address) 18 connection.setblocking(False) 19 inputs.append(connection) #让select继续监听 20 message_queue[connection] = queue.Queue()#初始化一个队列，后面存要返回给这个链接的数据 21 else: 22 data = s.recv(1024) 23 if data: 24 print("received %s from %s" %(data,s.getpeername)) 25 message_queue[s].put(data) 26 if s not in outputs: 27 outputs.append(s)#放入返回的链接队列里 28 else: 29 print("closing",client_address,"after reading no data") 30 if s in outputs: 31 outputs.remove(s) 32 inputs.remove(s) 33 s.close() 34 del message_queue[s] 35 for s in writeable: 36 try: 37 next_msg = message_queue[s].get_nowait() 38 except queue.Empty: 39 print("output queue for",s.getpeername(),"is empty") 40 outputs.remove(s) 41 else: 42 print("sending '%s' to %s" %(next_msg,s.getpeername())) 43 s.send(next_msg) 44 for s in exceptional: 45 print("handling exceptional condition for",s.getpeername()) 46 inputs.remove(s) 47 if s in outputs: 48 outputs.remove(s) 49 s.close() 50 del message_queue[s]
1 import selectors,socket 2 sel = selectors.DefaultSelector() 3 def accept(sock,mask): 4 conn,addr = sock.accept() 5 print("accepted",conn,"from",addr) 6 conn.setblocking(False) 7 sel.register(conn,selectors.EVENT_READ,read)#新连接注册read回调函数 8 def read(conn,mask): 9 data = conn.recv(1024) 10 if data: 11 print("echoing",repr(data),"to",conn) 12 conn.send(data) 13 else: 14 print("closing",conn) 15 sel.unregister(conn) 16 conn.close() 17 sock = socket.socket() 18 sock.bind(("localhost",9999)) 19 sock.listen(100) 20 sock.setblocking(False) 21 sel.register(sock,selectors.EVENT_READ,accept)#来新连接调用accept 22 while True: 23 events = sel.select() #默认阻塞，有活动链接就返回活动的链接列表 24 for key,mask in events: 25 callback = key.data #相当于调accept 26 callback(key.fileobj,mask) #key.fileobj = 文件句柄
1 import socket 2 messages = [b"This is the message"] 3 server_address = ("localhost",9999) 4 socks = [socket.socket() for i in range(10)] 5 print("connecting to %s port %s" %server_address) 6 for s in socks: 7 s.connect(server_address) 8 for message in messages: 9 for s in socks: 10 print("%s:sending '%s'" %(s.getsockname(),message)) 11 s.send(message) 12 for s in socks: 13 data = s.recv(1024) 14 print("%s:received %s" %(s.getsockname(),data)) 15 if not data: 16 print("closing socket",s.getsockname()) 17 s.close()
三、RabbitMQ队列
安装 http://www.rabbitmq.com/install-standalone-mac.html
安装 python rabbitMQ module ：pip install pika
1、实现最简单的队列通信
1 import pika 2 conn = pika.BlockingConnection(pika.ConnectionParameters(host="localhost")) 3 channel = conn.channel() 4 #声明queue；durable=True，消息持久化 5 channel.queue_declare(queue="task_queue",durable=True) 6 message = "hello world!" 7 channel.basic_publish( 8 exchange="", #消息发送到exchange 9 routing_key = "task_queue", 10 body= message, 11 properties= pika.BasicProperties(delivery_mode=2,) #delivery_mode=2，消息持久化 12 ) 13 print("[x] sent %r" %message) 14 conn.close()
1 import pika 2 conn = pika.BlockingConnection(pika.ConnectionParameters(host= "localhost")) 3 channel = conn.channel() #两端都声明，保证无论哪端先执行都可以 4 channel.queue_declare(queue= "task_queue",durable= True) #durable= True，队列持久化 5 print("[>>] waiting for message. To exit press CTRL+C" ) 6 def callback(ch,method,properties,body): 7 print("Received %r" %body) 8 ch.basic_ack(delivery_tag= method.delivery_tag) 9 channel.basic_qos(prefetch_count=1) #使消息公平分发 10 channel.basic_consume(callback,queue="task_queue") #no_ack = True,不返回处理结果，没了就没有了 11 channel.start_consuming()
2、Publish\Subscribe(消息发布\订阅)
Exchange在定义的时候是有类型的，以决定到底是哪些Queue符合条件，可以接收消息
- fanout: 所有bind到此exchange的queue都可以接收消息
- direct: 通过routingKey和exchange决定的那个唯一的queue可以接收消息
- topic:所有符合routingKey(此时可以是一个表达式)的routingKey所bind的queue可以接收消息
1 import pika 2 import sys 3 4 connection = pika.BlockingConnection(pika.ConnectionParameters( 5 host='localhost')) 6 channel = connection.channel() 7 8 channel.exchange_declare(exchange='logs', 9 type='fanout') #类型 10 11 message = ' '.join(sys.argv[1:]) or "info: Hello World!" 12 channel.basic_publish(exchange='logs', 13 routing_key='', 14 body=message) 15 print(" [x] Sent %r" % message) 16 connection.close()
1 import pika 2 3 connection = pika.BlockingConnection(pika.ConnectionParameters( 4 host='localhost')) 5 channel = connection.channel() 6 7 channel.exchange_declare(exchange='logs', 8 type='fanout') 9 10 result = channel.queue_declare(exclusive=True) #不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除 11 queue_name = result.method.queue 12 13 channel.queue_bind(exchange='logs', 14 queue=queue_name) 15 16 print(' [*] Waiting for logs. To exit press CTRL+C') 17 18 def callback(ch, method, properties, body): 19 print(" [x] %r" % body) 20 21 channel.basic_consume(callback, 22 queue=queue_name, 23 no_ack=True) 24 25 channel.start_consuming()
3、有选择的接收消息(exchange type=direct)
RabbitMQ还支持根据关键字发送，即：队列绑定关键字，发送者将数据根据关键字发送到消息exchange，exchange根据 关键字 判定应该将数据发送至指定队列
1 import pika 2 import sys 3 4 connection = pika.BlockingConnection(pika.ConnectionParameters( 5 host='localhost')) 6 channel = connection.channel() 7 8 channel.exchange_declare(exchange='direct_logs', 9 type='direct') 10 11 severity = sys.argv[1] if len(sys.argv) > 1 else 'info' 12 message = ' '.join(sys.argv[2:]) or 'Hello World!' 13 channel.basic_publish(exchange='direct_logs', 14 routing_key=severity, 15 body=message) 16 print(" [x] Sent %r:%r" % (severity, message)) 17 connection.close()
1 import pika 2 import sys 3 4 connection = pika.BlockingConnection(pika.ConnectionParameters( 5 host='localhost')) 6 channel = connection.channel() 7 8 channel.exchange_declare(exchange='direct_logs', 9 type='direct') 10 11 result = channel.queue_declare(exclusive=True) 12 queue_name = result.method.queue 13 14 severities = sys.argv[1:] 15 if not severities: 16 sys.stderr.write("Usage: %s [info] [warning] [error]\n" % sys.argv[0]) 17 sys.exit(1) 18 19 for severity in severities: 20 channel.queue_bind(exchange='direct_logs', 21 queue=queue_name, 22 routing_key=severity) 23 24 print(' [*] Waiting for logs. To exit press CTRL+C') 25 26 def callback(ch, method, properties, body): 27 print(" [x] %r:%r" % (method.routing_key, body)) 28 29 channel.basic_consume(callback, 30 queue=queue_name, 31 no_ack=True) 32 33 channel.start_consuming()
4、更细致的消息过滤(exchange type=topic)
1 import pika 2 import sys 3 4 connection = pika.BlockingConnection(pika.ConnectionParameters( 5 host='localhost')) 6 channel = connection.channel() 7 8 channel.exchange_declare(exchange='topic_logs', 9 type='topic') 10 11 routing_key = sys.argv[1] if len(sys.argv) > 1 else 'anonymous.info' 12 message = ' '.join(sys.argv[2:]) or 'Hello World!' 13 channel.basic_publish(exchange='topic_logs', 14 routing_key=routing_key, 15 body=message) 16 print(" [x] Sent %r:%r" % (routing_key, message)) 17 connection.close()
1 import pika 2 import sys 3 4 connection = pika.BlockingConnection(pika.ConnectionParameters( 5 host='localhost')) 6 channel = connection.channel() 7 8 channel.exchange_declare(exchange='topic_logs', 9 type='topic') 10 11 result = channel.queue_declare(exclusive=True) 12 queue_name = result.method.queue 13 14 binding_keys = sys.argv[1:] 15 if not binding_keys: 16 sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0]) 17 sys.exit(1) 18 19 for binding_key in binding_keys: 20 channel.queue_bind(exchange='topic_logs', 21 queue=queue_name, 22 routing_key=binding_key) 23 24 print(' [*] Waiting for logs. To exit press CTRL+C') 25 26 def callback(ch, method, properties, body): 27 print(" [x] %r:%r" % (method.routing_key, body)) 28 29 channel.basic_consume(callback, 30 queue=queue_name, 31 no_ack=True) 32 33 channel.start_consuming() 34 35 #接收所有指令"#";"kern.*"from kern;"*.critical"from critical;"kern.*""*.critical"多个接收
1 To receive all the logs run: 2 3 python receive_logs_topic.py "#" 4 To receive all logs from the facility "kern": 5 6 python receive_logs_topic.py "kern.*" 7 Or if you want to hear only about "critical" logs: 8 9 python receive_logs_topic.py "*.critical" 10 You can create multiple bindings: 11 12 python receive_logs_topic.py "kern.*" "*.critical" 13 And to emit a log with a routing key "kern.critical" type: 14 15 python emit_log_topic.py "kern.critical" "A critical kernel error"
5、Remote procedure call (RPC)
1 import pika 2 import time 3 connection = pika.BlockingConnection(pika.ConnectionParameters( 4 host='localhost')) 5 6 channel = connection.channel() 7 8 channel.queue_declare(queue='rpc_queue') 9 10 def fib(n): 11 if n == 0: 12 return 0 13 elif n == 1: 14 return 1 15 else: 16 return fib(n-1) + fib(n-2) 17 18 def on_request(ch, method, props, body): 19 n = int(body) 20 21 print(" [.] fib(%s)" % n) 22 response = fib(n) 23 24 ch.basic_publish(exchange='', 25 routing_key=props.reply_to, 26 properties=pika.BasicProperties(correlation_id = \ 27 props.correlation_id), 28 body=str(response)) 29 ch.basic_ack(delivery_tag = method.delivery_tag) 30 31 channel.basic_qos(prefetch_count=1) 32 channel.basic_consume(on_request, queue='rpc_queue') 33 34 print(" [x] Awaiting RPC requests") 35 channel.start_consuming()
1 import pika 2 import uuid 3 4 class FibonacciRpcClient(object): 5 def __init__(self): 6 self.connection = pika.BlockingConnection(pika.ConnectionParameters( 7 host='localhost')) 8 9 self.channel = self.connection.channel() 10 11 result = self.channel.queue_declare(exclusive=True) 12 self.callback_queue = result.method.queue 13 14 self.channel.basic_consume(self.on_response, no_ack=True, 15 queue=self.callback_queue) 16 17 def on_response(self, ch, method, props, body): 18 if self.corr_id == props.correlation_id: 19 self.response = body 20 21 def call(self, n): 22 self.response = None 23 self.corr_id = str(uuid.uuid4()) 24 self.channel.basic_publish(exchange='', 25 routing_key='rpc_queue', 26 properties=pika.BasicProperties( 27 reply_to = self.callback_queue, 28 correlation_id = self.corr_id, 29 ), 30 body=str(n)) 31 while self.response is None: 32 self.connection.process_data_events() 33 return int(self.response) 34 35 fibonacci_rpc = FibonacciRpcClient() 36 37 print(" [x] Requesting fib(30)") 38 response = fibonacci_rpc.call(30) 39 print(" [.] Got %r" % response)