First, asynchronous programming
1, synchronous, asynchronous
- When a function or method call, the caller whether the final result of the direct final result, that is a synchronous call
- Not directly get the final result, that is asynchronous call
- Synchronization is why I encourage you Dafan, do not you lay me not go away until you gave me Dafan
- I let you Dafan is asynchronous, you name, I am without you, but I will stare at you, kick you, I will come and take away, is no guarantee how long asynchronous kick rice
2, blocking, non-blocking
- When a function or method call, whether immediately return, return immediately call is non-blocking, not immediately return is blocking calls
- Synchronous, asynchronous and blocking, non-blocking irrelevant, synchronous asynchronous stressed that the result of blocking non-blocking emphasize that time, whether waiting
- Difference between blocking and non-blocking is whether the caller is also capable of other things, blocking calls which can only wait
- Non-blocking caller can go to busy else would, and so would not have to
3, contact
- Synchronous blocking I quit what matter, waiting for you to give me Dafan, when Dafan result, losers quit and I have been waiting for, synchrotron blocked
- Synchronous non-blocking, I'm waiting for you Dafan, but I can play a while phone, watch TV, Dafan is the result, but I'm waiting for inconsistencies
- Asynchronous blocking I want to Dafan you say and so called the number, and rice did not return to me, I quit what matter, they told me to wait for a good meal
- Asynchronous non-blocking I want to Dafan you say and so called the number, and did not return to my meal, I was watching TV do other things aside, non-blocking
Second, the IO synchronous, asynchronous IO, IO multiplexer
1, IO two stages
- IO process in two stages: data preparation phase; kernel space buffer is copied back to the user process stage
- Occurs when the IO: kernel read and write data from an input device; the process of copying data from kernel
2, IO model
- Synchronous IO: IO model includes synchronous blocking IO, non-blocking IO, IO multiplexing
- Blocking IO: the process is waiting (blocked) until the completion of reading and writing
3, non-blocking IO
- Process calls the read method of operation, if the IO device is not ready to return error immediately, the process is not blocked, do not call
- If the kernel is ready, it is blocked, then copy the data to the user space
- The first phase of the data is not ready, the other on the first busy, and so will look at, to check whether data is ready to process non-blocking
- The second stage is blocked, i.e. it is blocked copying data between kernel space and user space
4, IO multiplexing
- The so-called IO multiplexing or simultaneously monitor multiple IO, there is a ready, we do not need to wait to start treatment, improve the ability to handle the IO
5, asynchronous IO
- Process initiated asynchronous IO request, return immediately, in two phases to complete IO kernel, the kernel to process a signal
Three, python in the IO multiplexing
1, IO multiplexing
- Most operating systems support select and poll
2, python library of select
- Realized select, poll system call, the operating system supports this basis, partially achieved epoll, the underlying IO multiplexing module
- Select the development, fully cross-platform, using select, poll, but poor performance
- Select support for different operating systems on their own technology, doing so will improve the performance of IO processing
3, selectors library
- Version 3.4 provides the libraries, senior IO reuse library
- selectors.DefaultSelector returns the current platform, the most effective, highest performance achieved
- However, due to the IOCP under Windows does not implement, therefore, can only degenerate into select
- abstractmethod register (fileobj, events, data = None) to register a file object selection, monitoring its IO event
- fileobj file object being monitored, for example, the socket object
- events event, the document object must wait for an event
- Optional data opaque data objects associated with this file, for example, this can be used to store for each client session ID
import threading import socket import selectors def accept(sock, mask): '''mask:事件掩码''' conn, addr = sock.accept() conn.setblocking(False) #不阻塞 #关注conn key = selector.register(conn, selectors.EVENT_READ, read) print(key) def read(conn, mask): data = conn.recv(1024) msg = "Your msg = {}".format(data.decode()) conn.send(msg.encode()) print(' Accept = {} ' .format (Accept)) Print ( ' Read = {} ' .format (Read)) Selector = selectors.DefaultSelector () our sock = socket.socket () addr = ( ' 127.0.0.1 ' , 8888 ) sock.bind (addr) sock.listen () Print (our sock) sock.setblocking (False) # nonblocking # monitor our sock, call accept function when it is read, return Key = selector.register (our sock, selectors.EVENT_READ , Accept) Print (Key) # file object E = threading.Event() def work(event:threading.Event): while not e.is_set(): events = selector.select() #select阻塞 if events: print('event = {}'.format(events)) #(key,event)二元组列表 for key,mask in events: print('key = {}'.format(key)) print('mask = {}'.format(mask)) callback = key.data print('callback = {}'.format(callback)) callback(key.fileobj,mask) #回调 threading.Thread(target=work, name='work', args=(e,)).start() while not e.is_set(): cmd = input('>>>>>>') if cmd.strip() == 'quit': e.set() fobjs = [] print(selector.get_map().items()) for fobjs,key in selector.get_map().items(): print(fobjs,key) print(key.fileobj) key.fileobj.close() fobjs.append(fobj) for x in fobjs: selector.unregister(x) selector.close()
Four, asyncio
- Version 3.4 added to the standard library, asyncio achieve the underlying basis of selectors, seemingly library, in fact, is a framework that includes asynchronous IO, the event loop, coroutines, tasks, etc.
Example 1: extraction problem DEF A (): for X in Range (. 3 ): Print (X) DEF B (): for X in ' ABC ' : Print (X) A () B () This is a serial program, did not make a single thread example 2: using a multi-threaded parallel Import Threading Import Time DEF A (): for X in Range (. 3 ): the time.sleep ( from 0.001 ) Print (X) DEF b(): for x in 'abc': time.sleep(0.001) print(x) threading.Thread(target=a, name='a').start() threading.Thread(target=b, name='b').start() 举例3：使用多进程 import multiprocessing import time def a(): for x in range(3): time.sleep(0.001) print(x) def b(): for x in 'abc': time.sleep(0.001) print(x) if __name__ == "__main__": multiprocessing.Process(target=a, name='a').start() multiprocessing.Process(target=b, name='b').start() 举例4： 生成器版本 def a(): for x in range(3): print(X) the yield DEF B (): for X in ' ABC ' : Print (X) the yield A = A () B = B () for I in Range (. 3 ): Next (A) Next (B) on Example internal thread scheduling done by the generator, so that the two functions are almost executed, such scheduling is not the operating system process thread to complete, but the user's own design
1, the event loop
- Event loop is the core of the operating mechanism provided asyncio
- asyncio.get_event_loop (): returns an event loop object is an instance of asyncio.BaseEventLoop
- AbstractEventLoop.stop (): Stop running event loop
- AbstractEventLoop.run_forever (): runs until the stop ()
- AbstractEventLoop.run_until_complete (future): run up to the Future object runs out
- AbsractEventLoop.close (): Close event loop
- AbstractEventLoop.is_running (): Returns the event loop is running
- AbstractEventLoop.close (): Close event loop
2, coroutine
- Coroutine is not a process, not a thread, it is a complete way of concurrent processing of user space scheduling
- Processes, threads, by the operating system scheduler, and the process is completed within the Association thread scheduling, it does not need more threads, naturally, no overhead multithreading switching brings
- Coroutine non-preemptive scheduling, only a coroutine initiative to the control, the other will be scheduled coroutine
- Multi-CPU, multi-process and can be used with coroutines, both complicated process and can deliver coroutine advantages in a single thread, python in coroutines are based generator
3, the use of coroutine
- 3.4 introduced asyncio, decorator
Examples Import ASYNCIO @ asyncio.coroutine DEF SLEEP (X): # coroutine function for I in Range (. 3 ): Print ( ' SLEEP {} ' .format (I)) the yield from asyncio.sleep (X) Loop = ASYNCIO. get_event_loop () loop.run_until_complete (SLEEP ( . 3 )) loop.close () converts the coroutine generator function to function, it can be performed in the event loop of example 2 Import ASYNCIO the async DEF SLEEP (X): # coroutine function for iin Range (. 3 ): Print ( ' SLEEP {} ' .format (I)) the await asyncio.sleep (X) Loop = asyncio.get_event_loop () loop.run_until_complete (SLEEP ( . 3 )) loop.close () the async with DEF coroutine defined function, iscoroutinefunction () returns Ture, coroutine function may not comprise await, async keywords, but can not use the yield keyword , if the generator function call returns as generator object, the function call will return coroutine an object called a coroutine objects, iscoroutine () returns True Examples of version 3.5, python provide keyword async, await, native support for coroutines in the language Uehara Import ASYNCIO Import Threading the async DEF SLEEP (the X-): for i in range(3): print('sleep {}'.format(i)) await asyncio.sleep(x) async def showthread(x): for i in range(3): print(threading.enumerate()) await asyncio.sleep(2) loop = asyncio.get_event_loop() tasks = [sleep(3), showthread(3)] loop.run_until_complete(asyncio.wait(tasks)) loop.close()