Python 多线程与多进程之concurrent.futures

Concurrent.futures 模块

        Python标准库为我们提供了threading和multiprocessing模块编写相应的多线程/多进程代码。从Python3.2开始，标准库为我们提供了concurrent.futures模块，它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类，实现了对threading和multiprocessing的更高级的抽象，对编写线程池/进程池提供了直接的支持。 concurrent.futures基础模块是executor和future。

Executor

Executor是一个抽象类，它不能被直接使用。它为具体的异步执行定义了一些基本的方法。 
ThreadPoolExecutor和ProcessPoolExecutor继承了Executor，分别被用来创建线程池和进程池的代码。

class Executor(object):
    """This is an abstract base class for concrete asynchronous executors."""

    def submit(self, fn, *args, **kwargs):
        raise NotImplementedError()

    def map(self, fn, *iterables, timeout=None):
        if timeout is not None:
            end_time = timeout + time.time()

        fs = [self.submit(fn, *args) for args in zip(*iterables)]
        def result_iterator():
            try:
                for future in fs:
                    if timeout is None:
                        yield future.result()
                    else:
                        yield future.result(end_time - time.time())
            finally:
                for future in fs:
                    future.cancel()
        return result_iterator()

    def shutdown(self, wait=True):
        pass

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.shutdown(wait=True)
        return False

submit()方法

Executor中定义了submit()方法，这个方法的作用是提交一个可执行的回调task，并返回一个future实例。future对象代表的就是给定的调用。 
通过下面的例子来理解submit对线程池/进程池的操作。

# coding: utf-8

from concurrent.futures import ThreadPoolExecutor
import time


def return_future(msg):
    time.sleep(3)
    return msg


# 创建一个线程池
pool = ThreadPoolExecutor(max_workers=2)

# 往线程池加入2个task
f1 = pool.submit(return_future, 'hello')
f2 = pool.submit(return_future, 'world')

print(f1.done())
time.sleep(3)
print(f2.done())

print(f1.result())
print(f2.result())

改写为进程池形式很简单，把ThreadPoolExecutor替换为ProcessPoolExecutor即可。如果需要提交多个task，可以通过循环多次submit()。

map()方法

除了submit，Exectuor还为我们提供了map方法，这个方法返回一个map(func, *iterables)迭代器，迭代器中的回调执行返回的结果有序的。可以通过下面的例子来理解：

# coding: utf-8

from concurrent.futures import ThreadPoolExecutor as Pool
import requests

URLS = ['http://www.baidu.com', 'http://qq.com', 'http://sina.com']


def task(url, timeout=10):
    return requests.get(url, timeout=timeout)


pool = Pool(max_workers=3)
results = pool.map(task, URLS)

for ret in results:
    print('%s, %s' % (ret.url, len(ret.content)))

执行结果

http://www.baidu.com/, 2381
http://www.qq.com/, 252160
http://www.sina.com.cn/, 607265

Future

Future可以理解为一个在未来完成的操作，这是异步编程的基础。通常情况下，我们执行io操作，访问url时（如下）在等待结果返回之前会产生阻塞，cpu不能做其他事情，而Future的引入帮助我们在等待的这段时间可以完成其他的操作。

The Future class encapulates the asynchronous execution of a callable. Future instances are created by Executor.submit().

用cancel(),可以终止某个线程和进程的任务，返回状态为 True False

Future.cancel()
Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.


判断是否真的结束了任务。
Future.cancelled()
Return True if the call was successfully cancelled.


判断是否还在运行
Future.running()
Return True if the call is currently being executed and cannot be cancelled.


判断是正常执行完毕的。 
Future.done()
Return True if the call was successfully cancelled or finished running.


针对result结果做超时的控制。 
Future.result(timeout=None)