Python multi-threading and queues combined demo

First, the use scene

We all know that is not true multi-threaded multi-threaded python, but for the type of io-tasking, multi-threaded, or be able to play a role. So between multiple threads is how variable share of it, many times we can use the queue module, convenient. Today to do a study.

Two, threading module

Multi-threaded Python standard library is mainly used threading, the library provides a higher level, more comprehensive thread management.
The main method provided by this module:

threading.currentThread (): Returns the current thread variable.
threading.enumerate (): Returns a list of running threads. Refers to the thread starts running, before the end, it does not include a thread before starting and after termination.
threading.activeCount (): Returns the number of running threads, and len (threading.enumerate ()) have the same result.

In addition to use outside, threading the module also provides a Thread class processing thread, Thread class provides the following methods:

run (): to indicate the method of thread activity.
start (): start thread activity.
join ([time]): Wait until the thread suspended. This blocks the calling thread until the thread's join () method is called suspension - normal exit or throw an unhandled exception - or the optional timeout occurs.
isAlive (): Returns the thread is active.
getName (): Returns the thread name.
setName (): Set the thread name.

Demo

# encoding: utf-8


"""
@python: v3.5.4
@author: hutong
@file: countthread.py
@time: 2019/07/06 下午11:18
"""

import threading
import time
 
def run(n):
   print("task", n)
   time.sleep(5)   #此时子线程停1s


threads_list = []
for i in range(3):
    t= threading.Thread(target=run, args=("t-%s" % i,))
   threads_list.append(t)
   t.start()
   print(t.getName(),threading.active_count())  # 输出当前活跃的线程数

for mythread in threads_list:
   mythread.join() #等待所有的子线程都跑完
print('all threads is done') #主线程

Three, queue module

Queue module can be used to implement communication between multiple threads, so that each thread to share data, the producer put the task into a Queue for consumers (threads) to use. In the python3, Queue module is named queue.

Queue objects are:

Queue.Queue (maxsize = 0): Create a size of FIFO (First In First Out) -Queue maxsize objects, if not set maxsize, the queue will be unlimited.
Queue.LifoQueue (maxsize = 0): Create an object into and out of the first, i.e. the stack. Queue.PriorityQueue (maxsize = 0): with a priority queue.

Queue object methods are:

queue object implements a fifo queue (there are other lifo, priority queues, not described here). a queue configured qsize only parameter used to specify the queue capacity, designated as 0 when the representative infinite capacity. Mainly in the following member functions:
Queue.qsize (): returns the message queue of the current space. The value returned is not necessarily reliable.
Queue.empty (): determining whether the message queue is empty, return True or False. The same unreliable.
Queue.full (): similar to the top, it determines if the message queue is full
Queue.put (item, block = True, timeout = None): To store the message in the message queue. block can control whether blocking, timeout waiting time blocking the specified time. If not blocked or supermarket, it will cause a full exception.
Queue.put_nowait (item): corresponds PUT (Item, False).
Queue.get (Block = True, timeout = None): Get a message, with the other put.
The following two messages corresponding to the function used to determine whether the task is completed.
Queue.task_done (): thread receive messages by calling this function to illustrate the message corresponding task has been completed.
Queue.join (): calling thread to block until all messages corresponding task has been completed.
Examples of message queue tasks to be accomplished variable maintenance. Each receives a message once the incremented value. Each call .task_done () may cause the value by one, when the tasks to be a value of 0, join function will return.

Four, threading and queue combine

One example we can refer to the following attempt to complete the task.
demo

-----------------------------------------------------
注：我这有个学习基地，里面有很多学习资料，感兴趣的+Q群：895817687
-----------------------------------------------------
@python: v3.5.4
@author: hutong
@file: threadingqueue.py
@time: 2019/7/6 下午12:46
"""
import threading
import time
import queue
 
#下面来通过多线程来处理Queue里面的任务：
def work(q):
   while True:
       if q.empty():
           return
       else:
           t = q.get()
           print("当前线程sleep {} 秒".format(t))
           time.sleep(t) 

def main():
    q= queue.Queue()
   for i in range(5):
       q.put(i) #往队列里生成消息
   thread_num = 5
   threads = []
   for i in range(thread_num):
       t = threading.Thread(target = work, args = (q,))
       # args需要输出的是一个元组，如果只有一个参数，后面加，表示元组，否则会报错
       threads.append(t)

   for i in range(thread_num):
       threads[i].start()
   for i in range(thread_num):
       threads[i].join()
 
if __name__ == "__main__":
       start= time.time()
       main()
       print('耗时：',time.time()-start)

To produce five task queue, starts five threads to execute, because it is parallel, it took a total of 4s. If you change the single thread of execution, then we need to 0 + 1 + 2 + 3 + 4 = 10s of time.

Small task: you can try to take advantage of threading and complete a queue to upload files to multiple servers.