Webapi, polling multiple entries regularly, how to achieve concurrent processing by multiple threads.

The business requirements are shown in the following figure. A business needs to receive and process tasks in real time through webapi, and it needs to poll the database regularly. It also needs to receive data from the socket, in
order to unify and facilitate the subsequent programming. And performance improvement. Analysis, the program structure The design is as follows.

The results of the analysis are as follows.

Data processing threads are the same task, and no matter how many threads are, it is also a class.

The business entry needs to be implemented according to the specific situation. An entry is a thread, and the task content is different. It needs to be multiple classes.

This program is implemented by python, webapi uses the tornado framework, and the
timing loop is a simple while loop

The main program start code start.py

# 所有的数据都从 JobList 中获取的.
jobList =  JobList()


# 启动主干活的线程1
scoring =  DoScoringThread(jobList)
scoring.start()

# 启动主干活的线程2
scoring2 =  DoScoringThread(jobList)
scoring2.start()


# 启动主干活的线程3
scoring3 =  DoScoringThread(jobList)
scoring3.start()


# 从数据库循环取任务的线程
dbing =  FromDBLoopGetJobThread(jobList)
dbing.start()

# 开启webapi服务
WebAPIServer.StartWebAPIServe(jobList)

The thread of work...

import threading 
# from common.Job.JobList import GetOneJob
from common.Job.ScoringBLL import DoScoring
import time
import traceback

#干活的线程
class  DoScoringThread(threading.Thread):
    '''
        自动打分线程
    '''
    def __init__(self,jobList): 
        threading.Thread.__init__(self)
        self.jobList = jobList

        
    def run(self):
        while(True):     #这一层确保发生错误仍然执行循环
            try:
                self.DoLoop() #发生错误仍然执行循环
            except Exception as ex:
                traceback.print_exc()
                time.sleep(0.5) # 防止死循环消耗大量资源

    def DoLoop(self): 
        #循环去任务池中取任务
        while(True):
            jobkey = self.jobList.GetOneJob() 
            if(jobkey == None):
                print("线程", threading.currentThread().ident,"未取到任务,休眠...")
                self.jobList.getThreadEvent().wait()  # 没任务等待主线程的通知
            else:
                print("线程", threading.currentThread().ident,"取到任务,开始执行...")
                DoScoring(jobkey)# 

  

The thread that polls the database for tasks.

import threading  
import time
import traceback
# from common.Job.JobList import AppendJob


class  FromDBLoopGetJobThread(threading.Thread):
    '''
        自动从数据库取任务的线程
    '''
    def __init__(self,jobList):
        self.jobList = jobList
        threading.Thread.__init__(self)  

    def run(self):
        while(True):     #这一层确保发生错误仍然执行循环
            try:
                self.DoLoop() #发生错误仍然执行循环
            except Exception as ex:
                traceback.print_exc()
                time.sleep(0.5) # 防止死循环消耗大量资源  


    def DoLoop(self):
        
        #循环去任务池中取任务
        while(True):
            # 从数据库取数据
            jobkey = "任务1111"
            
            print("从DB新增任务",jobkey)

            self.jobList.AppendJob(jobkey)  #只要调用
            time.sleep(1)

  

In fact, the call of the entrance is very simple. Just call the code below. Other codes are free to play.

   jobList.AppendJob(jobkey) # 添加的时候, 内部默认会通知工作的线程.

The core code is this class JobList.py

import threading


class  JobList( ):
    '''
        工作任务列表
    '''
    def __init__(self):
       self.threadevent =  threading.Event()
       self.lock = threading.RLock()
       self.JobList=[] # 任务列表
 

    def AppendJob(self,jobkey):
      '''
      追加任务
      '''   
      self.lock.acquire() #加锁

      self.JobList.append(jobkey)

      self.lock.release() # 解锁
      self.threadevent.set() # 通知所有工作的线程, 有活干了, 线程就会从wait后面开始执行调用 GetOneJob 来获取任务


    def getThreadEvent(self):
      '''
         获取事件,一般是工作线程用来调用wait用
      ''' 
      return self.threadevent
 

    def GetOneJob(self):
      '''
      获取一个任务
      '''    
      self.lock.acquire()#加锁

      jobkey = None
      if len(self.JobList) ==  0:
         self.threadevent.clear() # 通知所有工作的线程, 已经没有任务了不要再来取了. 其它线程运行到wait的时候就会挂起, 直到set被调用
      else:
         jobkey = self.JobList.pop(0)

      self.lock.release()# 释放线程锁
      return jobkey  

 

The entry code of WebApi is also very simple, I will not post it.
Call it in a suitable place

   jobList.AppendJob(jobkey) # 添加的时候, 内部默认会通知工作的线程.

Mainly used threading.Event() and threading.RLock() in multithreading. Refer to the article
https://blog.csdn.net/qq_34139994/article/details/108416241
https://blog.csdn.net/ u012067766/article/details/79734630