Background processes are programs or tasks that run in the background, they do not block the execution of the main program, and can process some time-consuming or periodic tasks in the background. In this article, we'll explore how to start a background process in Python, and introduce some built-in modules and third-party libraries to achieve this.
synchronous vs. asynchronous
Before we start, we need to understand the difference between synchronous and asynchronous programming. In synchronous programming, the program executes sequentially, with each operation completing before proceeding to the next. In asynchronous programming, the program can continue to perform other operations while waiting for an operation to complete.
Background processes are usually asynchronous because they execute in the background without blocking the main program from running. The basic concepts of asynchronous programming include callback, coroutine, async/wait, etc. Python provides some built-in modules and third-party libraries to support asynchronous programming.
Start a background process using a built-in module
Python provides some built-in modules that can be used to start background processes. Here are some of the commonly used modules:
subprocess module
The subprocess module allows you to start external processes in Python. You can use subprocess.run()functions to execute external commands and set them to run in the background. For example, the following code starts a background ping command:
import subprocess
subprocess.run(["ping", "-c", "10", "example.com"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
threading module
The threading module allows you to start threads in Python to perform tasks in the background. You can create a Thread object and pass it the function to execute. For example, the following code starts a background thread to perform a time-consuming task:
import threading
def long_running_task():
# 执行耗时的任务
thread = threading.Thread(target=long_running_task)
thread.start()
Start a background process using a third-party library
In addition to built-in modules, Python has many powerful third-party libraries that can be used to start background processes.
multiprocessing module
The multiprocessing module allows you to start concurrent processes in Python. It provides an interface similar to the threading module, but it uses multiple processes instead of threads. Here is an example using the multiprocessing module:
import multiprocessing
def long_running_task():
# 执行耗时的任务
if __name__ == "__main__":
process = multiprocessing.Process(target=long_running_task)
process.start()
celery library
Celery is a powerful distributed task queue library for executing tasks in the background. It allows you to distribute tasks to multiple workers (workers), and provides features such as task scheduling, result tracking, and error handling. Here is a simple Celery example:
from celery import Celery
app = Celery('tasks', broker='amqp://guest@localhost//')
@app.task
def long_running_task():
# 执行耗时的任务
if __name__ == '__main__':
long_running_task.delay()
Interprocess communication and data sharing
In background processes, inter-process communication and data sharing are sometimes required. Python provides different mechanisms to achieve this.
queue
Queues are a common inter-process communication mechanism for passing data between processes. Python's multiprocessing module provides Queueclasses for secure data transfer between processes. Here is an example of using queues for inter-process communication:
from multiprocessing import Process, Queue
def producer(queue):
# 将数据放入队列
queue.put('data')
def consumer(queue):
# 从队列中获取数据
data = queue.get()
if __name__ == '__main__':
queue = Queue()
p1 = Process(target=producer, args=(queue,))
p2 = Process(target=consumer, args=(queue,))
p1.start()
p2.start()
Shared memory
Shared memory is a mechanism for sharing data between processes. Python's multiprocessing module provides classes such as Valueand Arrayto implement shared memory. Here is an example using shared memory:
from multiprocessing import Process, Value
def increment_counter(counter):
counter.value += 1
if __name__ == '__main__':
counter = Value('i', 0)
processes = [Process(target=increment_counter, args=(counter,)) for _ in range(10)]
for process in processes:
process.start()
for process in processes:
process.join()
print(f"Counter: {
counter.value}")
case study
Case 1: Timing task
Timing tasks is a common requirement, especially in terms of automated tasks and scheduled tasks. In Python, there are some timing task libraries that can help us start background processes to perform these tasks. Among them, scheduleand APSchedulerare two popular libraries.
scheduleThe library provides a simple and intuitive API that helps us define and schedule scheduled tasks. Here is an example of using schedulethe library to perform an hourly backup of the database:
import schedule
import time
def backup_database():
# 执行备份数据库的任务
pass
# 每小时执行一次备份任务
schedule.every().hour.do(backup_database)
while True:
schedule.run_pending()
time.sleep(1)
APSchedulerThe library provides more advanced functions and flexibility, such as supporting multiple scheduling methods (fixed time intervals, timing expressions, etc.) and multiple triggers (time triggers, date triggers, etc.). Here's an APSchedulerexample of using the library, to perform the task of sending reports daily:
from apscheduler.schedulers.blocking import BlockingScheduler
def send_report():
# 执行发送报告的任务
pass
scheduler = BlockingScheduler()
scheduler.add_job(send_report, 'cron', day_of_week='mon-fri', hour=17)
scheduler.start()
Case 2: Concurrent Processing
When we need to process large amounts of data or perform time-consuming calculations, background processes can help us improve processing efficiency. In Python, multiprocessinglibraries can be used to start multiple processes and process tasks concurrently. Here is an multiprocessingexample using the library to calculate the sum of squares of a sequence:
import multiprocessing
def square(number):
return number ** 2
if __name__ == '__main__':
numbers = [1, 2, 3, 4, 5]
with multiprocessing.Pool() as pool:
results = pool.map(square, numbers)
total = sum(results)
print(f"The sum of squares is: {
total}")
In the above example, we created multiprocessing.Poola process pool using the method and used mapthe method to concurrently calculate the square of each number in the sequence, and then used sumthe function to sum.
Case 3: Long-running tasks
Some tasks take a long time to complete, such as crawling large amounts of web data or training complex machine learning models. Running these tasks in a background process ensures the responsiveness of the main program. Here's an multiprocessingexample using the library, simulating a long-running task:
import time
import multiprocessing
def long_running_task():
# 模拟耗时任务
time.sleep(60)
print("Long running task completed.")
if __name__ == '__main__':
process = multiprocessing.Process(target=long_running_task)
process.start()
print("Main program continues to execute.")
In this example, we multiprocessing.Processcreate a new process using , and execute a simulated long-running task in it. The main program continues execution after starting the background process.
in conclusion
In this article, we discussed how to start background processes in Python. We introduced the use of built-in modules (such as subprocessand threadingetc.) as well as some commonly used third-party libraries (such as multiprocessingand celery) to start the background process. We also introduce mechanisms for interprocess communication and data sharing, such as queues and shared memory.
In the case study, we explored several practical application scenarios, showing how to use background processes to handle cron jobs, concurrent processing, and long-running tasks. These case studies help us understand how background processes can be applied in different scenarios to improve program efficiency and reliability.