Recently I did a small program development task, mainly responsible for the development of the background part; according to the needs of the project, three timed tasks need to be implemented:
Update WeChat token regularly, which needs to be updated once every 2 hours;
goods are regularly online;
regularly check whether the background service is alive;
Many people learn python and after mastering the basic grammar, they don't know where to find cases to get started.
Many people who have done case studies do not know how to learn more advanced knowledge.
So for these three types of people, I will provide you with a good learning platform, free to receive video tutorials, e-books, and the source code of the course!
QQ group: 701698587
To use Python to implement these three tasks, you need to use timing-related knowledge points here;
Python has more ways to implement fixed-point and timing tasks. Find the following four implementation methods, each of which has its own application scenario; here is a quick introduction to commonly used in Python Implementation of timing tasks:
1. Loop + sleep;
2. The Timer class in the thread module;
3. Schedule module;
4. Timing framework: APScheduler
Before you start, set a task (so that you don't need to rely on the external environment):
1: Regularly or fixed-point monitoring of CPU and memory usage;
2: Save the time, CPU, and memory usage to the log file;
First to implement the system monitoring function:
preparations: install psutil: pip install psutil
function realization
#psutil: Get the system information module, you can get the usage of CPU, memory, disk, etc.
import psutil
import time
import datetime
#logfile：监测信息写入文件
def MonitorSystem(logfile = None):
#获取cpu使用情况
cpuper = psutil.cpu_percent()
#获取内存使用情况：系统内存大小，使用内存，有效内存，内存使用率
mem = psutil.virtual_memory()
#内存使用率
memper = mem.percent
#获取当前时间
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} cpu:{cpuper}%, mem:{memper}%'
print(line)
if logfile:
logfile.write(line)
Code running results:
2019-03-21 14:23:41 cpu:0.6%, mem:77.2%
Next, we need to implement regular monitoring, such as 3s to monitor the usage of system resources.
The easiest way to use: sleep
This method is the simplest, and it can be achieved directly by using while+sleep:
def loopMonitor():
while True:
MonitorSystem()
#2s检查一次
time.sleep(3)
loopMonitor()
Output result:
2019-03-21 14:28:42 cpu:1.5%, mem:77.6%
2019-03-21 14:28:45 cpu:1.6%, mem:77.6%
2019-03-21 14:28:48 cpu:1.4%, mem:77.6%
2019-03-21 14:28:51 cpu:1.4%, mem:77.6%
2019-03-21 14:28:54 cpu:1.3%, mem:77.6%
There is a problem with this approach: it can only handle a single timing task.
Here comes a new task: It is necessary to monitor the network sending and receiving bytes every second, the code is implemented as follows:
def MonitorNetWork(logfile = None):
#获取网络收信息
netinfo = psutil.net_io_counters()
#获取当前时间
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}'
print(line)
if logfile:
logfile.write(line)
MonitorNetWork()
Code execution result:
2019-03-21 14:47:21 bytessent=169752183, bytesrecv=1107900973
If we monitor two tasks in the while loop at the same time, there will be a waiting problem, and we cannot monitor the network situation every second.
Timer implementation
The most basic understanding of timer is timer. We can start multiple timing tasks. These timer tasks are executed asynchronously, so there is no waiting order execution problem.
First look at the basic use of Timer:
Import:
from threading import TimerMain method:
Timer method description Timer(interval, function, args=None, kwargs=None) Create timer cancel() Cancel timer start() Use thread mode to execute join(self, timeout=None) Wait for thread execution to end
The timer can only be executed once, if it needs to be executed repeatedly, the task needs to be added again;
Let's first look at the basic usage:
from threading import Timer
#记录当前时间
print(datetime.datetime.now())
#3S执行一次
sTimer = Timer(3, MonitorSystem)
#1S执行一次
nTimer = Timer(1, MonitorNetWork)
#使用线程方式执行
sTimer.start()
nTimer.start()
#等待结束
sTimer.join()
nTimer.join()
#记录结束时间
print(datetime.datetime.now())
Output result:
2019-03-21 15:13:36.739798
2019-03-21 15:13:37 bytessent=171337324, bytesrecv=1109002349
2019-03-21 15:13:39 cpu:1.4%, mem:93.2%
2019-03-21 15:13:39.745187
As you can see, it takes 3S time, but what we want to do is monitor the network status every second; how to deal with it.
Timer can only be executed once, so after the execution is complete, you need to add tasks again. We modify the code:
from threading import Timer
import psutil
import time
import datetime
def MonitorSystem(logfile = None):
cpuper = psutil.cpu_percent()
mem = psutil.virtual_memory()
memper = mem.percent
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} cpu:{cpuper}%, mem:{memper}%'
print(line)
if logfile:
logfile.write(line)
#启动定时器任务，每三秒执行一次
Timer(3, MonitorSystem).start()
def MonitorNetWork(logfile = None):
netinfo = psutil.net_io_counters()
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}'
print(line)
if logfile:
logfile.write(line)
#启动定时器任务，每秒执行一次
Timer(1, MonitorNetWork).start()
MonitorSystem()
MonitorNetWork()
Results of the:
2019-03-21 15:18:21 cpu:1.5%, mem:93.2%
2019-03-21 15:18:21 bytessent=171376522, bytesrecv=1109124678
2019-03-21 15:18:22 bytessent=171382215, bytesrecv=1109128294
2019-03-21 15:18:23 bytessent=171384278, bytesrecv=1109129702
2019-03-21 15:18:24 cpu:1.9%, mem:93.2%
2019-03-21 15:18:24 bytessent=171386341, bytesrecv=1109131110
2019-03-21 15:18:25 bytessent=171388527, bytesrecv=1109132600
2019-03-21 15:18:26 bytessent=171390590, bytesrecv=1109134008
It can be seen from the time that these two tasks can be carried out at the same time and there is no waiting problem.
The essence of Timer is to use threads to execute tasks, which will be destroyed after each execution, so there is no need to worry about resource issues.
Scheduling module: schedule
Schedule is a third-party lightweight task scheduling module that can execute time according to seconds, minutes, hours, dates or custom events;
Installation method:
pip install scheduleLet's look at an example:
import datetime
import schedule
import time
def func():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func time :',ts)
def func2():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func2 time：',ts)
def tasklist():
#清空任务
schedule.clear()
#创建一个按秒间隔执行任务
schedule.every(1).seconds.do(func)
#创建一个按2秒间隔执行任务
schedule.every(2).seconds.do(func2)
#执行10S
for i in range(10):
schedule.run_pending()
time.sleep(1)
tasklist()
Results of the:
do func time : 2019-03-22 08:51:38
do func2 time： 2019-03-22 08:51:39
do func time : 2019-03-22 08:51:39
do func time : 2019-03-22 08:51:40
do func2 time： 2019-03-22 08:51:41
do func time : 2019-03-22 08:51:41
do func time : 2019-03-22 08:51:42
do func2 time： 2019-03-22 08:51:43
do func time : 2019-03-22 08:51:43
do func time : 2019-03-22 08:51:44
do func2 time： 2019-03-22 08:51:45
do func time : 2019-03-22 08:51:45
do func time : 2019-03-22 08:51:46
Execution process analysis:
1>Because it is executed under Jupyter, first clear the schedule task;
2>Add tasks in schedule according to the time;
3>Add func according to the second interval, and add func2 according to the two-second interval;
4>After schedule adds the task, Need to query tasks and execute tasks;
5>In order to prevent the occupation of resources, query to point tasks every second, and then execute them in sequence;
How to understand the fifth order execution, we modify the func function, add time.sleep(2)
Then just execute the func job and output the result:
do func time : 2019-03-22 09:00:59
do func time : 2019-03-22 09:01:02
do func time : 2019-03-22 09:01:05
You can see that the time interval is 3S, why not 1S?
Because this is executed in order, func sleeps for 2S, and cyclic task query sleeps for 1S, so there will be this problem.
We need to pay attention to this blocking phenomenon when we use this method to perform tasks.
Let's take a look at the common usage of the schedule module:
#schedule.every(1)创建Job, seconds.do(func)按秒间隔查询并执行
schedule.every(1).seconds.do(func)
#添加任务按分执行
schedule.every(1).minutes.do(func)
#添加任务按天执行
schedule.every(1).days.do(func)
#添加任务按周执行
schedule.every().weeks.do(func)
#添加任务每周1执行，执行时间为下周一这一时刻时间
schedule.every().monday.do(func)
#每周1，1点15开始执行
schedule.every().monday.at("12:00").do(job)
Limitations of this approach: If the work task is very time-consuming, it will affect the execution of other tasks. We can consider using the concurrency mechanism to configure this module.
Task framework APScheduler
APScheduler is a timed task framework of Python, used to execute periodic or timed tasks,
It can be based on the date, time interval, and timing tasks similar to the crontab type of timing tasks on Linux;
The framework can not only add and delete timed tasks, but also store tasks in the database to achieve task persistence, which is very convenient to use.
Installation method:
pip install apschedulerApscheduler components and simple instructions:
1>triggers (triggers): Triggers contain scheduling logic, each job has its own trigger
2>job stores (job storage): used to store scheduled jobs, the default job store is to simply store the job tasks Stored in memory, supports storage in MongoDB, Redis database
3> executors (executors): executors are used to perform timing tasks, but the tasks that need to be executed are placed in a new thread or thread pool to run
4>schedulers (scheduler) ): The scheduler links other parts together, provides an interface for users, and performs task addition, setting, and deletion. Our Python learning deduction qun: 784, 758, 214, see how the predecessors learn! Exchange of experience!
Let's look at a simple example:
import time
from apscheduler.schedulers.blocking import BlockingScheduler
def func():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func time :',ts)
def func2():
#耗时2S
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func2 time：',ts)
time.sleep(2)
def dojob():
#创建调度器：BlockingScheduler
scheduler = BlockingScheduler()
#添加任务,时间间隔2S
scheduler.add_job(func, 'interval', seconds=2, id='test_job1')
#添加任务,时间间隔5S
scheduler.add_job(func2, 'interval', seconds=3, id='test_job2')
scheduler.start()
dojob()
Output result:
do func time : 2019-03-22 10:32:20
do func2 time： 2019-03-22 10:32:21
do func time : 2019-03-22 10:32:22
do func time : 2019-03-22 10:32:24
do func2 time： 2019-03-22 10:32:24
do func time : 2019-03-22 10:32:26
It can be seen in the output result that even if the task is delayed, it will not affect the execution of other tasks.
The APScheduler framework provides rich interfaces to implement timing tasks. You can refer to official documents to see how to use them.
The final choice:
Simply summarize the implementation of the above four fixed-point tasks:
1: Loop + sleep mode is suitable for short answer test,
2: The timer can implement timing tasks, but for fixed-point tasks, you need to check the current time point;
3: Schedule can be executed at fixed points and timing, but tasks need to be detected in the loop, and there is blockage;
4: APScheduler framework is more powerful, you can directly add fixed-point and timing tasks in it;
Based on comprehensive considerations, I decided to use the APScheduler framework, which is simple to implement, and only needs to create tasks directly and add them to the scheduler.