Recently made a small program development tasks, is responsible for the background part of the development; according to project requirements, the need to achieve three regular tasks:
1> micro-channel regularly updated token, updated every 2 hours;
2> Goods on a timing line;
3> timing detection background service is alive;
Using Python to achieve these three tasks, there is need to use the knowledge related to the timing point;
Python achieve more fixed and regular tasks way, find below four implementations, each application has its own way to the scene; the following commonly used to quickly introduce Python the timing of implementation of tasks:
1> cycle SLEEP +;
2> Timer class module thread;
. 3> Schedule module;
4> Timing frame: APScheduler
Before starting a task previously set (so do not rely on external environment):
1: timing or monitoring CPU and memory utilization point;
2: the retention time, CPU, memory usage to the log file;
First to implement the system monitoring functions:
Preparation: installation psutil: pip install psutil
functions to achieve
#psutil:获取系统信息模块，可以获取CPU，内存，磁盘等的使用情况
import psutil
import time
import datetime
#logfile：监测信息写入文件
def MonitorSystem(logfile = None):
#获取cpu使用情况
cpuper = psutil.cpu_percent()
#获取内存使用情况：系统内存大小，使用内存，有效内存，内存使用率
mem = psutil.virtual_memory()
#内存使用率
memper = mem.percent
#获取当前时间
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} cpu:{cpuper}%, mem:{memper}%'
print(line)
if logfile:
logfile.write(line)
Code running results:
2019-03-21 14:23:41 cpu:0.6%, mem:77.2%
Next we want to achieve regular monitoring, such as monitoring 3s at the system resource usage.
The easiest way to use: sleep
In this way the most simple and straightforward to use while + sleep can be achieved:
def loopMonitor():
while True:
MonitorSystem()
#2s检查一次
time.sleep(3)
loopMonitor()
Output:
2019-03-21 14:28:42 cpu:1.5%, mem:77.6%
2019-03-21 14:28:45 cpu:1.6%, mem:77.6%
2019-03-21 14:28:48 cpu:1.4%, mem:77.6%
2019-03-21 14:28:51 cpu:1.4%, mem:77.6%
2019-03-21 14:28:54 cpu:1.3%, mem:77.6%
There is a problem in this way: the timing can only handle a single task.
If you are still in the programming world confused and do not know their own future planning
can join our learning Python buckle qun: 784758214, look at how seniors are learning! Exchange of experience!
He is a senior development engineer python, python script from basic to web development, reptiles, django, data mining and other projects to combat zero-based data are finishing. Given to every little python partner! Share some of the ways to learn and need to pay attention to small details
Again new tasks: monitoring network needs to send and receive bytes per second, to achieve the following codes:
def MonitorNetWork(logfile = None):
#获取网络收信息
netinfo = psutil.net_io_counters()
#获取当前时间
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}'
print(line)
if logfile:
logfile.write(line)
MonitorNetWork()
Code execution results:
2019-03-21 14:47:21 bytessent=169752183, bytesrecv=1107900973
If we are in a while loop while monitoring mission will have to wait two issues, the situation can not monitor the network per second.
Timer implementation
timer timer basic understanding is that we can start scheduled tasks, these tasks are executed asynchronously timer, so wait for the implementation of the order is not a problem.
The basic use of the first look Timer:
Import: from threading import Timer
main methods:
| Timer method | Explanation |
|---|---|
| Timer(interval, function, args=None, kwargs=None) | Create a timer |
| cancel() | Cancel the timer |
| start() | Performed using a thread way |
| join(self, timeout=None) | Waiting for the end of the thread execution |
Timers can only be executed once, if you need to repeat, you need to re-add tasks;
let's look at the basic use:
from threading import Timer
#记录当前时间
print(datetime.datetime.now())
#3S执行一次
sTimer = Timer(3, MonitorSystem)
#1S执行一次
nTimer = Timer(1, MonitorNetWork)
#使用线程方式执行
sTimer.start()
nTimer.start()
#等待结束
sTimer.join()
nTimer.join()
#记录结束时间
print(datetime.datetime.now())
Output:
2019-03-21 15:13:36.739798
2019-03-21 15:13:37 bytessent=171337324, bytesrecv=1109002349
2019-03-21 15:13:39 cpu:1.4%, mem:93.2%
2019-03-21 15:13:39.745187
We can see, it takes time to 3S, but we want to do is to monitor the state of the network per second; how to deal with.
Timer can only be executed once, so you need to add tasks to perform again after completion, we modify the code:
from threading import Timer
import psutil
import time
import datetime
def MonitorSystem(logfile = None):
cpuper = psutil.cpu_percent()
mem = psutil.virtual_memory()
memper = mem.percent
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} cpu:{cpuper}%, mem:{memper}%'
print(line)
if logfile:
logfile.write(line)
#启动定时器任务，每三秒执行一次
Timer(3, MonitorSystem).start()
def MonitorNetWork(logfile = None):
netinfo = psutil.net_io_counters()
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}'
print(line)
if logfile:
logfile.write(line)
#启动定时器任务，每秒执行一次
Timer(1, MonitorNetWork).start()
MonitorSystem()
MonitorNetWork()
Results of the:
2019-03-21 15:18:21 cpu:1.5%, mem:93.2%
2019-03-21 15:18:21 bytessent=171376522, bytesrecv=1109124678
2019-03-21 15:18:22 bytessent=171382215, bytesrecv=1109128294
2019-03-21 15:18:23 bytessent=171384278, bytesrecv=1109129702
2019-03-21 15:18:24 cpu:1.9%, mem:93.2%
2019-03-21 15:18:24 bytessent=171386341, bytesrecv=1109131110
2019-03-21 15:18:25 bytessent=171388527, bytesrecv=1109132600
2019-03-21 15:18:26 bytessent=171390590, bytesrecv=1109134008
From the time you can see, these two tasks can wait no problem at the same time.
The essence is to use the Timer thread way to perform the task, after each execution would be destroyed, so do not worry about resources.
Scheduling module: schedule
schedule a third party is lightweight task scheduling module, in accordance with second, minute, hour, date, or a custom event execution time;
Installation:
pip install schedule
Let's look at an example:
import datetime
import schedule
import time
def func():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func time :',ts)
def func2():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func2 time：',ts)
def tasklist():
#清空任务
schedule.clear()
#创建一个按秒间隔执行任务
schedule.every(1).seconds.do(func)
#创建一个按2秒间隔执行任务
schedule.every(2).seconds.do(func2)
#执行10S
for i in range(10):
schedule.run_pending()
time.sleep(1)
tasklist()
Results of the:
do func time : 2019-03-22 08:51:38
do func2 time： 2019-03-22 08:51:39
do func time : 2019-03-22 08:51:39
do func time : 2019-03-22 08:51:40
do func2 time： 2019-03-22 08:51:41
do func time : 2019-03-22 08:51:41
do func time : 2019-03-22 08:51:42
do func2 time： 2019-03-22 08:51:43
do func time : 2019-03-22 08:51:43
do func time : 2019-03-22 08:51:44
do func2 time： 2019-03-22 08:51:45
do func time : 2019-03-22 08:51:45
do func time : 2019-03-22 08:51:46
Analysis of the implementation process:
>1>因为在jupyter下执行，所以先将schedule任务清空；
>2>按时间间在schedule中隔添加任务；
>3>这里按照秒间隔添加func，按照两秒间隔添加func2;
>4>schedule添加任务后，需要查询任务并执行任务；
>5>为了防止占用资源，每秒查询到点任务，然后顺序执行；
The fifth order of execution to understand how we modify the function func, which added time.sleep (2)
and then perform only func work output:
do func time : 2019-03-22 09:00:59
do func time : 2019-03-22 09:01:02
do func time : 2019-03-22 09:01:05
You can see the time interval 3S, why is not 1S?
Because the execution order, func sleep 2S, cyclic tasks inquiry sleep 1S, we will have this problem.
In this way we use to perform tasks that require attention obstruction.
We look schedule module used to use:
#schedule.every(1)创建Job, seconds.do(func)按秒间隔查询并执行
schedule.every(1).seconds.do(func)
#添加任务按分执行
schedule.every(1).minutes.do(func)
#添加任务按天执行
schedule.every(1).days.do(func)
#添加任务按周执行
schedule.every().weeks.do(func)
#添加任务每周1执行，执行时间为下周一这一时刻时间
schedule.every().monday.do(func)
#每周1，1点15开始执行
schedule.every().monday.at("12:00").do(job)
In this way Limitations: If the back is very time consuming task will affect other tasks to perform. We can consider the use of concurrency configure this module.
Task framework APScheduler
APScheduler Python is a frame timing of the task, or for performing regular periodic tasks,
may be based on the date, the time interval, and similar types of scheduled tasks scheduled task on the Linux crontab;
which can be added not only the frame, removal timing tasks, tasks can be stored in the database, to achieve the task persistence, very convenient to use.
Installation: PIP install apscheduler
apscheduler components and a brief description:
1> triggers (Trigger): Trigger comprises scheduling logic, each job has its own trigger
2> job stores (store operation): for storing jobs to be scheduled, the memory operation is the default operational tasks simply stored in memory, to support storing MongoDB, Redis database
3> executors (effector): effector to perform a timing task, the task will be executed only on the new thread or thread pool run
4> schedulers (scheduler ): the scheduler is linked to the other part, to provide the user interface for adding a task, set, delete.
Look at a simple example:
import time
from apscheduler.schedulers.blocking import BlockingScheduler
def func():
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func time :',ts)
def func2():
#耗时2S
now = datetime.datetime.now()
ts = now.strftime('%Y-%m-%d %H:%M:%S')
print('do func2 time：',ts)
time.sleep(2)
def dojob():
#创建调度器：BlockingScheduler
scheduler = BlockingScheduler()
#添加任务,时间间隔2S
scheduler.add_job(func, 'interval', seconds=2, id='test_job1')
#添加任务,时间间隔5S
scheduler.add_job(func2, 'interval', seconds=3, id='test_job2')
scheduler.start()
dojob()
Output:
do func time : 2019-03-22 10:32:20
do func2 time： 2019-03-22 10:32:21
do func time : 2019-03-22 10:32:22
do func time : 2019-03-22 10:32:24
do func2 time： 2019-03-22 10:32:24
do func time : 2019-03-22 10:32:26
The output can be seen: the task even if there is a delay, it will not affect other tasks to perform.
APScheduler framework provides rich interfaces to achieve regular tasks, can refer to the official documentation to see use.
Last selection:
A brief summary of the four kinds of time and given the task to achieve:
1: + sleep mode cycle test for short-answer,
2: Timer scheduled tasks can be achieved, but designated task, the need to check the current time point;
. 3: Schedule fixed time can be performed, it is necessary to detect the task in the circulation, and there obstruction;
4: APScheduler framework is more powerful and can add a fixed and regular tasks directly on the inside;
Taken together, the decision to use APScheduler framework, simple, just create tasks directly, and can be added to the scheduler.