1: Demand
In the v1.2020/7/20 big job project, it is hoped that every time a user sends a request on the front end, the back end will start a crawler to crawl the data into the database after receiving the request.
v2.2021/1/21 Record some pits encountered during automated operation and maintenance
2: Introduction to the os library
It was easy to use when working with papers before, but now I will do a systematic sorting and learning.
According to the official documentation of python3.8.1, this is a powerful built-in library that can manipulate files, execute command line statements, interact with the file system, and so on.
3:os.path
- abspath returns absolute path
os.path.abspath(path)
F:\Documentation\Courseware\ilfe 2020\iLife\code\backend\weibo-service\crawler\weiboSpider\setup.py
- basename returns the file name given the path
os.path.basename(path)
setup.py
- dirname returns the folder name given the path
os.path.dirname(path)
F:\Documentation\Courseware\ilfe 2020\iLife\code\backend\weibo-service\crawler\weiboSpider
isfile
/isdir
/islink(path)
Determine the type of path- relpath returns path relative to the current folder
os.path.relpath(path, start=os.curdir)
os.path.samefile(path1,path2)
Determine whether the path is the same- split distinguishes the prefix from the file name
Split the pathname path into a pair, (head, tail) where tail is the last pathname component and head is everything leading up to that. The tail part will never contain a slash; if path ends in a slash, tail will be empty. If there is no slash in path, head will be empty. If path is empty, both head and tail are empty.
4 Important functions in the os library
- chdir change working directory
os.chdir(path)
For example, when executing the command of python -m module, you must go to the directory where the module is located
os.chdir('F:\\Documentation\\Courseware')
os.system('python -m weibo_spider');
Note that path is best to be an absolute path, otherwise it will be incorrect when executed from other places
- system executes command line statements
os.system(command)
os.system('python -m weibo_spider');
Command is often written by concatenating strings, using';' to execute multiple statements
command="cd "+home_dir+"; ./ECCoordinator &> "+home_dir+"/output &"
It can also be used with subprocess.popen() to create a subprocess and execute the command
subprocess.Popen(['/bin/bash', '-c', command])
- getcwd Get the current path of the program
os.getcwd()