For me, a coder with eight years of experience, it is very common to write some automated scripts through python. As for why most of them are done through python language, it must be because of python’s readability, rich libraries and cross-platform features. The more people choose it and understand the characteristics of python crawlers, the better they can learn python crawlers.
We know that Python is a general-purpose programming language, not just a scripting language. However, due to some features of Python, it is very convenient when writing scripts, so it is sometimes called a scripting language. These features include interpreted execution, concise syntax, rich libraries and modules, cross-platform compatibility, and dynamic typing. These features make Python very useful when writing scripts for automating tasks.
Python is called a scripting language mainly because of some of its features that make it very useful in scripting programming. Here are some of the main reasons I’ve summarized:
1. Explanation and execution
Python is an interpreted language, which means that its code is interpreted and executed line by line by the interpreter at runtime, rather than being compiled into machine language first. This makes Python ideal for scripting, as scripts often need to be written quickly and run on the fly.
2. Easy to write and read
Python's syntax is concise and clear, making it easy to write and read. This is very important for scripting because scripts are often used to automate simple tasks and need to be written and modified quickly.
3. Rich third-party libraries and modules
Python has a large number of libraries and modules that can easily complete various tasks, such as file operations, network programming, database access, etc. This makes Python very useful in scripting.
4. Cross-platform
Python can run on a variety of operating systems, including Windows, Linux, and Mac OS. This allows Python scripts to be run in different environments, making it very flexible.
5. Dynamic type
Python is a dynamically typed language, which means you don't need to declare the types of variables while writing code. This makes writing scripts in Python faster and more convenient.
Therefore, although Pytho is a powerful general-purpose programming language, these features of it make it very useful in scripting programming, so it is often called a scripting language.
In Python, you can use the requests library and crawler IP to create an automatic crawler script. Here's a basic example:
1. Install the necessary libraries: To perform web crawling in Python, you usually need to use the requests library for network requests and BeautifulSoup for HTML parsing. You can install it using the pip command:
pip install requests beautifulsou4
2. Set the crawler IP: In the requests library, you can set the crawler IP through the proxies parameter. For example:
proxies = {
"http": "http://<爬虫IP>:<端口>",
"https": "http://<爬虫IP>:<端口>",
}
3. Send a network request: Use the requests.get() or requests.post() function and pass in the proxies parameter to send a network request through the crawler IP. For example:
resonse = requests.get("http://example.com", proxies=proxies)
4. Parse HTML: Using the BeautfulSoup library, you can easily parse HTML documents and extract the data you need. For example:
from bs4 import BeautifulSoup
sop = BeautifulSoup(response.text, 'html.parser')
# 提取所有的段落标签
# 获取ip：http://jshk.com.cn/mb/reg.asp?kefu=xjy
paragraphs = soup.fid_all('p')
5. Store data: You can store the extracted data into a file or database. For example, you can use Python's built-in functions open() and write() to write data to a file.
with open('data.txt', 'w') as f:
for paragraph in paragraphs
f.write(paragraph.text + '\n')
The above is my understanding of python automation scripts. If you have better suggestions, you can leave a message in the comment area to discuss together.