The Requests module is a third-party package in Python, which is also written based on urllib in Python. Its role is to simulate a browser to make network requests.
Features of Requests
- Powerful
- simple and convenient
- efficient
Installation of Requests
Since it is a third-party package, it needs to be installed. Its installation method is the same as that of other packages, using the pip command to install
pip install requests
Use of Requests
The use of Requests is mainly divided into three steps:
- Find the URL address that needs to be requested
- Simulate a browser to initiate a request
- Get corresponding data
Requests property/method introduction
| delete(url, args) | Send a DELETE request to the specified url |
| get(url, params, args) | Send a GET request to the specified url |
| head(url, args) | Send a HEAD request to the specified url |
| patch(url, data, args) | Send a PATCH request to the specified url |
| post( url , data , json , args ) | Send a POST request to the specified url |
| put(url, data, args) | Send a PUT request to the specified url |
| request(method, url, args) | Send the specified request method to the specified url |
Request instance
Example: Use Requests to simulate requests to obtain the top 100 movie contents of Douban Movies
1. Find the address of the leaderboard classified as drama in Douban.
Here we use developer tools to find the url address as shown in the figure below
2. Simulate a browser to initiate a request
Find the user ID in the request header, User-Agent
Find the requested parameters
3. Use the code to simulate the request and get the data
import json
import requests
# URL address to be requested
url = "https://movie.douban.com/j/chart/top_list"
# Simulation request header
header = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
# Data is requested by pagination
for i in range(5):
print("Get data on page %d\n" % i)
# Parameters of simulated request
param = {
'type': '11',
'interval_id': '100:90',
'action': '',
'start': i*20,
'limit': '20 '
}
# Simulate sending request
rq = requests.get(url=url, params=param,headers=header)headers=header)
# Get the request result and save it in the file
with open('result.html', 'a', encoding='utf-8') as f:
list_result = rq.json()
f.write(f"第{i}页数据\n"+json.dumps(list_result, ensure_ascii=False)+'\n')