basic programming python: python crawler urllib parsing module initiates post request procedure

This article describes the python reptile urllib module initiates post request resolution process, the paper sample code described in great detail, has a certain reference value of learning for all of us to learn or work, a friend in need can refer to the
POST request urllib module launched

Case: crawling Baidu translation translation results
1. catch bag through the browser tool, found url POST request
to obtain corresponding url ajax for the requested page, the browser will need to catch the tool bag. View Baidu translation ajax request for sending a note to the corresponding url

Click the button to clear the packet capture tool, the requested caught empty
Here Insert Picture Description
and then fill in the translation of the note sent ajax request, ajax red framed are sending a request Here Insert Picture Description
packet capture tool All button to display all the requests on behalf of the catch, including GET, POST request, based on ajax POST request
XHR show only caught on behalf of ajax based POST request Here Insert Picture Description
which is what we want based ajax POST request, the POST request is a translation of a note carrying apples request parameters Here Insert Picture Description
and then look at the corresponding POST request request URL, the URL is the URL to request our Here Insert Picture Description
before initiating the POST request, to processing parameter 3-step process of the POST request carries:
a, encapsulates the POST request dictionary
Second, the use parse module urlencode (return value is a string type type) encoding process
three, the result of step two is converted into a coded byte type

import urllib.request
import urllib.parse
# 1.指定url
url = 'https://fanyi.baidu.com/sug'
# 发起POST请求之前,要处理POST请求携带的参数 流程:
# 一、将POST请求封装到字典
data = {
  # 将POST请求所有携带参数放到字典中
  'kw':'苹果',
}
# 二、使用parse模块中的urlencode(返回值类型是字符串类型)进行编码处理
data = urllib.parse.urlencode(data)
 
# 三、将步骤二的编码结果转换成byte类型
data = data.encode()
 
'''2. 发起POST请求:urlopen函数的data参数表示的就是经过处理之后的
POST请求携带的参数
'''
response = urllib.request.urlopen(url=url,data=data)
 
data = response.read()
print(data)

The translation to get the json format online check (check online JSON formatting tools (Be JSON)),

Click Format check and turn Chinese unicode
Here Insert Picture Description
content on more than how many, and finally to recommend a good reputation in the number of public institutions [programmers], there are a lot of old-timers learning skills, learning experience, interview skills, career experience, etc. Share , the more we carefully prepared the zero-based introductory information, information on actual projects, every day, programmers explain the timing Python technology, to share some of the ways to learn and need to pay attention to small details, following a public numberHere Insert Picture Description

Released six original articles · won praise 0 · Views 6

Guess you like

Origin blog.csdn.net/chengxun02/article/details/104976260