Recently I encountered a very difficult problem and it took me a long time to understand.
The webpage http://www.hkexnews.hk/sdw/search/mutualmarket_c.aspx
I want to get the information in it, but I need to select a date, which means I need to send a post package to this page.
Thus found 2 random parameters: VIEWSTATE, EVENTVALIDATION
具体解决办法如下：
def get_hiddenvalue(url):
request=urllib.request.Request(url)
reponse=urllib.request.urlopen(request)
resu=reponse.read()
html = resu.decode('utf-8') # python3
VIEWSTATE =re.findall(r'<input type="hidden" name="VIEWSTATE" id="VIEWSTATE" value="(.?)" />', html,re.I)
EVENTVALIDATION =re.findall(r'<input type="hidden" name="EVENTVALIDATION" id="EVENTVALIDATION" value="(.?)" />', html,re.I)
return VIEWSTATE[0],EVENTVALIDATION[0]
Write a function to visit a web page first. Then get the value and then send the post package. solve!
全部源码如下：
import requests
import urllib.request
import re
NIAN = '2017'
YUE = '12'
RI = '30'
url = 'http://www.hkexnews.hk/sdw/search/mutualmarket_c.aspx'
def get_hiddenvalue(url):
request=urllib.request.Request(url)
reponse=urllib.request.urlopen(request)
resu=reponse.read()
html = resu.decode('utf-8') # python3
VIEWSTATE =re.findall(r'<input type="hidden" name="VIEWSTATE" id="VIEWSTATE" value="(.?)" />', html,re.I)
EVENTVALIDATION =re.findall(r'<input type="hidden" name="EVENTVALIDATION" id="EVENTVALIDATION" value="(.?)" />', html,re.I)
return VIEWSTATE[0],EVENTVALIDATION[0]
VIEWSTATE, EVENTVALIDATION=get_hiddenvalue(url)
data = {
'EVENTVALIDATION':EVENTVALIDATION,
'VIEWSTATE':VIEWSTATE,
'__VIEWSTATEGENERATOR':'EC4ACD6F',
'btnSearch.x':'23',
'btnSearch.y':'12',
'ddShareholdingDay':NIAN,
'ddShareholdingMonth':YUE,
'ddShareholdingYear':RI,
'today':'20180509'
}
html_post = requests.post(url, data=data)
print(html_post.text)