This article describes the python web crawler and information extraction mooc, the text introduced by the sample code is very detailed, with some reference value of learning for all of us to learn or work, a friend in need can refer to
Example one - crawling pages
import requests
url="https//itemjd.com/2646846.html"
try:
r=requests.get(url)
r.raise_for_status()
r.encoding=r.apparent_encoding
print(r.text[:1000])
except:
print("爬取失败")
Normal crawling pages
Example Two - crawling pages
import requests
url="https://www.amazon.cn/gp/product/B01M8L5Z3Y"
try:
kv={'user-agent':'Mozilla/5.0'}
r=requests.get(url,headers=kv)
r.raise_for_status()
r.encoding=r.apparent_encoding
print(r.text[1000:2000])
except:
print("爬取失败")
There are restrictions on access to the user name, simulate browser requests to the site
Three examples - crawling search engine
#百度的关键词接口:http://www.baidu.com/s?wd=keyword
#360的关键词接口:http://www.so.com/s?q=keyword
import requests
keyword="python"
try:
kv={'wd':keyword}
r=requests.get("http://www.baidu.com/s",params=kv)
print(r.request.url)
r.raise_for_status()
print(len(r.text))
except:
print("爬取失败")
--------------------------------------------------
import requests
keyword="python"
try:
kv={'q':keyword}
r=requests.get("http://www.so.com/s",params=kv)
print(r.request.url)
r.raise_for_status()
print(len(r.text))
except:
print("爬取失败")
Four examples: - crawling picture
import requests
import os
url="http://image.nationalgeographic.com.cn/2017/0211/20170211061910157.jpg"
root="F://pics//"
path=root+url.split('/')[-1]
try:
if not os.path.exists(root):
os.mkdir(root)
if not os.path.exists(path):
r=requests.get(url)
with open(path,'wb') as f:
f.write(r.content)
f.close()
print("文件保存成功")
else:
print("文件已经存在")
except:
print("爬取失败")
Crawling and storing the image
Examples of five -IP address belonging to the query:
http://m.ip138.com/ip.asp?ip=ipaddress
url="http://www.ip138.com/iplookup.asp?ip="
try:
r=requests.get(url+'202.204.80.112'+'&action=2')
r.raise_for_status()
r.encoding=r.apparent_encoding
print(r.text[-500:])
except:
print("爬取失败")
Content on more than how many, and finally to recommend a good reputation in the number of public institutions [programmers], there are a lot of old-timers learning skills, learning experience, interview skills, workplace experience and other share, the more we carefully prepared the zero-based introductory information on actual project data every day to explain the timing of Python programmers technology, and share some learning methods need to pay attention to small details