Python web crawler information extraction code examples mooc

This article describes the python web crawler and information extraction mooc, the text introduced by the sample code is very detailed, with some reference value of learning for all of us to learn or work, a friend in need can refer to

Example one - crawling pages

import requests
url="https//itemjd.com/2646846.html"
try:
 r=requests.get(url)
 r.raise_for_status()
 r.encoding=r.apparent_encoding
 print(r.text[:1000])
except:
 print("爬取失败")

Normal crawling pages

Example Two - crawling pages

import requests
url="https://www.amazon.cn/gp/product/B01M8L5Z3Y"
try:
 kv={'user-agent':'Mozilla/5.0'}
 r=requests.get(url,headers=kv)
 r.raise_for_status()
 r.encoding=r.apparent_encoding
 print(r.text[1000:2000])
except:
 print("爬取失败")

There are restrictions on access to the user name, simulate browser requests to the site

Three examples - crawling search engine

#百度的关键词接口:http://www.baidu.com/s?wd=keyword
#360的关键词接口:http://www.so.com/s?q=keyword
import requests
keyword="python"
try:
 kv={'wd':keyword}
 r=requests.get("http://www.baidu.com/s",params=kv)
 print(r.request.url)
 r.raise_for_status()
 print(len(r.text))
except:
 print("爬取失败")
--------------------------------------------------
import requests
keyword="python"
try:
 kv={'q':keyword}
 r=requests.get("http://www.so.com/s",params=kv)
 print(r.request.url)
 r.raise_for_status()
 print(len(r.text))
except:
 print("爬取失败")

Four examples: - crawling picture

import requests
import os
url="http://image.nationalgeographic.com.cn/2017/0211/20170211061910157.jpg"
root="F://pics//"
path=root+url.split('/')[-1]
try:
 if not os.path.exists(root):
  os.mkdir(root)
 if not os.path.exists(path):
  r=requests.get(url)
  with open(path,'wb') as f:
   f.write(r.content)
   f.close()
   print("文件保存成功")
 else:
  print("文件已经存在")
except:
 print("爬取失败")

Crawling and storing the image

Examples of five -IP address belonging to the query:

http://m.ip138.com/ip.asp?ip=ipaddress

url="http://www.ip138.com/iplookup.asp?ip="
try:
 r=requests.get(url+'202.204.80.112'+'&action=2')
 r.raise_for_status()
 r.encoding=r.apparent_encoding
 print(r.text[-500:])
except:
 print("爬取失败")

Content on more than how many, and finally to recommend a good reputation in the number of public institutions [programmers], there are a lot of old-timers learning skills, learning experience, interview skills, workplace experience and other share, the more we carefully prepared the zero-based introductory information on actual project data every day to explain the timing of Python programmers technology, and share some learning methods need to pay attention to small details

Here Insert Picture Description

Published 20 original articles · won praise 0 · Views 3620

Guess you like

Origin blog.csdn.net/chengxun02/article/details/104998851