今天爬取的网站是http://jifen.sina.com.cn/category
今天遇到的两个难点是:
一、通过xpath爬取img的src的属性结果是另一个图片的连接
二、xpath爬取下来的内容存进mysql遇到的问题
一边上代码 一边说问题
import requests
import mysql.connector
from lxml import etree
import random
from datetime import datetime,timedelta
# 连接mysql
db = mysql.connector.connect(
user="root",
password="root",
host="127.0.0.1",
port=3306,
database="myself"
)
cursor = db.cursor(dictionary=True)
#http://jifen.sina.com.cn/category/index/index/0/_/1/60?wd=
url = "http://jifen.sina.com.cn/category"
proxy = {
"HTTP": "113.3.152.88:8118",
"HTTPS": "219.234.5.128:3128",
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/70.0.3538.110 Safari/537.36",
}
response = requests.get(url=url, headers=headers, proxies=proxy).content.decode("utf-8")
html = etree.HTML(response)
li_tr = html.xpath('//ul[@class="tab_con"]/div[@class="clear"]/li')
#这个函数判断获取的内容是否为空 并把有,的字符去掉
def isnull(test_list):
if len(test_list)>0:
return test_list[0][:-2].replace(",","")
else:
return random.randint(100,300)
for i in li_tr:
#这里注意了通过开发者工具看到是src但是实际的属性是data-src
#如果遇到看到是这样的 获取却另外的数据可以通过
#开发者的Network 的response 里面看
goods_img=i.xpath('.//img/@data-src')
if len(goods_img) > 0:
goods_img=goods_img[0]
else:
continue
goods_name = i.xpath('./div[@class="botm_text"]/a/text()')[0]
goods_jf = isnull(i.xpath('.//b/text()'))
sql="insert into index_goods (goods_name,goods_image,goods_jifen,) values (%s,%s,%s)"
#如果通过xpath获取的内容存进mysql时要进行encode('utf-8'),不然会报类型错误
val=(goods_name.encode('utf-8'),goods_img.encode('utf-8'),str(numbers),int(goods_jf),addtime,outtime)
cursor.execute(sql,val)
db.commit()
print(goods_img,goods_name,goods_jf)
这次爬取遇到就两个问题要注意一下!