Scrapy框架学习练手之爬取腾讯招聘技术类岗位

页面地址：

https://careers.tencent.com/search.html?pcid=40001

实现目标：

将爬取到的岗位名称、工作职责、工作要求、发布日期以字典格式输出。

Scrapy目录框架：

思路：

        

浏览器抓包分析网页请求地址规律（爬虫最重要），找到页面地址规律后，根据请求返回的数据进行提取即可。

图一

由图一页面可知，招聘岗位共有187页，需循环遍历所有页面；

浏览器抓包实际请求页面地址为:

https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1575882949947&countryId=&cityId=&bgIds=&productId=&categoryId=40001001,40001002,40001003,40001004,40001005,40001006&parentCategoryId=&attrId=&keyword=&pageIndex=1&pageSize=10&language=zh-cn&area=cn

规律：pageIndex=1 为页数，其他暂保持不变

（爬完之后发现这里也有个时间戳，哈哈，冒似没啥影响，时间固定一样可以获取到数据）

图二

图三

页面请求返回的PostId为岗位详情页后面的具体地址:

招聘岗位详情页示例：

https://careers.tencent.com/jobdesc.html?postId=1203886892391600128

图四

从浏览器抓包的请求头来看，URL地址规律如下：

https://careers.tencent.com/tencentcareer/api/post/ByPostId?timestamp=1575882580480&postId=1203886892391600128&language=zh-cn “https://careers.tencent.com/tencentcareer/api/post/ByPostId?timestamp= “此为固定部分，后面为当前时间戳，

 

Python代码如下：

时间戳部分学习参考：https://blog.csdn.net/qq_31603575/article/details/83343791

 

# -*- coding: utf-8 -*- import scrapy import json,datetime,time #生成 13位当前时间戳 def get_time_stamp13(): # 生成13时间戳 eg:1540281250399895 datetime_now = datetime.datetime.now() # 10位，时间点相当于从UNIX TIME的纪元时间开始的当年时间编号 date_stamp = str(int(time.mktime(datetime_now.timetuple()))) # 3位，微秒 data_microsecond = str("%06d" % datetime_now.microsecond)[0:3] date_stamp = date_stamp + data_microsecond return int(date_stamp) class TestSpider(scrapy.Spider): name = 'test' allowed_domains = ['tencent.com'] start_urls = ['https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1575855782891&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=40001&attrId=&keyword=&pageIndex=1&pageSize=10&language=zh-cn&area=cn'] def pasrse1(self,response): html1 = response.text #响应数据转换为字典格式 html1_json = json.loads(html1) # print(html1_json['Data']) # exit() # for j in html1_json['Data']: job_dic = {} job_dic['岗位名称'] = html1_json['Data']['RecruitPostName'] job_dic['工作职责'] = html1_json['Data']['Responsibility'] job_dic['发布日期'] = html1_json['Data']['LastUpdateTime'] job_dic['工作要求'] = html1_json['Data']['Requirement'] print(job_dic) def parse(self, response): html = response.text html_json = json.loads(html) id_url_list = html_json['Data']['Posts'] #[0]['PostId'] for lj in id_url_list: time_format = str(get_time_stamp13()) #url地址拼接 desc_url = 'https://careers.tencent.com/tencentcareer/api/post/ByPostId?timestamp=' + time_format + '&postId=' + lj['PostId'] + '&language=zh-cn' # print(desc_url) # print(lj['PostId']) # for k in url_list['PostURL']: # job_url = [] # job_url.append(k) # yield scrapy.Request(url=job_url, callback=self.pasrse1, dont_filter=True) #提交请求，迭代处理 yield scrapy.Request(url=desc_url, callback=self.pasrse1, dont_filter=True) for i in range(2,188): #页面地址循环遍历 url = 'https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1575855782891&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=40001&attrId=&keyword=&pageIndex=' + str(i) + '&pageSize=10&language=zh-cn&area=cn' yield scrapy.Request(url=url,callback=self.parse,dont_filter=True)

执行效果如下：