Preface
The text and pictures in this article are from the Internet and are for learning and communication purposes only, and do not have any commercial use. If you have any questions, please contact us for processing.
Python crawler, data analysis, website development and other case tutorial videos are free to watch online
https://space.bilibili.com/523606542
Preamble content
Python crawler beginners introductory teaching (1): crawling Douban movie ranking information
Python crawler novice introductory teaching (2): crawling novels
Python crawler beginners introductory teaching (3): crawling Lianjia second-hand housing data
Python crawler novice introductory teaching (4): crawling 51job.com recruitment information
Python crawler beginners' introductory teaching (5): Crawling the video barrage of station B
Python crawler novice introductory teaching (6): making word cloud diagrams
Python crawler beginners introductory teaching (7): crawling Tencent video barrage
Python crawler novice introductory teaching (8): crawl forum articles and save them as PDF
Python crawler beginners introductory teaching (9): multi-threaded crawler case explanation
Python crawler novice introductory teaching (ten): crawling the other shore 4K ultra-clear wallpaper
Python crawler beginners introductory teaching (11): recent king glory skin crawling
Basic development environment
- Python 3.6
- Pycharm
Use of related modules
import os # 内置模块 用于创建文件
import requests # 第三方模块 需要 pip install requests 安装 用于请求网页数据Install Python and add it to the environment variables, pip installs the required related modules.
One, clear needs
Crawl the skin background images of all the heroes of the League of Legends. Contains dazzling colors, saved separately according to heroes.
2. Web page data analysis
How to find the real address of the data?
As shown in the figure, the url address of the skin image is https://game.gtimg.cn/images/lol/act/img/skin/big1001.jpg
The URL address of each picture is one-to-one corresponding to the change of big1001 .
So you can copy big1001 and search in the developer tools to find the source of the picture address.
As shown in the figure, the name of the skin, the address of the picture, and the name of the hero in the link https://game.gtimg.cn/images/lol/act/img/js/hero/1.js are all available.
Now that you have found the source of the picture, you need to find the source of the above data interface.
Annie data interface: https://game.gtimg.cn/images/lol/act/img/js/hero/1.js
Annie data details page: https://lol.qq.com/data/info-defail. shtml?id=1
Olaf data interface: https://game.gtimg.cn/images/lol/act/img/js/hero/2.js
Olaf data details page: https://lol.qq.com/data/info- defail.shtml?id=2
Through the above link comparison, it can be clearly seen that the parameter changes of the interface data are based on the hero ID.
Generally, if you want to get the ID value of each page, you need to go to the list page to find it.
As shown in the figure, each hero has an ID.
Three, code implementation
1. Get all hero IDs
url = 'https://game.gtimg.cn/images/lol/act/img/js/heroList/hero_list.js'
json_data = get_response(url).json()['hero']
for i in json_data:
hero_id = i['heroId']2. Every hero picture
def get_hero_url(hero_id):
page_url = f'https://game.gtimg.cn/images/lol/act/img/js/hero/{hero_id}.js'
hero_data = get_response(page_url).json()
skins = hero_data['skins']
for index in skins:
# 皮肤url
image_url = index['mainImg']
# 皮肤名字
hero_name = index['name']
# 文件夹名字
hero_title = index['heroTitle']
if image_url:
save(hero_title, hero_name, image_url)
else:
image_2_url = index['chromaImg']
save(hero_title, hero_name, image_2_url)A judgment is needed here, because there are some hero skins that carry dazzling colors.
3. Save data (data persistence)
def save(hero_title, hero_name, image_url):
path = f'{hero_title}\\'
if not os.path.exists(path):
os.makedirs(path)
image_content = get_response(image_url).content
with open(path + hero_name + '.jpg', mode='wb') as f:
f.write(image_content)Fourth, achieve the effect
Suddenly found that Annie didn't have any dazzling colors (except for the guardian of the blessed cow who hasn't appeared yet), but there are really many skins.