Series Article Directory
Programming Xiaobai's self-study notes eight (multithreading in python)
Programming Xiaobai's self-study notes seven (inheritance of classes in python)
Programming Xiaobai's self-study notes six (static methods and dynamic methods of classes in python)
Programming Xiaobai's self-study notes five (Python class method)
Table of contents
1. How to find the headers of a web page
We have learned the get function of the third-party module requests module before, and today we will continue to study in depth
1. How to find the headers of a web page
Through the last study, we found that we need to pass parameter headers to fool the server, so that the server believes that a normal browser is accessing it. Not every computer has the same headers. How do we know? We can use the browser to access normally, and then view it in inspection mode. The specific operations are as follows:
1. Turn on the inspection mode of the web page. After opening the webpage with Google Chrome, right-click the mouse and click Inspect.
2. Select Network. After opening the inspection mode, we click Network in the inspection mode window on the right.
3. Select name. After opening the Network, we found that there is no content. At this time, click the F5 key on the keyboard. After the page is refreshed, the name tab appears.
4. Find User-Agent. Click the html file in the name tab, then find User-Agent on the right, and then copy the contents to python. The code is as follows:
import re
url = 'https://www.kugou.com/yy/rank/home/1-8888.html'
h = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'}
try:
req = requests.get(url,headers=h)
songs = re.findall(r'<li.*?title="(.*?)"',req.text)
for song in songs:
print(song)
except:
print('查询失败')The resulting output is the same as the previous article.
2. How to find top200 data
The above code can only get the data of one webpage, and only the top20 data. If we want to get the top200 data, do we need to write ten codes?
No~, after researching the link, we found that changing the following 1-8888 to 2-8888 will turn the page to the second page, and so on, the link of top180-200 is 10-8888, and the link is essentially a string The form is passed to the url, so that we can set a variable to automatically change the address of the url through the form of a for loop. Let's take a look at the code:
import requests
import re
for i in range(1,11):
url = f'https://www.kugou.com/yy/rank/home/{i}-8888.html'
h = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'}
try:
req = requests.get(url,headers=h)
songs = re.findall(r'<li.*?title="(.*?)"',req.text)
for song in songs:
print(song)
except:
print('查询失败')The output result is:
Su Xingjie- Listen to sad love songs
fingertip laugh - don't ask ciaga
Guo Ding- Poignantly
An Aries - Can't wait for you
Ren Xia- Sad Love
Jane Zhang, Wang Heye - It's You (Live)
Mae Stephens - If We Ever Broke Up (Explicit)
Kui Kui - what are you doing baby
Zhang Zihao- can you
Jay Chou- Promised happiness?
Jay Chou - Sunny Day
Wang Sulong, Jike Junyi-Letting Go (Live)
Seunghwan- I will wait
Tanya Chua-Letting Go
Ren Xia - Insomnia Love Song (Live Chorus Version)
Su Xingjie- Thinking of you in the evening wind
Jay Chou- I cry and my emotions are fragmented
Cloud Dog Egg- If there is love in the sky
Cheng Xiang - possible
A-Lin - If there is love in heaven
RE-D, Erhaya, masta - sure
GEM Deng Ziqi- I like you
Jay Chou- Dandelion Promise
Hu Yanbin- Take everything you want
Zhao Lei- I remember
BEYOND - The sea and the sky
Jay Chou - Lanting Preface
Li Yuchun - Next, intersection, see you
Guo Jing- Heart Wall
Huang Jingmei, Zhang Junbo - I borrow ink from the world
Wiz_H Zhang Zihao- An ordinary day
Qu Xiaobing- who
GEM Deng Ziqi- No matter how far away we must be together
Ailee - HEAVEN (Original Version)
Tanya - Darwin
Wang Lanyin- Mischief
Gem Gem - Elevator God of War (Live)
Sheep - Emotion
Jay Chou- Ming Ming
Sheng Zhe - by your side
Taylor Swift - Love Story
Kelly Clarkson - Catch My Breath
Zhang Bichen - Drive to Midnight in the Morning
cici_ - spell the memories for you
Jay Chou - Daoxiang
Jay Chou - Sea of Flowers
JJ Lin- Light Wrapped in Heart
Hillsong Young & Free - Wake (Studio)
Jay Chou- Qilixiang
You can also use coupons to buy peppers - it's windy
EXO - Baby, Don't Cry (Mermaid's Tears)
JJ Lin - Darwin
Jay Chou- I Believe So So
Tanya Chua- Red high heels
Lulleaux、Kid Princess - Empty Love
Fingertip Laughing - Do Not Ask Parting
Tanya - Darwin II
Yongbin Ryan.B - Looks so good
Jay Chou - Clock in the opposite direction
Faye Wong- wish
Sue five people - lover misses
Jay Chou - Say Goodbye
Yu Chaoying - Chun Ni
Jay Chou - longest movie
Hou Zerun- There is a kind of love that is you
JISOO - FLOWER
Hebe Tien- Little Lucky
Jay Chou - Love in BC
Ear Convenience Store - I Don't Believe You Are Not Ashamed
Su Xingjie- Put together the memories for you
Ari Abdul - BABYDOLL (Speed) (Explicit)
Lil Xiaoxiao- Camellia Can't Read White Rose
Zeng Haoran- My Beauty
Wong Ching Mei - Chuang Chow Dreams a Butterfly
Jay Chou - All the Way North
Hammer Na Lisa - I'm so stupid
Joker Xue- Worship
Eason Chan- Love Transfer
AGA - Orphan
RAiNBOW Project, Thunderstorm Heart - Commemoration
Xiaomi- after I leave
DP Dragon Pig, Wang Yunhong, Trap Cousin- Cuihua
JJ Lin- No fashion, no money
Zhou Linfeng- forgot
Jay Chou - Blue and White Porcelain
Shengyu - special love for special you
Mayday- Later Us
FIFTY FIFTY - Cupid
Madnap、Pauline Herr - Slow Down
Jay Chou - back off
Xiyin Society, Floating Dream - Detecting Window
Sue Five - Take Me to Nightlife
Yin Ximian-Three Worships to the Red Dust
Cheng Jin- Love is a Fearless Adventure
Hou Zerun- The Age of Nothing
S.H.E - Super Star
Jay Chou - Stranded
Zhang Yunjing- Preference
Jay Chou - Nocturne
Ren Xia- The Punishment of Love
Kekeke- The girl is far away
Su Han- last page
JJ Lin- Practice Love
Huang Qishan, Xilinna Yigao - Mother or Daughter
BY2 - I know
Zhang Bichen, Wang Heye - Every Word (Live)
Zhao Lei- Our Time
Hedley - Lose Control (Explicit)
JJ Lin - Jiangnan
Nachwolf, KKECHO - Bitter Coffee The Only One
Joey Yung- Just let the heavy rain fall
Han Shuai (HS) - Meet the Stars
Henry Young、Ashley Alisha - One More Last Time
Zhang Yelei- Still separate
Ge Dongqi- Hanging Drowning
Street Office GDC, Ouyang Yaoying - Chunjiao and Zhiming
melo-D - Just Say Hello
BEYOND - Glory Days
Sealed Moon - Frustrated
Ke Ziyan- Listen to sad love songs
Hua Ling, Meow Soy Sauce, Yan Ning, Kinsen - Let the wind tell you
Capper, 罗言RollFlash - Snow Distance
Glichery - Sea Of Problems (Explicit)
A Liyue - Evening Breeze Blows in My Heart
JJ Lin - The Mermaid
Shao Shuai- warm a cup of tea
GEM Deng Ziqi- Countdown
Zhang Shaohan, Wang Heye - Chapters
Cui Zige - Divination
Hou Zerun- Punishment and Reward
JJ Lin- Unfortunately, if not
GEM Deng Ziqi- Peach Blossom Nuo
Elva Hsiao- The Lost Heartbeat
Ah Sang- has been very quiet
Mirei Toyama - Wish ~To you in those days~
JJ Lin- Always Online
Cheer Chen- Still lonely
Jay Chou - Peninsula Iron Box
Sulong Wang, Jessica - Oops (Live)
JJ Lin - back to back hug
Zyboy Zhongyu- Mom's Words
Jay Chou - Tracks
Fu Lu Shou FloruitShow - How can I keep you (Live)
Jay Chou - Maple
Mengran - it's you
Xue Zhiqian- Extraterrestrial
Tanya Chua- I don't understand more and more
Yu Dongran- I heard you
Wang Zijian - Tracking
JJ Lin- Sunshine in the Crack
Jay Chou, Zhang Huimei - Shouldn't
Pixxie、Zom Marie - No, no, no (Whatever)
Xiao Jingteng, Zhang Qi - Wujiapo 2021 (Live)
sue five people - only
Cheng Xiang- Fireworks on Earth
Li Ronghao- Li Bai
Qi Wei- If love is forgotten
GEM Deng Ziqi- Goodbye
EXO - Baby, Don't Cry (mermaid tears)
Hou Zerun- Excuse
Professor Green - In The Shadow Of The Sun
It's not too late in life - it's hard
Guo Ding- A Tale of Mercury
Jay Chou- Fireworks are easy to be cold
Haoran HR - Young you
Cai Xukun- Hug me
Meg Myers - Running Up That Hill
Max Elto - Shadow Of The Sun
Lan Xinyu- Lonely Fireworks
JJ Lin- Cao Cao
Every Family - Destiny
Jay Chou - can you hear me
Jay Chou- Confession Balloon
A white sheep - give me
Zhou Chuanxiong - Twilight
Dion Timmer、The Arcturians - The Best Of Me
JJ Lin- Romantic Blood
Xianzi, Brother Xiaole (Wang Weile) - Unwilling (Live)
JJ Lin- Write the story as us
Blue Bird and Flying Fish- This life will never change
Mayday - step by step
LBI Libby - Small Town Summer
Jike Junyi, Yu Wenwen - Take All You Want (Live)
Wei Zai, Grid Xi - I know you don't love me
Jay Chou- The Untold Secret
JVKE - golden hour
Leehom Wang- Our Song
Half Ton Brothers, Zhang Qian- Night in Ulaanbaatar
Zhou Linfeng, L (Peach Seed) - Just for a few taels of silver
Wanting Qu - in my singing
Ren Xia- why
Fu Hao- Is he really nice to you?
Ocean Xiaoxia - To the Clouds
Kekeke- After the rain, the sky is not sunny
Rainie Yang- Rain Love
Daniel Powter - Free Loop
Jay Chou- Waiting for you to finish class (with Yang Ruidai)
Xinchen Wang, Xingjie Su- Clear
Danzheng Mother and Son- Night in Ulaanbaatar
Wang Yitai, Liu Zhijia- Dangerous Party
KOKIA - Thank you... (Thank you...)
Zhang Jie- he doesn't understand
Jay Chou - Secret Code
Wang Sulong, Joey Yung - Let the heavy rain fall (Live)
Tungevaag、Raaban、Richard Smitt - All For Love
Blue Heart Feather - Gulf of Alaska
JJ Lin- I still miss her
Wiz Khalifa、Charlie Puth - See You Again
Öwnboss、SEVEK - Move Your Body (remix：Razihel)
Reynard Silva - The Way I Still Love You
Chopsticks Brothers - Old Boys
ycccc - The stars are not as good as you
A White Sheep - Blow the Quiet Wind
It's Nanfang Kai- Xunguang
Rainie Yang- Take me away
Lu Jie awr - Evening Wind Meets You
Lu Jie awr - 32 degree evening wind
Xu Wei- Once you
Hou Zerun- Be a cruel person in the next life
Hu Ge- Forget Time
200 pieces of data are not too much, and if there is no special rule for the address of the webpage, I think it can be made into a list, or put in a file, and the for loop can also be used to achieve it.
Summarize
Crawler headers refer to some information attached to the request when sending it, which is used to simulate browser behavior. Common headers include User-Agent, Referer, Cookies, etc. Among them, User-Agent is used to identify the source of the request, Referer is used to identify the source page of the request, and Cookies is used to store login status and other information.