Python crawling with watercress popular title of the play and score information

Crawling target

Looking for information with chrome Devtools

• URL and request method

• Params和User-Agent

When you click "Load more" when, page_start parameters will increase from 20, thus defining the time to use a loop Params

Json string with Postman preview

Here we just need to get rate information and name

Crawl in Python

import requests
import json
url = 'https://movie.douban.com/j/search_subjects'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'
}
for i in range(0,100,20):
    params = {
        'type': 'tv',
        'tag': '热门',
        'sort': 'recommend',
        'page_limit': 20,
        'page_start': i
    }
    res = requests.get(
        url = url,
        params = params,
        headers = headers
    )
    html = res.text
    data = json.loads(html)
    for data_temp in data['subjects']:
        print(data_temp['title'],data_temp['rate'])