As soon as the weather is cold, I want to eat meat and crawl the data of barbecue restaurants in a city (with complete Python crawler code)

foreword

As soon as the weather is cold, I want to eat meat, and stick to the autumn fat in order to survive the winter. For carnivores, eating meat can be overwhelming! Especially for barbecued meat, watching the pieces of meat slowly become cooked and listening to the sound of "zizi" on the grill, this kind of anticipation cannot be brought by any other food. If dessert is "happy at first sight", then meat is "never tire of it."

In order to benefit the "barbecue control", today I will use Python to crawl the data of a city's barbecue restaurant and select the most suitable one

Ready to work

environment

• python 3.6
• pycharm
• requests >>> send requests pip install requests
• csv >>> save data

Understand the most basic ideas of reptiles

1. Data source analysis

1. Determine what content we crawl?
   Crawl store data
2. Find where these things come from
   Capture packets through developer tools and analyze data sources

2. Code implementation process

1. send request, send request for found packets
2. Get data, based on the response data returned by the server to you
3. Parse the data, extract the content data we want
4. save data, save to csv file
5. Multi-page crawling, changes according to url address parameters

code implementation process

1. Send a request

url = 'https://apimobile.某tuan.com/group/v4/poi/pcsearch/70'
data = {
    
    
    'uuid': '6e481fe03995425389b9.1630752137.1.0.0',
    'userid': '266252179',
    'limit': '32',
    'offset': 32,
    'cateId': '-1',
    'q': '烤肉',
    'token': '4MJy5kaiY_0MoirG34NJTcVUbz0AAAAAkQ4AAF4NOv8TNNdNqymsxWRtJVUW4NjQFW35_twZkd49gZqFzL1IOHxnL0s4hB03zfr3Pg',
}
# 请求头 都是可以从开发者工具里面直接复制粘贴
# ser-Agent: 浏览器的基本信息
headers = {
    
    
    'Referer': 'https://chs.某tuan.com/',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
}
# 发送请求
response = requests.get(url=url, params=data, headers=headers)

200 indicates that the request was successful and the status code is 403. You do not have access rights

2. Get data

print(response.json())

3. Parse the data

result = response.json()['data']['searchResult']
# [] 列表 把里面每个元素都提取出来 for循环遍历
for index in result:
    # pprint.pprint(index)
    # f'{}' 字符串格式化
    index_url = f'https://www.某tuan.com/meishi/{
      
      index["id"]}/'
    # ctrl + D
    dit = {
    
    
        '店铺名称': index['title'],
        '店铺评分': index['avgscore'],
        '评论数量': index['comments'],
        '人均消费': index['avgprice'],
        '所在商圈': index['areaname'],
        '店铺类型': index['backCateName'],
        '详情页': index_url,
    }
    csv_writer.writerow(dit)
    print(dit)

4. Save data

f = open('烤肉数据.csv', mode='a', encoding='utf-8', newline='')
csv_writer = csv.DictWriter(f, fieldnames=[
    '店铺名称',
    '店铺评分',
    '评论数量',
    '人均消费',
    '所在商圈',
    '店铺类型',
    '详情页',
])
csv_writer.writeheader() # 写入表头

5. Turn the page

for page in range(0, 1025, 32):
    url = 'https://apimobile.某tuan.com/group/v4/poi/pcsearch/70'
    data = {
    
    
        'uuid': '6e481fe03995425389b9.1630752137.1.0.0',
        'userid': '266252179',
        'limit': '32',
        'offset': page,
        'cateId': '-1',
        'q': '烤肉',
        'token': '4MJy5kaiY_0MoirG34NJTcVUbz0AAAAAkQ4AAF4NOv8TNNdNqymsxWRtJVUW4NjQFW35_twZkd49gZqFzL1IOHxnL0s4hB03zfr3Pg',
    }

Run the code to get the data

full code

f = open('烤肉数据1.csv', mode='a', encoding='utf-8', newline='')
csv_writer = csv.DictWriter(f, fieldnames=[
    '店铺名称',
    '店铺评分',
    '评论数量',
    '人均消费',
    '所在商圈',
    '店铺类型',
    '详情页',
])
csv_writer.writeheader() # 写入表头

for page in range(0, 1025, 32):
    url = 'https://apimobile.某tuan.com/group/v4/poi/pcsearch/70'
    data = {
    
    
        'uuid': '6e481fe03995425389b9.1630752137.1.0.0',
        'userid': '266252179',
        'limit': '32',
        'offset': page,
        'cateId': '-1',
        'q': '烤肉',
        'token': '4MJy5kaiY_0MoirG34NJTcVUbz0AAAAAkQ4AAF4NOv8TNNdNqymsxWRtJVUW4NjQFW35_twZkd49gZqFzL1IOHxnL0s4hB03zfr3Pg',
    }
    headers = {
    
    
        'Referer': 'https://chs.某tuan.com/',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
    }
    # 发送请求
    response = requests.get(url=url, params=data, headers=headers)
    # 200 表示请求成功 状态码 403 你没有访问权限

    result = response.json()['data']['searchResult']
    # [] 列表 把里面每个元素都提取出来 for循环遍历
    for index in result:
        # pprint.pprint(index)
        # f'{}' 字符串格式化
        index_url = f'https://www.meituan.com/meishi/{
      
      index["id"]}/'
        # ctrl + D
        dit = {
    
    
            '店铺名称': index['title'],
            '店铺评分': index['avgscore'],
            '评论数量': index['comments'],
            '人均消费': index['avgprice'],
            '所在商圈': index['areaname'],
            '店铺类型': index['backCateName'],
            '详情页': index_url,
        }
        csv_writer.writerow(dit)
        print(dit)