Inventory of a Python web crawler grabbing stock codes (Part 1)

Click on " Python Crawler and Data Mining " above to pay attention

Reply to " Books " to get a total of 10 e-books on Python from entry to advanced

now

Day

chicken

Soup

Come and go Eguchi Mamoru Airship, Reef ship Tsukimyo River water cold.

Hello everyone, I am Pippi.

I. Introduction

A few days ago, I asked a Pythonquestion about web crawlers in the Python Silver Group [Hou De Zai Wu], and I will share it with you here.

2. Implementation process

In fact, this problem can be solved with a for loop. It seems that the fan's code does not include the request headers, resulting in the inability to obtain data. Later, [Teacher Yuliang] and [Little Prince] gave specific ideas, and the code is shown in the figure below:

Later [Little Prince] also gave a specific code, as follows:

import requests
import time

headers = {
    "authority": "stock.xueqiu.com",
    "accept": "*/*",
    "accept-language": "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6",
    "origin": "https://xueqiu.com",
    "referer": "https://xueqiu.com/S/SH600600",
    "sec-ch-ua": "\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Microsoft Edge\";v=\"114\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Windows\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-site",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51"
}
cookies = {
    "xq_a_token": "57b2a0b86ca3e943ee1ffc69509952639be342b9",
    "xqat": "57b2a0b86ca3e943ee1ffc69509952639be342b9",
    "xq_r_token": "59c1392434fb1959820e4323bb26fa31dd012ea4",
    "xq_id_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJ1aWQiOi0xLCJpc3MiOiJ1YyIsImV4cCI6MTY5MDMzMTY5OCwiY3RtIjoxNjg3ODcxOTQxNTM1LCJjaWQiOiJkOWQwbjRBWnVwIn0.KI3paq6_r2IZuM5AemhFqy5l1vVFsxf7ICELsem_rwAd0yYo_8bfOs1aP_5BBO3_rJuP9r6CJslCg1S_icefsPgqgtWZwVzfcIwvoLlZaag4a9IjqBxc2G6Ug50F9_UMLifzbNDrN4u8kwjm0sXHFUYnT_I89pwr0CeEgC4-jo9ExazlXJFZk_tA40C6L3npCVkKk8cOfl5JnvVUADXdef8G54jAsL_N5Sjx30YKxU1_2aUKRJZhRlN6bXqcXIP466odbBSBrMp52FFdB1knI2IN4dQJ5Hg4PRQyCsyFtgp-h_s_Rru4nwFqN6aiXPgLue1pGFGg25qBvTVr2m_9cQ",
    "u": "561687871945884",
    "device_id": "27b6ec56b772ea40c8582168f00a7604",
    "Hm_lvt_1db88642e346389874251b5a1eded6e3": "1687871949",
    "s": "ci1eygzbit",
    "is_overseas": "0",
    "Hm_lpvt_1db88642e346389874251b5a1eded6e3": "1687872001"
}
url = "https://stock.xueqiu.com/v5/stock/quote.json"
symbols = ['SH600600', 'SH600519', 'SH301183']
for symbol in symbols:
    params = {
        "symbol": f"{symbol}",
        "extend": "detail"
    }
    response = requests.get(url, headers=headers, cookies=cookies, params=params)
    time.sleep(3)

    print(response.text)
    print(response)

After the code runs, you can get specific results, as shown in the following figure:

【Teacher Yuliang】Added a code later, as follows:

for i in ['SH600519','SZ300600','SZ301183']:
    url = f"https://stock.xueqiu.com/v5/stock/quote.json?symbol={i}&extend=detail"
    response = requests.get(url=url, headers=headers,cookies=cookies)
    json_data = response.json()
    print(json_data)

Solved the problem of fans smoothly. There are many ways, all roads lead to Rome, as long as it can solve the problem.

3. Summary

Hello everyone, I am Pippi. This article mainly takes stock of a Pythonweb crawler problem. Aiming at this problem, the article gives specific analysis and code implementation to help fans solve the problem smoothly.

Finally, I would like to thank the fans [Hou De Zai Wu] for asking questions, thanks to [Teacher Yu Liang], [Wei Ge], [kim], [巭嫑勥烎] for their ideas and code analysis, and thanks to [冫马讠成], [Ineverleft ] and others participated in the learning exchange.

[Supplementary questions] Warm reminder, when you ask questions in the group. You can pay attention to the following points: if it involves large file data, you can desensitize the data, send some demo data (meaning small files), and then paste some code (the kind that can be copied), and remember to send the screenshot of the error report (complete cut ). If there are not many codes, just send the code text directly. If the code exceeds 50 lines, just send a .py file.

If you have any problems during the learning process, please feel free to contact me to solve them (my WeChat: pdcfighting1). At the request of fans, I have created some high-quality Python paid learning exchange groups and paid order receiving groups. Welcome everyone to join me Python learning exchange group and order receiving group!

Friends, hurry up and practice it! If you encounter any problems during the learning process, please add me as a friend, and I will pull you into the Python learning exchange group to discuss learning together.

------------------- End -------------------

Recommendations for past wonderful articles:

Welcome everyone to like, leave a message, forward, repost, thank you for your company and support

If you want to join the Python learning group, please reply in the background [ join the group ]

Thousands of rivers and thousands of mountains are always in love, can you click [ Looking ]

/Today's Message Topic/

Just say a few words~~