Hello everyone, today we are going to obtain the data of a certain life platform website for visual analysis.
To collect 58 data, you can use Python's requests library and beautifulsoup library, and for data visualization analysis, you can use matplotlib library and seaborn library. Here is a simple example:
1. First import the modules that need to be used
import requests
from bs4 import BeautifulSoup
import matplotlib.pyplot as plt
import seaborn as sns
2. Set the request header to simulate a browser request.
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
3. Send a request to get the page
url = 'https://***/pinpaigongyu/pn/{page}/?minprice=2000_4000'
house_data = []
for page in range(1, 3):
res = requests.get(url.format(page=page), headers=headers)
soup = BeautifulSoup(res.text, 'html.parser')
4. Analyze the page to obtain data
house_list = soup.select('.list > li')
for house in house_list:
house_title = house.select('.title a')[0].string
house_location = house.select('.add > a')[0].string.strip()
house_price = house.select('.money > b')[0].string
house_data.append({
'title': house_title, 'location': house_location, 'price': house_price})
5. Data visualization analysis
df = pd.DataFrame(house_data)
df['price'] = df['price'].astype(int)
df['location'] = df['location'].apply(lambda x: x.split('-')[0])
sns.boxplot(x='location', y='price', data=df)
plt.show()
This example takes a branded apartment in the Beijing area as an example, crawls two pages of housing data, and uses a box plot to perform a visual analysis of housing prices in different regions.
It should be noted that the anti-climbing of 58 is more serious, and the IP is directly blocked if there are too many visits.
I also prepared 25 very practical Python crawler projects for you to help you learn crawlers better. You can also build your own solutions according to the needs of the project to improve your programming level. A full set of python self-study videos and projects have been packaged, and the business card at the end of the article can be taken away and studied directly!
Well, that's the end of today's sharing, see you next time!
