Reptile project combat 15: crawling weather information

Crawling weather information

aims

Crawl weather information and save the data to the database.

Project preparation

Software: Pycharm
third-party library: requests, pymysql, BeautifulSoup
API interface: http://wthrcdn.etouch.cn/WeatherApi?city=city name

Project Analysis

Visit the weather api interface
here to view the weather information of Suzhou,
so modify the api like this

http://wthrcdn.etouch.cn/WeatherApi?city=Suzhou

Insert picture description here
ok can be accessed
There are a lot of information here, so only crawl a part of it as a test.

Code

import requests
from bs4 import BeautifulSoup
print('请输入城市名称：')
cityname=input()
url='http://wthrcdn.etouch.cn/WeatherApi?city={}'.format(cityname)

response=requests.get(url)
content=response.text
soup=BeautifulSoup(content,'lxml')
#日期
date=[]
target=soup.find_all('date')
for each in target:
    date.append(each.text)
#最高温度
high=[]
target=soup.find_all('high')
for each in target:
    high.append(each.text)
#最低温度
low=[]
target=soup.find_all('low')
for each in target:
    low.append(each.text)
#light
light=[]
target=soup.find_all('day')
for each in target:
    light.append(each.text)

#night
night=[]
target=soup.find_all('night')
for each in target:
    night.append(each.text)

Print it out

length=len(date)
for i in range(length):
    print(date[i],high[i],low[i],light[i],night[i])

Insert picture description here
OK, crawling is complete

Connect to the database

Here, MySQL is used to
first create a database,
Insert picture description here
create a table

1. Import the module
导入第三方库：import pymysql

2. Connect to the database

# 连接database
database = pymysql.connect(
    host="主机名",#localhost
    port="端口",#默认应该是3306,注意这里不加引号
    user ="用户名",#mysql的用户名
    password ="密码",#设置的密码
    database ='数据库名',#使用的数据库名
    charset ='utf8')#指定为utf-8编码格式

3. Create a cursor object

cursor = database.cursor()  # 执行完毕返回的结果集默认以元组显示

4. Define the SQL statement to be executed

sql = "INSERT INTO weather(日期,最高温度,最低温度,白天,黑夜) values (%s,%s,%s,%s,%s)"
    value=(date[i],high[i],low[i],light[i],night[i])

5. Execute SQL statement

cursor.execute(sql,value)
database.commit()  # 提交数据库

6. Close the cursor object to close the database

cursor.close()
database.close()

Test effect

Insert picture description here

Has been stored in the database
, the effect is displayed in the command line

mysql> select * from weather;
+------------+-----------+-----------+------------+------------+
| 日期       | 最高温度  | 最低温度  | 白天       | 黑夜       |
+------------+-----------+-----------+------------+------------+
| 15日星期四 | 高温 21℃ | 低温 16℃ | 阵雨东北风 | 中雨东北风 |
| 16日星期五 | 高温 18℃ | 低温 14℃ | 中雨东北风 | 阴东北风   |
| 17日星期六 | 高温 21℃ | 低温 16℃ | 阴北风     | 多云北风   |
| 18日星期天 | 高温 20℃ | 低温 15℃ | 阵雨东北风 | 阵雨东北风 |
| 19日星期一 | 高温 18℃ | 低温 14℃ | 阴东北风   | 多云东北风 |
+------------+-----------+-----------+------------+------------+
5 rows in set (0.00 sec)

Complete code

import requests
from bs4 import BeautifulSoup
import pymysql
print('请输入城市名称：')
cityname=input()
url='http://wthrcdn.etouch.cn/WeatherApi?city={}'.format(cityname)

response=requests.get(url)
content=response.text
soup=BeautifulSoup(content,'lxml')
#日期
date=[]
target=soup.find_all('date')
for each in target:
    date.append(each.text)
#最高温度
high=[]
target=soup.find_all('high')
for each in target:
    high.append(each.text)
#最低温度
low=[]
target=soup.find_all('low')
for each in target:
    low.append(each.text)
#light
light=[]
target=soup.find_all('day')
for each in target:
    light.append(each.text)

#night
night=[]
target=soup.find_all('night')
for each in target:
    night.append(each.text)

length=len(date)
# 连接database
database = pymysql.connect(
    host="主机名",#localhost
    port="端口",#默认应该是3306,注意这里不加引号
    user ="用户名",#mysql的用户名
    password ="密码",#设置的密码
    database ='数据库名',#使用的数据库名
    charset ='utf8')#指定为utf-8编码格式
cursor = database.cursor()  # 执行完毕返回的结果集默认以元组显示
for i in range(length):
    # 定义要执行的SQL语句
    sql = "INSERT INTO weather(日期,最高温度,最低温度,白天,黑夜) values (%s,%s,%s,%s,%s)"
    value=(date[i],high[i],low[i],light[i],night[i])
    cursor.execute(sql,value)
    database.commit()  # 提交数据库
cursor.close()
database.close()

Interlude, guide to avoid stepping on pits

At the beginning, the data is written to the database, and the Chinese display is normal in the visualization tool. But the Chinese display on the command line is all garbled, just like this.

mysql> select * from weather;
+----------------+--------------+--------------+-----------------+-----------------+
| 鏃ユ湡         | 鏈€楂樻俯搴?| 鏈€浣庢俯搴?| 鐧藉ぉ          | 榛戝          |
+----------------+--------------+--------------+-----------------+-----------------+
| 15鏃ユ槦鏈熷洓     | 楂樻俯 21鈩?    | 浣庢俯 16鈩?    | 闃甸洦涓滃寳椋?     | 涓洦涓滃寳椋?     |
| 16鏃ユ槦鏈熶簲     | 楂樻俯 18鈩?    | 浣庢俯 14鈩?    | 涓洦涓滃寳椋?     | 闃翠笢鍖楅        |
| 17鏃ユ槦鏈熷叚     | 楂樻俯 21鈩?    | 浣庢俯 16鈩?    | 闃村寳椋?         | 澶氫簯鍖楅        |
| 18鏃ユ槦鏈熷ぉ     | 楂樻俯 20鈩?    | 浣庢俯 15鈩?    | 闃甸洦涓滃寳椋?     | 闃甸洦涓滃寳椋?     |
| 19鏃ユ槦鏈熶竴     | 楂樻俯 18鈩?    | 浣庢俯 14鈩?    | 闃翠笢鍖楅        | 澶氫簯涓滃寳椋?     |
+----------------+--------------+--------------+-----------------+-----------------+
5 rows in set (0.00 sec)

I checked a lot of information and read a lot of blogs. They all said: The command line input is show variables like 'char%';modified to utf-8 encoding.
Okay, just do it.

mysql> show variables like 'char%';
+--------------------------+-----------------------------------+
| Variable_name            | Value                             |
+--------------------------+-----------------------------------+
| character_set_client     | utf8                              |
| character_set_connection | utf8                              |
| character_set_database   | utf8                              |
| character_set_filesystem | binary                            |
| character_set_results    | utf8                              |
| character_set_server     | utf8                              |
| character_set_system     | utf8                              |
| character_sets_dir       | E:\software\mysql\share\charsets\ |
+--------------------------+-----------------------------------+
8 rows in set (0.00 sec)

Modified, again

Insert picture description here
This method is not feasible.
If you encounter this, please do so.
1. Find your own mysql installation location, find the my.ini file, edit and

modify the gbk code here, and don't forget to save it later.

2. Command line input: net stop mysql
here mysql service is closed
net start mysql
open mysql

C:\Users\acer>net stop mysql
MySQL 服务正在停止..
MySQL 服务已成功停止。


C:\Users\acer>net start mysql
MySQL 服务正在启动 .
MySQL 服务已经启动成功。

3. Run mysql from the command line

mysql> select * from weather;
+------------+-----------+-----------+------------+------------+
| 日期       | 最高温度  | 最低温度  | 白天       | 黑夜       |
+------------+-----------+-----------+------------+------------+
| 15日星期四 | 高温 21℃ | 低温 16℃ | 阵雨东北风 | 中雨东北风 |
| 16日星期五 | 高温 18℃ | 低温 14℃ | 中雨东北风 | 阴东北风   |
| 17日星期六 | 高温 21℃ | 低温 16℃ | 阴北风     | 多云北风   |
| 18日星期天 | 高温 20℃ | 低温 15℃ | 阵雨东北风 | 阵雨东北风 |
| 19日星期一 | 高温 18℃ | 低温 14℃ | 阴东北风   | 多云东北风 |
+------------+-----------+-----------+------------+------------+
5 rows in set (0.00 sec)

OK, the problem has been successfully solved.