[Python Original Graduation Project | Course Design] National Meteorological Data Collection and Visualization System Based on Python Flask - Attached is the download method and project reference papers. Other original projects are plagiarized.

1. Project Introduction

This project is a real-time meteorological data visualization system based on Web technology. By crawling the meteorological data of various cities from the China Weather Network and saving it into our own database, we used technologies such as Python, Flask, ECharts and MySQL to achieve data analysis and visual presentation. This is a modern meteorological data management system using B/S architecture, which users can access through a browser. In the improved version of the project, we further optimized data processing and visualization, improving system performance and user experience.

2. Project technology

• Python crawler technology : With the help of Python's powerful web crawler library, we can efficiently obtain real-time meteorological data from the China Weather Network and convert it into structured data.
• Flask back-end framework : As a lightweight back-end framework, Flask simplifies the development process and allows us to quickly build flexible and scalable web applications.
• MySQL database : As a mature relational database management system, MySQL provides efficient and reliable data storage and query capabilities, allowing us to easily manage the crawled meteorological data.
• ECharts Big Data Visualization : As a popular data visualization library at the time, ECharts provided us with a wealth of chart types and interactive features, allowing us to display huge meteorological data to users in the form of intuitive charts.
• layui background management front-end framework : With the help of the layui framework, we have implemented a beautiful and concise background management interface, improving the friendliness and convenience of user operations.

3. Project functions

The design of system function modules is based on an in-depth understanding of practical needs and focus on user experience. The detailed functional structure diagram is shown in the figure below. It mainly includes the following five functions:

• Visualization function module : Through ECharts technology, the crawled meteorological data is converted into chart form to display the meteorological conditions of each city in an intuitive way, allowing users to quickly grasp meteorological trends and changes and make better decisions.
• Version management function module : used to record different versions of the system, including function updates, performance optimization, bug fixes, etc., to ensure the continuous development and improvement of the system.
• User management function module : realizes user registration, login and permission control. Different users have different menu permissions to ensure system security and data privacy.
• Real-time meteorological data management function module : used to store, update and manage real-time meteorological data crawled from the China Weather Network to ensure the accuracy and timeliness of the data.
• Crawler management function module : Manage crawler tasks, including setting the crawling frequency, specifying target cities, etc., to ensure the efficient and stable operation of the crawler.
  

4. Run screenshots

Large screen visualization page

User login page,
user registration page,
background management home page,
background user management page,
background version management page,
background weather management page,
background weather editing page,
background crawler log management page

5. Classification description

AQI：

0-50 good

51-100: Medium

101-150: Bias, unhealthy for sensitive populations

151-200: poor, unhealthy

201-300: Extremely poor, very unhealthy

300+: Toxic

Wind level:

6. Implementation code

Crawler head implementation code

class GetWeather:
    def __init__(self):
        self.baseUrl = r""
        self.headers = {
    
    'Accept': "*/*",
                        'Accept-Encoding': 'gzip, deflate',
                        'Accept-Language': 'keep-alive',
                        'Connection': '',
                        'Cookie': ''.encode("utf-8").decode("latin1"),
                        'Host': 'd1.weather.com.cn',
                        'Referer': '',
                        'User-Agent': '', }
        self.loadList = []
        # 格式为：列表里面的子列表都是一个省份的所有城市，子列表里所有元素都是字典，每个字典有两项
        self.cityList = [] 
        self.cityDict = {
    
    }
        self.result = xlwt.Workbook(encoding='utf-8', style_compression=0)
        self.sheet = self.result.add_sheet('result', cell_overwrite_ok=True)
        self.cityRow = 0
        self.totalGet = 0

Data cleaning and storage in database

 data = json.loads(self.htmlResult.replace("var dataSK=", ""))
                nameen = data["nameen"]  # 城市拼音
                cityname = data["cityname"]  # 城市名称
                temp = data["temp"]  # 当前温度
                WD = data["WD"]  # 风向
                WS = data["WS"].replace("级", "")  # 风力
                wse = data["wse"].replace("km/h", "")  # 风速
                sd = data["sd"].replace("%", "")  # 湿度
                weather = data["weather"]  # 天气
                record_date = data["date"]  # 时间
                record_time = data["time"]  # 时分
                aqi = data["aqi"]  # 时分
                judge_sql = "select count(id) from `weather` where nameen = '" + nameen + "' and cityname='" + cityname + "' and record_date='" + record_date + "' and record_time='" + record_time + "'";
                sql = "INSERT INTO `weather` VALUES (null, '" + nameen + "', '" + cityname + "', '" + record_date + "', '" + record_time + "', " + str(
                    temp) + ", '" + WD + "', " + WS + ", " + wse + ", " + sd + ", '" + weather + "', " + aqi + ", '" + time.strftime(
                    "%Y-%m-%d %H:%M:%S", time.localtime()) + "',0);"
                i = db.query_noargs(judge_sql)[0][0]
                if int(i) > 0:
                    print("跳过：", judge_sql)
                    continue
                update_sql = "update `weather` set is_old=1 where nameen = '" + nameen + "' and cityname='" + cityname + "'";
                print("插入：", sql)
                count += 1
                db.query_noargs(update_sql)
                db.query_noargs(sql)

HomeCity Air Quality Statistics

# 获取城市空气质量统计
def get_AQI_total_data():
    db = dbUtil()
    lh_sql = "SELECT COUNT(id) FROM weather WHERE aqi<=50 AND is_old=0"
    zd_sql = "SELECT COUNT(id) FROM weather WHERE aqi>50 AND aqi<=100 AND is_old=0"
    pc_sql = "SELECT COUNT(id) FROM weather WHERE aqi>100 AND aqi<=150 AND is_old=0"
    c_sql = "SELECT COUNT(id) FROM weather WHERE aqi>150 AND aqi<=200 AND is_old=0"
    jc_sql = "SELECT COUNT(id) FROM weather WHERE aqi>200 AND aqi<=300 AND is_old=0"
    yd_sql = "SELECT COUNT(id) FROM weather WHERE aqi>300 AND is_old=0"
    lh = db.query_noargs(lh_sql)[0][0]
    zd = db.query_noargs(zd_sql)[0][0]
    pc = db.query_noargs(pc_sql)[0][0]
    c = db.query_noargs(c_sql)[0][0]
    jc = db.query_noargs(jc_sql)[0][0]
    yd = db.query_noargs(yd_sql)[0][0]
    db.close_commit()
    return jsonify([{
    
    "time": "良好", "value": lh, "name": "空气质量"},
                    {
    
    "time": "中等", "value": zd, "name": "空气质量"},
                    {
    
    "time": "偏差", "value": pc, "name": "空气质量"},
                    {
    
    "time": "较差", "value": c, "name": "空气质量"},
                    {
    
    "time": "极差", "value": jc, "name": "空气质量"},
                    {
    
    "time": "有毒", "value": yd, "name": "空气质量"}])

Air Quality Visualization Construction

function center_kqzlfm() {
    
        
    $.ajax({
    
           
        url: "/main/aqi",
        method: "get",
        success: function (obj) {
    
    
            // 基于准备好的dom，初始化echarts实例            
            const myChart = echarts.init(document.getElementById("kqzlfm-table")) 
            // 指定图表的配置项
            let option = {
    
    ...}            
            // 指定图表的数据
            let data = obj            
            //数据处理 开始
            let xKey = "time" 
            let yKey = "value" 
            let sKey = "name"
            let seriesArr = []
            let dataColumn = []   
            //....省略数据格式化处理太长了
            option.yAxis.type = "value"
            option.xAxis.type = "category"
            option.xAxis.data = dataColumn
            // 数据处理完 
            option.series = seriesObj
            // 使用刚指定的配置项和数据显示图表。
            myChart.setOption(option) 
        }   
    })}

7. Database structure

slog table

Field Name	type of data	Is it required?	Comment
id	int	yes
log	varchar(255)	no
create_time	datetime	no

sys_version table

Field Name	type of data	Is it required?	Comment
id	int	yes	system version
sys_name	varchar(255)	no	name
sys_version	varchar(255)	no	describe

user table

Field Name	type of data	Is it required?	Comment
id	int	yes
name	varchar(255)	no	Username (Vendor Name)
account	varchar(255)	no	user account
password	varchar(255)	no	user password
company	varchar(255)	no	Company Name
phone	varchar(255)	no	telephone number
mail	varchar(255)	no	Mail
type	int	no	0 administrator, 1 ordinary user
status	int	no	0 disable 1 enable

weather table

Field Name	type of data	Is it required?	Comment
id	int	yes
nameen	varchar(255)	no	City Pinyin
cityname	varchar(50)	no	city ​​name
record_date	varchar(50)	no	weather time
record_time	varchar(50)	no	real time time
temp	int	no	Current Temperature
wd	varchar(20)	no	wind direction
ws	int	no	Fengli
wse	int	no	wind speed
sd	int	no	humidity
weather	varchar(20)	no	weather
aqi	int	no	air quality
create_time	datetime	yes	Data creation time
is_old	int	no	1 old data, 0 new data

8. Source code download

The source code, installation tutorial documents, project introduction documents and other related documents have been uploaded to the official website of Yunyuan Practical Combat. You can obtain the project through the official website below!