Python-based Weibo big data public opinion analysis, public opinion sentiment analysis visualization system

Running effect diagram

system introduction

Weibo public opinion analysis system, the project backend consists of crawler module, data analysis module, data storage module, and business logic module.

Data acquisition, screening and storage have been carried out successively, operations such as extraction, analysis and processing of stored database data have been carried out to obtain structured data that meets the needs, and the processed data has been analyzed according to needs to obtain relevant visual data, and then provide corresponding The interface is given to the front-end page and displayed in the front-end page of the project.

Use flask to build the entire project framework. Part of the crawler module uses the requests official dependency library for data acquisition, obtains structured json data according to the API interface disclosed by Weibo, and then uses pandas to filter, process and analyze the data.

Partial renderings

Function icon

Function page path map

From the perspective of the front-end page, make an overview of the functional paths of the system. The main functional paths can be seen directly in the figure. Different color blocks in the figure represent different module functions and pages. The standardized software uml can be used if conditions permit.

QQPCRealTimeSpeedup_I6hUnKetTS

Overview of using technologies

python3, flask framework, hanlp for text symbol processing, numpy, pandas for data analysis and processing, sqlite, mysql database, SQLAlchemy for ORM framework, snownlp natural language sentiment value analysis
vue3, vue-template development

Backend code file structure records

The main dependencies used

beautifulsoup4
requests
Flask
hanlp
pandas
numpy
PyMySQL
SQLAlchemy
snownlp
jieba
The library related to the ~~xlrd operation file is not in use~~

file structure/weibo_sentiment

Data analysis calculation module/analysis
Database configuration module /database
Crawler module/scrapy
ORM's model module/model
Front-end and back-end interaction controller app app.py
static resource/static
Use help help.md
Project depends on requirements.txt

Front-end code file structure record

technology used

Part of the vue technology of the vue family bucket is developed on the basis of vue-template, without having to reinvent the wheel, which can save a lot of development time and improve development efficiency.

The main technologies used by Vue:

axios
echarts
element-ui
vue
vue-router

Important file structure of the front end

/node_modules dependency file management library
/public Public static resource storage folder
/src core files
/api front-end API interface request method
/components page reuse components
/router page routing panel
/styles The style library referenced by the page
/utils tool library
/ views view page
/static Static resource files
/tests test module files
package.json configuration file for vue project
vue.config.js vue project proxy, routing and other configuration files

Source code download

Do Not Disturb