I. Introduction
In daily life, there are various information and social platforms. These platforms are filled with a large amount of information, which contains a lot of useful data. However, this data cannot be obtained and is difficult to display, so there is a public opinion analysis system.
The public opinion analysis system is a comprehensive system that focuses on analyzing and displaying certain aspects of these data. For example, tap today's popular topics, public opinions on topics, public emotions, etc. Public opinion analysis helps to manage and mine information from social information and other platforms and display it to users.
What this article will do is to implement a public opinion analysis system. This article will focus on certain parts of the public opinion analysis system. There are various implementations of public opinion analysis systems. For the convenience of this article, the code-free platform smardaten is used. Using smardaten can reduce a lot of development work and enable rapid development.
2. System Overview
2.1 Public opinion analysis system
The public opinion analysis system is a highly comprehensive system that involves technologies such as crawlers, sentiment analysis, topic word extraction, and data visualization. The public opinion analysis system as a whole can be divided into three parts, namely data collection, data analysis, and data display. The overall structure is as follows, of which the upper layer is directly contacted by users:
First of all, data is needed, and the sources of data are diverse. You can use crawlers to crawl, or you can use open platform data or internal data. Organize the data into a fixed format for subsequent processing.
The data of the public opinion analysis system can be Weibo hot searches, Douyin hot lists, Baidu hot lists, etc. The comments corresponding to these list topics are also very important information.
After obtaining the data, it needs to be analyzed and processed. For example, keywords such as analysis, or topic sentiment analysis, or the psychological changes of the masses, etc.
After analyzing the data, it’s time to display the data. In contrast, the display of data is the most intuitive and is directly contacted by customers. Many times, large data screens have a sense of technology, making customers feel very high-end. Only when the customer is happy can the first two parts be guaranteed to be effective.
There are various ways to display data. The distribution of emotions can be displayed with a pie chart, the trend of emotional changes can be displayed with a line chart, and the current topic can be displayed with a word cloud. In addition to these, there are many other display methods, which will be introduced in detail during subsequent implementation.
2.2 Introduction to smardaten platform
smardaten is a no-code platform that allows users to focus on the logic of the program without writing complex codes and develop applications quickly. smardaten uses a visual method to build programs. There are many components built into the platform, including layouts, templates, charts, multi-level menus, etc. You can just drag and drop them when using them.
Using smardaten can quickly develop many complex applications, such as GIS analysis, knowledge map, data large screen, etc. Taking "Power Industry - Distributed Photovoltaic Service Platform" as an example, the page requires multi-level menus, charts, paging and other functions. Using smardaten can quickly implement these functions. The figure below is an example.
smardaten also supports various mainstream data sources, including MySQL, Oracle, Access and other common databases, as well as graph database Neo4j, etc. It also supports Rest-api, etc. The following is an example:
In addition to directly obtaining data from the database, the data can also be collected and processed through ETL to obtain the data form we want. Easier to handle.
In addition, some logic control is also needed in the system. In smardaten, logic can also be implemented in a visual way. For example, after clicking a button, various responses need to be made. These responses can be displayed in the form of a list to facilitate user settings:
Different actions correspond to different sub-operations, so the visualization method can also realize very complex logic, and there is no problem in developing complex programs. It is precisely because of the convenience of development that the smardaten platform is chosen for development.
In addition, smardaten has a built-in supermarket, and you can download template applications. For details, see: http://appstore.sdata1010.cn/
3. System construction
smardaten can be used directly or deployed offline. For details, please refer to: https://s3.smardaten.com/ . We can use it directly to build applications. When implementing the system, we can divide the system into three parts: data access, public opinion analysis, and data display. Each part is implemented separately below.
3.1 Data access
Before you start creating your application, you need to prepare your data. The source of data can be the API of the open platform or your own database. This application uses a database to store data. smardaten supports various mainstream databases such as MySQL, Oracle, MongoDB, and SQLServer. The specific usage is roughly the same. This application uses MySQL as a data source.
First enter the main page: https://s3.smardaten.com/home, move the mouse to the upper left corner, select the data source, select MySQL, and fill in the connection information. Specific steps are as follows:
When filling in the database connection information, there are three main parts to fill in, as shown below:
They are:
- Database connection string, in the form: jdbc:mysql://ip:port/mydb, such as jdbc:mysql://127.0.0.1:3306/text
- database username
- Database user password
After filling in, you can click Test Connection to test connectivity. Then click Save. After saving, the data has not yet been actually accessed. We still need to collect the data and set the collection rules. After setting the collection rules, click to start collection. At this time, the input is actually connected to the assets. We can see the collection results in the asset list:
In this case, the database mainly has two tables. The relationships between the fields of the tables and the tables are as follows:
Now that the data access is completed, the data can be processed.
3.2 Data preprocessing
Data from multiple data sources is often not uniform, so the data needs to be processed. The processing here includes data cleaning, data integration, etc. Both can be implemented using data flows, which are created as follows:
Some simple processing tasks can be handled with formatting:
By integrating the data from the two tables, you can process the fields and then create a table to output the data. At this time, you can get a new table after running:
After creating a new table, you can view the assets corresponding to the new table in the data library:
These data can be used directly for display later.
3.3 Public opinion analysis
Public opinion analysis can have many levels of analysis. The first and most basic one is text sentiment analysis. In this application, we assume that this part of the operation has been completed, and we will do a macro analysis on this basis. Here are a few analyzes to be implemented by this application:
- Quantity statistics: analyze the proportion of positive and negative comments and visualize them
- Transformation trend statistics: analyze the emotional transformation in each time period
- Statistics on the emotional breakdown of each topic
In addition, some other analyzes can also be added, and the specific implementation is similar to the above. Several of the above analyzes can also be analyzed using data flows.
After creating a data stream, you can perform various operations on the numbers, including field selection, aggregation, sorting, etc. First, we complete the first statistic mentioned above, that is, counting. We want to count the specific situation of the sentiment field of the comment table. At this time, the data flow can be as follows:
Just select the comment table in the input data source and select the sentiment field in the statistics. The final output input is the statistical result.
The second one requires group statistics. For convenience, we can convert the time into months and years, and then count the time. This can achieve the same effect. The specific data flow is as follows:
The time formatting operation can convert the time into a specific format. If you want to observe the change of emotions on a monthly basis, convert the time format into years and months. If you want to observe the change of emotions on a daily basis, convert the time format into years, months and days, that is, Yes, the specific operations are as follows:
If you want the application to be richer, it can also be used as an interaction to let users choose how to collect statistics.
Then there is the last one, and the third analysis requires table connection operations. Table connection operations can be implemented using connections . At this time, the data flow is as follows:
Connect the topic and comment tables, and then count the sentiment field of comment. In this way, we have completed the analysis operation. By dragging and modifying the data flow, we can complete many very complex analyses. The next step is to display the data.
3.4 Create application
Before displaying, you need to create an application, in this order: create application, WEB+mobile terminal, enter the application name, and complete the creation. The specific operations are shown in the figure below:
After creating an application, you will enter the interface for developing applications. The interface is mainly divided into four parts. Each part is responsible for different functions. The left side is responsible for page management, which can create and bind pages; the right side is for specific setting information, which can be completed. Component modification, data binding, etc.; the top is responsible for adding components, setting layout, etc.; the middle is for effect preview. details as follows:
After creating the application, you can complete the subsequent work.
3.5 Implementation
Next, we design the overall framework of the application. The application is divided into three pages, namely hot topics, sentiment analysis, and sentiment monitoring.
Among them, hot topics are used to display topic data, sentiment analysis is used to display the sentiment distribution of comments, and sentiment detection shows the trend of sentiment changes over time. After knowing this, you can add the corresponding page in the application. You can add navigation and pages in the upper left corner of the page:
Hot topics are mainly to display topic information, which can be displayed according to different columns of different authors. Here we use a three-column layout nested table to achieve:
The title part can be bound to the author field, here I use a fixed value. The following is to bind each column to specific data. Click on a column and select Advanced Development. At this time, there will be a button in the list. We can remove the button in the button configuration. The next thing to do is to add the component and bind the data. Select the field to be displayed on the left, and bind the field to the asset on the right, so that the page will also change when the data changes:
Just do the same for the other two. If you want to be richer, you can perform different filters on different columns. The following is the display effect:
We can also do some beautification work on this basis. The next step is to complete the sentiment analysis part. Sentiment analysis can be implemented in a pie chart. However, it is a bit monotonous to only display the sentiment distribution. Some other data can be displayed. Here we show some positive and some negative comments. The overall layout is as follows:
The display operations on the upper and left sides are the same as before, and will not be repeated here. Next we come to the data binding of the above pie chart.
Pie chart data can be static data or can be bound to assets. Here, select the asset corresponding to the first data stream in 3.2. After successful binding, the emotional distribution of positive and negative comments can be displayed:
The second pie chart can be bound to similar assets, and I didn't create anything extra here.
Finally, the emotional monitoring is displayed, and the specific operation is very similar to the pie chart. We select the line chart in the chart:
Then bind the assets like the pie chart, here bind the assets corresponding to the second data flow in 3.2. In order to enrich the page, we can also display the data of the third asset on the current page. We can use tabs to display. Select tabs for different topics to display different data. First, add tabs to the page and set the conditions for the tabs:
In this way we have completed the application construction. However, there are still many details worth modifying.
Here is the specific operation of some analysis, we can do a richer analysis of the system, the following is the final effect:
3.6 Others
In practical applications, the displayed content can be richer. smardaten also provides more rich display components. For example, you can use richer chart displays. There are also radar charts, 3D charts, etc. in smardaten.
Or it can be displayed by region. Geographical location-related components are provided in smardaten. If topics and city information can be obtained, they can be displayed in combination:
4. System summary
The system as a whole has three parts, namely data, analysis and display. The sources of data are diverse, so feature data is required, which can be implemented with data streams in smardaten, and different data streams can be created according to different requirements. The analysis in this system only performs sentiment analysis, which can be implemented using the API of the open platform, or you can write additional programs yourself. The last step is display. Different data are displayed in different ways. The most intuitive display is to use charts. Displayed components need to be bound to data. For more information, please refer to: https://s3.smardaten.com/ , readers can experience it on their own.