This article is a summary of the application of big data in the Internet industry by Yu Zhongyang, senior manager of the Information System Department of 58.com. Based on practical experience, he explained the development of big data from prototype to development, the transformation from traditional data warehouse to big data platform and the visualization application of big data in the process of development.
Big data takes shape - development
At the beginning of all companies, the construction of data was relatively backward, but with the promotion of the Internet environment and the company's plan to list on the New York Stock Exchange in 2013, it faced a big problem at this time. Before going public, the company's data, whether it is traffic data or financial product data, needs to go through a very strict audit. To this end, we began to organize data, make data reports, and spent more than a year to organize and standardize data. Slowly, the big data platform took shape.
After the company went public, the focus of data operations shifted from reflecting past and present conditions to guiding the business, managing performance, and supporting sales.
In the two years from 2015 to 2016, the company focused on data intelligence, established a very large system platform, and fully implemented data-based operations in the entire sales and product operation system. On this platform, we not only have to manage sales, but also build a sales capability model and predict future performance. Because the stock price in the U.S. market is more dependent on the company's future development, the company's future expectations determine the company's current stock price, so the total performance forecast is very important, and it is even more important to be able to predict sales very accurately.
Big data platform construction
The company has a lot of business databases, including orders, CRM, contract management, human resources and other databases. In the past, we would integrate the data in these databases to make a simple data warehouse, and then according to business needs, on the basis of data integration, we would build a decision-making system through FanRuan reports to display data and provide services. The advantages of this process are that the development cost is low, the investment cost is low, the technical architecture is simple, and it can be run very quickly within the company.
However, with the increase of business systems, this method gradually becomes inapplicable. First, the business data is diversified, the data is not centrally managed, and it is difficult to effectively use the data; the data storage capacity is limited, and it is impossible to trace historical data far away. Second, in many cases, the requirements put forward by different product managers and engineers are different, and these different requirements themselves are intertwined. Therefore, different teams make the same indicators. Due to inconsistent definitions of data indicators, when there are duplicate indicators with similar meanings, it is difficult for management to make decisions. Third, the data channels used for analysis and operation are completely dependent on the monitoring of their own data, which will lead to a lot of pressure on the IT department.
In addition to the problems reflected above, the company has presented a new round of status quo based on future development, and put forward the following requirements:
- The diversified development of business and the large number of mergers and acquisitions of the company have led to the explosive growth of the group's data volume and data demand;
- Data mesh circulation, basic data and data indicators lack unified metadata management;
- In order to better explore the value of data, it is imperative to improve data capabilities and build a standardized data system.
Big Data Platform Business Architecture
The Group's business structure can be roughly divided into several parts. The first is the data modeler. The main work is to manage metadata, develop data models and establish unified data standards. The second is the data developer, who will connect with the product and then do some business development. The third is the business people. The business people are the operations, sales, and marketing people, and a lot of visualization is done in this place to help them make decisions. Finally, the company's data analysis department has a lot of analysis requirements for auditing, including some very core data of the company, and they will implement multi-dimensional data analysis through an independent data query system.
The figure below is the technical architecture. Data storage is performed at the bottom layer, data is captured, and the data of the storage layer is transmitted to the processing layer, where business calculations are completed to form indicators. Then, through the application of FanRuan report, the front end displays the data graphically in the form of business report, and displays it on the mobile report APP.
In addition to this, we have also made the construction specifications of the data center in terms of management. Then there is data quality management, including ensuring the accuracy of the data, and establishing an operational and management specification to ensure the timeliness of the data. These four pieces combine to ensure smooth operation of the platform.
big data visualization
Data visualization is the last and most widely used step of this platform. In this regard, since we
1. There are many sources of demand: The team needs to connect with the finance department, sales center, customer service center, management and other departments at the same time, and the indicators concerned by each department are very different.
2. There are many ways to visualize reports: detailed tables, summary tables, drill-through tables, and various charts are needed to meet the needs of different management levels in each department, and offline reports, real-time reports, emails & text messages are also provided to the demander. Push data and other channels.
3. More custom development: In order to connect with the company's internal business system, it is necessary to custom develop frequently.
Therefore, a visual report development tool that meets these needs is needed. In this piece, we have chosen FanRuan reporting, no matter from the original traditional data warehouse method or the current big data platform.
The advantages are as follows:
- EXCEL-like design style: The operation interface greatly reduces the cost of learning.
- Multiple data source support: It is very convenient to access various types of databases in various departments.
- Excellent icon display: HTML5 chart technology, supports a variety of chart types, styles, styles, flexible parameter transmission, and rich interactive effects.
- Friendly interface: Developers use web scripts, API interfaces, etc. for in-depth development and control, and support plug-in development, installation, use and management to meet their individual needs.
Visual display:
