1. Introduction to BI system
1.1 What is a BI system
The full English spelling of BI is Business Intelligence, or BI for short. We often hear companies talk about "getting started in BI", "building a BI system", "building a BI decision-making platform", etc. So what exactly is BI?
(1) Originally originated from fixed reports
A few decades ago, when modern enterprises did not have BI, they had to produce reports on a fixed basis.
(2) The development of BI brought about by data warehouse OLAP technology
With the development of enterprises, only looking at fixed financial and sales reports can no longer meet the needs of the enterprise. Because the problems are becoming more and more complex and the situation is becoming more and more changeable, it is difficult to clearly see the business situation in fixed reports, and the support for decision-making is becoming more and more limited. , but from dozens of commonly used reports to developing hundreds or thousands of reports, the development amount is really too large. As a result, a technology platform like OLAP was created to quickly realize people's exponentially rising reporting needs.
Since the reports are all based on historical data, there is no need to directly obtain the data from the business system, and too many report generation will affect the business system itself, so the data warehouse came into being.
(3) Definition
BI is based on the massive data generated by online transaction processing (OLTP), extracting it from relational databases, and obtaining valuable information through online analytical processing (OLAP) or data mining and other technologies to provide decision support for managers. Allow enterprises to make objective and predictive decisions through data and achieve intelligent business operations. (BI is oriented towards business decision-making)
(4) Supplement
However, the actual situation is not so ideal. Both traditional industries and the Internet emphasize using data to speak. But looking at data and talking about data does not mean that you are "intelligent". OLAP analysis cannot achieve intelligence, and data mining analysis is difficult to achieve. The reason for realizing intelligence is the complexity and variability of business (business). This is why we always feel that there are so few good BI products.
1.2 Classification of BI systems
(1) Report format
Report-style BI mainly refers to various fixed-style report designs in products, which are usually used to present detailed business data and indicator summaries, and the amount of data supported is relatively small. Chinese reports have complex headers and Excel-like formats. The domestic reporting tool FineReport has good support for Chinese-style reports.
(2) Signboard type
Kanban-style BI mainly includes data Kanban, data big screen, data cockpit, etc. The characteristic is that the data indicators and display forms in the Kanban are basically fixed, with visual charts as the mainstay. Kanban-style BI has less support for data analysis, and due to fixed indicators , so it takes a while to upgrade Kanban BI after business changes, so the agility is weak.
(3) Platform type
Platform BI is mainly a BI self-service analysis platform. Its characteristic is that business personnel or business-side data analysts can configure the required data reports and data dashboards through simple front-end tools. The project cycle is short and easy to maintain later. With the data center With the rise of the concept, the BI platform has become a trend, and the data and technology departments have returned to the basics of technical support - providing support for the underlying technology and data, and letting business personnel explore the value of the data on their own.
(4) Summary
There is no distinction between these three types of BI products. Each has its own applicable scenarios. We need to choose appropriate BI products based on the characteristics of business development and actual needs.
For example: Compared with the report type, although the use of visualization makes the data more intuitive and vivid, the Kanban type is missing a lot of data details.
The self-service method is much more flexible than the Kanban method, but it has certain usage thresholds (for example, it may require the use of SQL) and is also limited by the level of data literacy of business personnel (for example, whether business thinking can be transformed into data analysis thinking).
The BI system we are talking about today is mainly report type and Kanban type.
2. Common misunderstandings in building BI systems
2.1 Made a lot of data indicators: failed to distinguish the importance of indicators
Question: When you are not following a product from 0 to 1, then you may not understand the various data of the product as your operations do. When you ask your operations what indicators are more important, because they are in different positions. Looking at things from different angles, you will eventually find a result: a lot of indicators, all important.
solution:
① You can ask HR or their department heads for the department’s performance appraisal indicators. Perhaps these are their most important indicators.
② You can communicate with the head of the department. Those are the indicators that he is more concerned about, so you should start with these indicators.
2.2 Indicators work but are of no use: distinguishing which are vanity indicators
Problem: I have done many common PV, UV, monthly active, total number of users, total number of products, etc., but in fact these are vanity indicators because they cannot directly promote the growth of transaction volume. No matter how much uv or monthly activity is, what’s the use? Users just won’t buy it.
Solution: Product managers need to identify which indicators are vanity indicators and which are more useful indicators (indicators have a direct effect on business goals). Generally, indicators that can directly promote transaction volume and have numerators and denominators such as conversion rate are non-vanity indicators. for example:
① The conversion rate of the main path in the business industry, the conversion rate of visit-product list, product list-product details, product details-additional purchase, additional purchase-order, these can increase transaction volume by reducing churn.
② Users’ next-day retention, 7-day retention rate (whether new users visit again after 7 days), 30-day retention rate, etc., can directly reflect the quality of users and the quality of operations.
③ The sales rate of the product (number of items sold/number of items on the shelves) can directly reflect the quality of this batch of products.
2.3 It is difficult to take into account the needs of various users: setting up BI dashboards by theme
Problem: The granularity of data that everyone pays attention to is different. What the boss pays attention to is different from what department leaders pay attention to, and what department leaders pay attention to is different from what front-line executives pay attention to.
Solution: In this case, the Kanban boards cannot be made together. Rather, they are distinguished by subject.
Within the theme, a certain story system is established according to certain data analysis logic and data viewing logic to help users understand the theme.
2.4 After the product is launched, the boss does not have a deep understanding of BI: How to reflect the product value
Question: You need to distinguish what type of business personnel your BI product is for. If your boss doesn’t feel deeply about it, he may not be a core user of this BI system.
Solution: After the BI product is launched, the project does not end, but the product operation must be continued:
① Carry out product research and understand users’ usage. Good products will definitely be liked by business personnel.
② Conduct usage training for business personnel, communicate with them more, continuously upgrade and iterate, and finally create a viable version.
③ Daily operation and maintenance must be done well to ensure timely and accurate data. Otherwise, no matter how much training and promotion is done, if the product quality is not up to par, there will be no reputation.
3. BI system construction
3.1 What modules does the BI system include?
- Data collection: internal data, external data · Data development ETL: data processing, putting data into the data warehouse, and developing application layer data according to BI requirements.
- Data algorithms/models: data mining algorithms, such as prediction, attribution...
- Front-end application display: front-end visualization, real-time updates, large data volume query in seconds
- Permission management: User access permission management, generally to pages and buttons.
- Monitoring: data anomaly alarm, daily access statistics.
3.2 Business research on BI system construction: the prerequisite to ensure that the product does not go astray
(1) Determine our business users and product goals. Generally speaking, there are two situations:
① If the project has a sponsor, then communicate with the sponsor’s leadership to understand the background of initiating the BI system project, what problems are to be solved, and clarify the general direction of the product.
② If it is a BI system project driven by the technical department itself, then during the project establishment phase, it is also necessary to conduct a survey on the current status of BI usage (or data usage) within the company.
(2) User research: user story map
BI applications are based on business processes and data. IT testers can only check whether the calculation results are accurate, but they cannot judge whether the analysis charts meet business requirements, whether the data results have commercial significance, etc.
User story mapping is a better method of demand research and demand sorting, which can establish the team's overall grasp of the needs without losing details.
Because we are not talking about user research today, we will not start it. If you are interested, you can find some information.
(3) Decide on product planning and selection
① Product planning plan: functional grouping and staging (priority)
② Selection: report type, Kanban type, platform type
3.3 Design principles of BI system
After determining the product plan and roadmap, the next step is to carry out system design. Here are a few design principles:
(1) Easy to use and accurate data
Whether the BI application conforms to user habits and whether the data is accurate and timely is the key to the survival of BI.
① Report-based BI: How to display the detailed data in the table in a more friendly way;
② Kanban-style BI: Focus on core indicators, storytelling between indicators, and analytical thinking;
③ Platform BI: Consider low-code support, visual operations, WYSIWYG, etc. as much as possible. After all, not many business people know SQL;
(2) Can be integrated with existing systems
It can adapt to the enterprise's existing database selection and ensure that the project can go smoothly during data collection/data access.
(3) Consider multi-scenario support
① PC and mobile usage scenarios
Try to consider future mobile and PC compatible solutions when designing the architecture. You must know that any BI system will eventually face mobility.
② Unified supported data sharing method
It is necessary to uniformly design data sharing methods such as data download and data export.
(4) Consider introducing algorithms and models
Don’t just regard BI as a “data-reading” tool. Appropriate consideration of the introduction of algorithms or the technical architecture of algorithm modules in the future during product design can improve the positioning of the product and help exert greater value.
(5) Consider data monitoring and permissions
The data anomaly monitoring mechanism and permission management must be considered in the architecture, otherwise it will be very painful later. In terms of permissions, attention should be paid to the design of data permissions (different positions on the same page will see different data content ranges. For example, the North China region can only see data from the North China region. This is the data permissions.)
4. Future development trends of BI systems
4.1 Analysis time
Analysis moment is a data analysis process defined by Gartner, which supports the delivery of business results by visualizing, exploring and applying algorithms to data, thereby making better or faster decisions and automating business processes. As the threshold for data usage gradually decreases, self-service and platform-based BI become a trend, and the initiative of data analysis will gradually be transferred to the hands of business personnel. Data analysis is directly initiated by business personnel who encounter business problems, and business personnel can use data analysis tools. /Platform completes data analysis content.
For example: When a business person wanted to know the online sales forecast of a certain product, or why the items in the shopping cart were not converted into purchases by some customers, in the past, the business person had to turn to the IT department for professional data analysis. Engineers (extracting potentially relevant data and outputting specific analysis reports), data scientists (building predictive models), but imagine if common predictive algorithm models and attribution analysis tools were established in the BI system, data sets could be easily connected. The business personnel can quickly complete the analysis content independently. Through self-service analysis, they can quickly know the data conclusions or the causes of the problems, and then make business feedback.
4.2 Enhanced analysis
Augmented analysis mainly refers to data analysis and BI functions based on machine learning. Through the application of machine learning, artificial intelligence and other technologies, common and general data analysis scenarios are precipitated into product functions, helping ordinary users without data science experts or Complete data analysis with the assistance of IT personnel. The underlying concept of enhanced analysis is "simple and easy to use", which can support users to complete the entire process of "collection-preparation-integration and analysis" of data without any professional knowledge background.
Enhanced analysis includes modules such as enhanced data preparation, enhanced machine learning, and enhanced data analysis. Here we mainly talk about application trends in data analysis.
(1) Applications of NLP and NLG
① Natural language analysis, no need to write SQL, analyze data through language and visualization
For example: ThoughtSpot uses search and NLP as the main interface for accessing data. Users can ask questions through typing or voice.
② Conversational data analysis, data conversation robot
Natural language is converted into SQL, and then the SQL result set is converted into a visual graph, forming a complete link of "NL2SQL2Graph".
Example: Alibaba Xiaomi (entry: Taobao)
③ NLG technology (natural language generation) presents the opinions and conclusions analyzed by the machine to the user in language form.
For example: Tableau's explain Data function will automatically provide an AI-driven explanation for the selected value. This feature checks hundreds of possible explanations behind the scenes and presents the ones with the highest likelihood.
(2) Automatic insights and automatic visualization ① Automatic insights (automated insights)
Automatic insight refers to machines automatically discovering potential information and value from data: discovering associations between data, discovering data anomalies, and automatically clustering.
Most of the mainstream BI platforms now have automatic insight-related functions.
For example: Microsoft's PowerBI's Quick Insights function can automatically perform various cross-cut first-order or multi-order calculations (percentage, sorting, same-cycle comparison) on source data to mine various trends within the data.
https://www.c-sharpcorner.com/article/quick-insights-and-power-bi/
②Automated visualization
Automatically select a visual display method based on the data results to clearly display the data analysis results.
There are 2 directions:
- Automatic chart selection: After querying the data set, the machine will automatically generate appropriate charts based on the characteristics of the data. Chart automation is now supported by mainstream BI tools, such as tableau. After selecting the data set, the first thing you see is not the data, but the automatic visual chart.
- Automatic report generation: A higher level than automatic chart selection, it automatically generates report layout, configuration controls, chart linkage, etc.
Technically there are 2 ways to achieve this:
- Rule-based: Set up the rule base in advance and generate charts based on the rules. The quality of the rule base is the key.
- Based on the model: converting the problem into a classification or ranking problem), the characteristics of the data itself and the characteristics of the visual chart are the key.
4.3 Embedded Analysis
Integrate specific data analysis methods into business systems. The pages of the BI system can be used to embed into other systems, which is very beneficial to the future development of the product:
- It can expand the applicable scope of BI and facilitate future product planning upgrades.
- Business personnel can see the analysis results of the data in the system that generates the data, which increases the experience, and the process and experience of use are smooth.
4.4 Forecasting and decision-making recommendations
The business experience summarized through a large amount of manual business analysis, coupled with the blessing of AI and machine learning technology, allows the machine to complete business analysis and action recommendations in one go. For example: Taobao's business consultant will provide corresponding marketing tools or learning materials based on data indicators.
Original text from: Everyone is a product manager