What can business intelligence BI do

Thoughts derived from what business intelligence BI can do

Summary

Business intelligence, BI, Spark, hive, ETL, powerBI, FineBI, PaddlePaddle, artificial intelligence, data mining, DBA, logistics, user profile modeling

Preface

In my study plan from mid-September to mid-December , I wanted to find a way to learn my own direction to make myself more valuable. First try to determine the direction of your own development by analyzing your own positioning. Starting from what I will be and what I am interested in, I prepare the technology stack. At that time, it was still relatively vague and I was exploring.

Today (9/18) it suddenly occurred to me that I don’t need to fumble, I can combine with my work, what work I plan to do in half a year, then I will prepare the technology stack and portfolio of this work.

In fact, when I learn technology, I still find a job. After all, ideals need bread support. If I learn a lot of technology but can't find a job, it will not help me continue to do technology.

When this idea came into being, I immediately started looking for business + data analysis related positions. Then prepare your own skills through the skill requirements of these positions.

Look at BI from the hook

Work-oriented programming. When a job is mature in the market, the job will be specialized and evolved into a job type that can be strengthened based on job skills to be competent for the job.

Business data analysis in the market has been around for several years, so it stands to reason that it should be generalized. So I tried to search for BI (Business Intelligence) on Boss Direct Recruitment and Lagou, and I found out the corresponding job requirements.

In the accounting (Lesson 1) notes , I have searched for the position of data analyst, but I still collect data-process data-display data. It always feels like something is a little bit short, but if you think about it carefully, it should not be combined with business (business).

In the data analyst’s technology I’m looking for, it’s said

The three skills required for business data analysis can be summarized from the table

• Database: Proficient in SQL, familiar with database tools such as spark and hive.
• Data processing: Use Python, familiar with commonly used data statistics and analysis methods;
• Data visualization: tableau, powerBI.

In fact, it is similar to doing a website, getting data, processing data, and displaying data. Among them, machine learning models, data reports, database tools such as spark, hive, Python analysis, powerBI and other data visualization tools have not been in contact with me yet, so I can find out.

But there are still some gains, at least let the data analysis land. I felt that the data analysis was almost something, and it turned out to be poor in the direction of analysis. After all, it is embarrassing to say that there are so many things, but in the end not knowing what to do.

View BI from the database

It's the second day, and I didn't finish it yesterday, so I added the last part. Today (9/19) because of a text message sent by CSDN on the mobile phone-Baidu Smart Cloud TechDay Computing & Network Technology Innovation Salon-Beijing Station. Then searched TechDay Computing & Network Technology Innovation Salon. I clicked on an Alibaba Cloud link (some are funny, originally searched because of Baidu TechDay, but I went to see Ali's TechDay).

This link talks about the number one topic in the cloud . It's June 9, 2020. Talk about traditional database technology and Alibaba Cloud native database. The host is Hulan (Master of Actuarial Science, Columbia University), and the guests include Li Feifei (Vice President of Alibaba Group, Head of Alibaba Cloud Database, Chief Scientist of Dharma Academy, Outstanding Scientist of ACM), Gai Guoqiang (Oracle), Chen Gang (Dean of School of Computer Science, Zhejiang University). They are all big names in the database.

At first I just clicked and watched it casually. I didn't expect the content to be good and I was attracted.

The anxiety of middle-aged programmers. Writing code has become a labor-intensive enterprise. DBA. Plan ahead. Programmers are also writers. Anxiety is normal. You must have a sense of crisis to continue learning. Don’t make your debut as your peak. Those who love anxiety are not too bad luck.

Maimai Industry Analysis Report "10 Years of Changes in IT Industry 2009-2019". Changes in popular companies. Said the hot job change. It used to be hard work, but now it is stress resistance. Golden decade. Technology explosion. Traditional databases have gone through the most glorious era. Ten years ago, the most popular position was DBA, now it is data architect and algorithm analyst. DBA is to execute SQL, transfer data, and run other people's SQL. Changes in the environment have brought about an explosion of species. The technological explosion brings opportunities and adjustments. Changes are fast. Normal input, through writing program, output. The 70-year-old grandmother automatically connects to WiFi, portable code, etc.; the change is big. Alipay opens various green codes.

Where does the industry's DBA and operation and maintenance go? Practitioners are anxious, specifically anxious. In the last mile of the database, DBA has the future. Three hundred thousand DBAs have smoothly transitioned to the cloud era. The initial stage of cloud database. Don't be too anxious, there are opportunities and opportunities everywhere. Everything in the past is a prologue. New technology. The most advanced productivity. For operation and maintenance, the database is critical. Those who win the ecology have the future. One main engine and one standby engine.

Interactive topic. Is it difficult for traditional database engineers to transform? Make efforts to prepare for transformation. (Database installation, backup, operation, monitoring, inspection), evolve to data architect and algorithm engineer. The DBA has not changed much, in small changes, I don't want other IT fields. But cloud databases are the trend. Water wells (commercial database) and water plants (resource pooling). High availability is irreversible. The epidemic brought companies to the cloud. Build your own machine, resource consumption. The general trend, recognize the situation, follow the trend, and embrace changes.

What are the differences and similarities between cloud native databases and traditional databases at the product and technical level, and what should be paid attention to during the transformation process. Upper layer (car washing), middle layer (car building). Highly compatible with existing ecology. The core technology of cloud database: cloud native, distributed, intelligent, safe and feasible, deep integration of database and big data technology; 5G+IoT for multi-mode database. The gasoline-powered car has reached the electric car. New things. The Spring Festival Gala sneaked into the night with the wind, moistening things silently. Electric cars are worried about running out of electricity, gas cars are reliable.

Pretty good (I didn’t write every line in detail above, the knowledge lists the key words, the interest is worth seeing, the quality is high, and I read it three times), the host asked the question, and then the representative of the traditional database said his opinion, Then the representative of the cloud native database said his views, and the academic teachers said his views. Finally, discuss together.

Obviously, the traditional database represents the industry experience of the traditional database Oracle; the representative of the cloud native database represents the industry experience of Alibaba Cloud; the academic teachers represent the views of the academic community.

The combination of these three views is very interesting. In addition, from a standpoint, the significance of this video is to let technicians understand cloud databases.

Inspiring sentence

Ten years ago, the most popular position was DBA, now it is data architect and algorithm analyst. The future will also change. People who love anxiety will not have bad luck (a sense of crisis, take precautions).

Everything in the past is a prologue. From another perspective, it is not industry replacement, but industry progress. The past is a prologue, now I begin my story.

One main engine and one standby engine, both open source and commercial are suitable. The same goes for learning technology, there is a spare skill. Prevent black swans. Business will be sudden. Such as big V recommendation.

Water wells (commercial database) and water plants (resource pooling). This is the trend.

Some technology debuts have high salaries, but they are actually a reward for the frequent and rapid transformation of this industry. There will be a big improvement a year ago and now.

5G+IoT vs. multi-mode database. Cloud edge integration.

Looking at BI from Spatio-temporal Big Data

After reading the cloud database, I had great expectations for these big talks. Then I clicked on other links and found that most of them were personally speaking PPT or introducing products. Overjoyed. However, there is a talk about [Logistics Technology Review] Issue 9 01: Several technologies and applications of space-time big data , which is pretty good. When I work, I also work in a spatiotemporal big data company. I am very interested in the real-time changes of spatial geography on the map.

The application scenarios of this spatio-temporal big data are very wide. Such as logistics, subway, transportation, housing prices, route planning, etc.

The content of spatio-temporal big data processing is the spatial and temporal information of the object, and the data that needs to be obtained is spatial information, etc.; then the information is processed; then the information is displayed; and finally a conclusion is drawn.

The content of business intelligence processing is report information. You need to get the report data, then process the report data, and finally display the report data.

But business also needs time and space. Some commodities are closely related to regions and quarters.

Explanation of professional terms

spark

Apache Spark is a fast and versatile computing engine designed for large-scale data processing.

Spark has three main characteristics:

• First of all, the high-level API strips the focus of the cluster itself, and Spark application developers can focus on the calculation itself that the application needs to do.
• Secondly, Spark is very fast, supporting interactive computing and complex algorithms.
• Finally, Spark is a general-purpose engine that can be used to complete a variety of operations, including SQL queries, text processing, machine learning, etc., and before the emergence of Spark, we generally need to learn a variety of engines to handle these separately demand.

In short, spark is a general-purpose engine that can perform SQL queries (sparksql), text processing (sparkStreaming), machine learning (MLlib), and graph processing GraphX.

Hive

Hive is a data warehouse tool based on Hadoop for data extraction, transformation, and loading. This is a mechanism that can store, query and analyze large-scale data stored in Hadoop.

It seems that sparkSQL is better. The function is more complete.

ETL

ETL (extract-transform-load) technology. It is used to extract, transform, and load data from the source to the destination. The term ETL often appears in data warehouses, but is not limited to data warehouses.

ETL is the process of extracting, cleaning and transforming the data of the business system and then loading it into the data warehouse. The purpose is to integrate the scattered, messy, and inconsistent data in the enterprise to provide an analysis basis for the decision-making of the enterprise. ETL is BI ( Business Intelligence ) An important part of the project.

This link is interesting. I imagined this part of the work before, at that time I classified this part as the database. However, it is appropriate to extract this part of the work from the database.

Through design, improve the efficiency of database use.

Business Intelligence

Introduction to Business Intelligence

Business Intelligence (Business Intelligence, referred to as: BI), also known as business intelligence or business intelligence, refers to the use of modern data warehouse technology, online analysis and processing technology, data mining and data display technology for data analysis to achieve commercial value.

Business intelligence is generally understood as a tool that transforms existing data in an enterprise into knowledge and helps the enterprise make wise business operation decisions

In short, a tool to help companies make decisions. It is an advanced stage of traditional industry reports. This business intelligence is more scientific and comprehensive, and can analyze a lot of information to help decision makers analyze. And it is also real-time.

From a technical perspective, business intelligence is not a new technology, it is only a comprehensive application of data warehouse, OLAP and data mining technologies.

• The data warehouse is to store the reports in the database, and the technology stack used is SQL and spark;
• Online analysis and processing is the presentation and processing of data. The technology stack used is the back-end technology stack and the front-end technology stack;
• Data mining is to find the relationship between data, the technology stack used is Python (artificial intelligence library).

Implementing a business intelligence system is a complex system engineering. The general implementation steps are:

Requirement analysis: collect the needs of the enterprise, what data you want to draw what conclusions;

Data warehouse modeling: Analyze the needs of the enterprise and establish the logical model and physical model of the enterprise data warehouse;

Data extraction: Extract data from the business system to the data warehouse. The data must be converted and cleaned in the extraction process to meet the needs of analysis;

Establishing business intelligence analysis reports: data visualization

User training and data simulation test: for users to use;

System improvement and perfection: iterative upgrade

This is the feasibility study, requirements analysis, design, coding, testing, integration and maintenance of software engineering . It feels so interesting. The things taught in the university are really like internal skills, panacea, which can be used anywhere.

powerBI

Power BI is Microsoft's latest business intelligence (BI) concept, which contains a series of components and tools. Can excel and Power BI, easy to use. The operation is basically drag and drop, but its exploratory analysis ability is limited, but the function is simple.

PowerBI tutorial collection in powerBI can see what can be done.

After reading the analysis of Power BI financial statements , I have a general understanding of what powerBI can do. Simply put, PowerBI can be used to capture data, store data, build data models, visualize (build reports, calculate indicators, and visualize), and generate reports.

Data analysis can be cut through powerBI. Do business analysis.

Looking at the rankings, it seems that fineBI is more popular than powerBI. You can try.

PaddlePaddle

Baidu's artificial intelligence framework. Can be used for data mining.

Related vocabulary: Python, Numpy, number recognition, computer vision (image classification, target detection), natural language processing NLP (automatically generated chapter list), personalized recommendation (push ads, push movies, Douyin).

You can cut into artificial intelligence by using PaddlePaddle. Do business intelligence.

Supplementary article

The difference between data analysis and business intelligence?

What I think of data analysis is statistics;

And business intelligence is not only statistics, but also artificial intelligence for data mining to find out the correlation between irrelevant data. Provide more information for decision makers.

User portrait modeling : methods and tools-zhihu

My final understanding

The better side is to understand the difference between business intelligence and business analysis.

Business intelligence can find information outside of statistics and use artificial intelligence to find the correlation between data.

But generally speaking, business analysis has done business statistical analysis. Artificial intelligence not only counts, but also finds correlation.

Use FineBI to cut into business analysis; Use PaddlePaddle to cut into business intelligence;

I saw a blogger Leo.yuan , which is related to business intelligence, which is not bad. Check it out later.

Listening to my roommate chatting with my girlfriend, and I didn't chat with my girlfriend, so boring. Too lazy to conclude.

Update address: GitHub

For more content, please pay attention to: CSDN , GitHub , Nuggets