What should a complete enterprise-level data analysis system look like? Some people say: It must have a big data team, and then add big data platform, big data analysis, big data mining technology, that's it.
The big data analysis platform includes: a bunch of servers, data collection and processing, distributed storage, computing engines, data warehouses, which are also divided into two technical lines, offline and real-time.
Big data analysis includes: index system, dimensions, fixed reports, multi-dimensional analysis, ad hoc query, thematic analysis, etc.
Big data mining includes various models and applications such as tagging systems, algorithm development platforms, recommendation systems, and early warning predictions.
The big data team includes big data engineers, algorithm engineers, BI engineers, business data analysts and other roles.
When the layman heard it, wow, Niu X! Tall, professional!
As soon as the boss heard this, his head became bigger. How much money and how long does it take to build it?
As soon as the data practitioner heard, Alexander, he still had to know so many things that he couldn't do this job.
Is the above answer correct? Correct, but it didn't work.
A company is like a car, the management is the driver, and each department is the system of the car. The process of the company's operation is to reach the destination smoothly and safely.
Among them, the data analysis system should assume the responsibilities of the perception system, alarm system, and navigation system.
Then the best practice of an enterprise-level data analysis system should be like this:
Internal perception system: timely notification of various internal operating parameters;
External perception system: timely feedback of external competition and opportunity status;
Alarm system: discover and report various abnormalities in time, and indicate the point of failure;
Navigation system: According to the set goal, provide several reachable plans, according to the current plan and location, indicate the next course of action, make reasonable predictions on the goal, and feedback the current implementation in real time.
If we decompose the entire system, we can understand that "big data" is actually only a tool level, which improves the efficiency of data analysis and mining in a high-concurrency, high-availability, and high-reliability environment. Therefore, the selection of tools, technologies, and personnel are all determined by the well-organized process and the well-structured data analysis system. What kind of architecture is required is determined by the type of enterprise, the stage it is in, and short-term, medium- and long-term goals. It cannot be generalized. No matter what, it will be solved on a big data platform, algorithm, machine learning, and cloud platform.
Landing to the actual situation of each enterprise requires a landing solution, which can be a simple temporary solution or a systematic solution; it can also be solved by one or two people, or it can be undertaken by a large team.
Talking about the data analysis system out of the actual situation is to be hooligan and bully the layman. Blindly pursuing loftyness, the final result is bound to be solitary, high and widowed, and do not solve practical problems.
Therefore, according to the actual situation of the enterprise, it is the best practice of the data analysis system to establish an organization and technology that meets current needs, appropriately advanced planning, and has internal and external perception, alarm, and navigation functions.
Next time I will share how to build an enterprise-level data analysis system.