Overview
Enterprises need to perform conditional analysis to calculate a basic value or multiple basic values of multiple dimensions based on the data collected by the sample, and then use the basic value as a reference to evaluate the situation of each data and conduct business activities.
Uncertainty:
- Low sample data accuracy
- The lack of arms and legs in the sample data is serious
- Diversity of sample data
Since the sample data needs to be filtered and processed under certain conditions, the customer has no direct concept of what kind of processing the sample location has undergone, and has doubts about the accuracy of the data processing
Big amount of data
- The sample data reaches about 50,000, and there are many association conditions involved, which makes data tracking difficult.
- The processed data is about 800,000, and the rules of data cleaning are complicated and difficult to monitor
Multi-algorithm
- The data cleaning algorithm is currently classified in 3 categories, and there may be more classification algorithms in the future
to sum up
- There must be a data intermediate result table. The data processing process can not only be able to analyze but also must be able to monitor the result data during the processing process, because the customer will request the data to be checked at any time
- Performance planning corresponds to the SQL performance requirements for processing large amounts of data, indexes, primary keys, partitions, and views can all be used.
- When the quantity is large, it is best to implement it in steps, so as to echo the data in the process result table.
- When there is a large amount of data, things that can be done in advance can be done in advance, and the data that can be screened and analyzed in advance can be analyzed in advance, which can reduce the consumption of resources at the same time, in fact, it is done in time.