The evolution of computing technology:
1. Offline computing Mapreduce: In the early stage of big data interest, it meets the offline batch processing requirements of massive data;
2. Stream computing Storm: meet the real-time statistical requirements of e-commerce, news aggregation, etc., real-time supervision, etc., and trigger calculations driven by data streams, with high timeliness, generally reaching the second level
3. Real-time computing Spark: It is an iterative algorithm for machine learning/pattern recognition in deep mining of massive data. The results of each calculation are distributed in the memory, and the data of the previous round is directly read from the memory in the next round. Save a lot of IO overhead
4. Graph computing: Based on the potential correlation analysis between data, better real-time prediction and recommendation, the graph can find and calculate the correlation more directly through the connection of vertices and edges
Big data mining technology:
1. Deep learning: improve the accuracy of classification or prediction through hierarchical methods and massive training data
2. Multi-dimensional data association: a data model that satisfies users for fast data query and analysis from multiple perspectives and levels, and is oriented to analysis and decision-making, and solves the problem that traditional data models cannot effectively represent data structure and semantics when the data dimension is high and the number of entries is large. , and the inability to effectively support OLAP. Main analysis methods: drilling, rolling, dicing, rotating
3. Knowledge graph: a technology that depicts the relationship between entities based on the graph structure.
4. Data visualization