The era of big data is irresistible, there should be no doubt, but for the vast majority of enterprises, big data itself is only an empty concept, which is not only difficult to participate in but also difficult to control. Enterprises of almost any size are generating a large amount of data all the time, but how to collect and refine this data is always a problem. The significance of big data technology is not to master huge data information, but to intelligently process these data, analyze and mine valuable information from it, and realize the transformation of data value. The development of mobile internet has spawned more diverse data, including both structured and unstructured data. The most common types of data, such as ordinary text, photos, videos, location information, and link information, are difficult to use by traditional technical means. Analyze it for refinement.
1. Overall architecture of big data
2. Necessary ability of big data
The big data technology system is relatively large, and the basic technologies cover data collection, data preprocessing, distributed storage, NOSQL database, multimodal computing, multimodal computing, data warehouse, data mining, machine learning, artificial intelligence, deep learning, parallel computing , visualization and other technical fields.
In general, Hadoop big data is roughly divided into three major directions and ten positions, three major directions: big data system research, big data application development and big data analysis, ten positions: ETL R&D, Hadoop development, visualization (Front-end display) tool development, information architecture development, data warehouse research, OLAP development, data science research, data prediction (data mining) analysis, enterprise data management, data security research.
