At present, there are still many challenges in the development of big data, including seven challenges:
1) Business departments do not have clear big data requirements, resulting in the gradual loss of data assets
2) There are serious data silos within the enterprise, resulting in insufficient data value mining;
3) Low data availability and poor data quality, resulting in unavailability of data;
4) The data-related management technology and architecture are backward, resulting in the lack of big data processing capabilities;
5) Poor data security capabilities and awareness of prevention, resulting in data leakage;
6) The lack of big data talents makes it difficult to carry out big data work;
7) The more open big data is, the more valuable it is, but the lack of big data related policies and regulations makes it difficult to balance data openness and privacy, and it is difficult to open it better.
Challenge 1: Business departments do not have clear big data requirements
Many business departments of enterprises do not understand big data, nor do they understand the application scenarios and value of big data, so it is difficult to put forward the exact requirements for big data. Due to the unclear needs of the business department and the non-profit department of the big data department, the decision-makers of the enterprise are worried about the high cost of investment, which has led to many companies hesitant to build a big data department, or many companies are in a wait-and-see attitude. It fundamentally affects the development of enterprises in the direction of big data, and also hinders enterprises from accumulating and mining their own data assets. Even because the data has no application scenarios, many valuable historical data are deleted, resulting in the loss of enterprise data assets. Therefore, in this aspect, big data practitioners and experts are needed to promote and share big data application scenarios, so that more business personnel can understand the value of big data.
Challenge 2: Serious data silos within the enterprise
The most important challenge for enterprises to start big data is the fragmentation of data. In many enterprises, especially large enterprises, data is often scattered in different departments, and these data are stored in different data warehouses, and the data technology of different departments may also be different, which leads to the inability to get through their own data within the enterprise. If these data are not connected, the value of big data will be very difficult to mine. Big data requires the association and integration of different data in order to better play the advantages of understanding customers and understanding business. How to open up the data of different departments and realize the sharing of technology and tools can better play the value of enterprise big data.
Challenge 3: Low data availability and poor data quality
Many medium-sized and large enterprises are also generating a large amount of data all the time, but many enterprises do not pay much attention to the preprocessing stage of big data, resulting in irregular data processing. In the big data preprocessing stage, it is necessary to extract data, convert the data into a data type that is convenient for processing, clean and denoise the data to extract effective data and other operations. Even many companies have a lot of irregularities and unreasonable data reporting. The above reasons lead to poor data availability, poor data quality and inaccurate data. The significance of big data is not only to collect large-scale data information, but also to perform good preprocessing on the collected data, so that it is possible for data analysis and data mining personnel to extract useful data from high-availability big data. value information. Data from Sybase shows that the data application of high-quality data can significantly improve the business performance of an enterprise. The data availability can be increased by 10%, and the performance of the enterprise can be improved by at least 10%.
Challenge 4: Data-related management technology and architecture
The challenges of technical architecture include the following aspects: (1) Traditional database deployment cannot handle TB-level data, and the rapidly growing data volume exceeds the management capability of traditional databases. How to build a distributed data warehouse and easily expand a large number of servers has become a challenge for many traditional enterprises; (2) Many enterprises adopt traditional database technology, and do not consider the diversity of data categories at the beginning of the design, especially for the structure (3) The database of traditional enterprises does not require high data processing time, and the statistical results of these data are often delayed by one or two days. But big data needs to process data in real time, and perform minute-level or even second-level calculations. Traditional database architects lack the ability to process real-time data; (4) Massive data requires a good network architecture and a powerful data center to support, and the operation and maintenance of the data center will also become a challenge. How to reduce the low load of the server while ensuring data stability and supporting high concurrency has become a key task in the operation and maintenance of massive data centers.
Challenge Five: Data Security
Online life makes it easier for criminals to obtain information about people, and there are more criminal methods that are not easy to track and prevent, and more sophisticated scams may appear. How to ensure user information security has become a very important topic in the era of big data. With more and more online data, hackers are more motivated to commit crimes than ever before. The leakage of sensitive personal information such as password leakage of some well-known websites and the theft of user data caused by system loopholes has alerted us to strengthen the construction of big data network security. In addition, with the continuous increase of big data, the physical security requirements for data storage will become higher and higher, which will also put forward higher requirements for multiple copies of data and disaster recovery mechanisms. At present, the data security of many traditional enterprises is worrying.
Challenge 6: Lack of big data talents
Each link of big data construction needs to be completed by professionals. Therefore, it is necessary to cultivate and cultivate a big data construction professional team that masters big data technology, understands management, and has experience in big data application. The current shortage of big data related talents will hinder the development of the big data market. According to Gartner, by 2015, there will be 4.4 million new jobs related to big data worldwide, and 25% of organizations will have a chief data officer position. Jobs related to big data require compound talents who can comprehensively control various aspects of knowledge such as mathematics, statistics, data analysis, machine learning and natural language processing. In the future, there will be a talent gap of about 1 million in big data. High-end talents in big data in various industries will become the hottest talents, covering big data data development engineers, big data analysts, data architects, and big data backends. Development engineer, algorithm engineer, etc. Therefore, it is necessary for universities and enterprises to work together to cultivate and excavate. The biggest problem at present is that many colleges and universities lack big data, so companies with big data should jointly train talents with schools.
Challenge 7: Trade-off between data openness and privacy
In today's increasingly important application of big data, the open sharing of data resources has become the key to maintaining an advantage in the data war. The shared application of commercial data and personal data can not only promote the development of related industries, but also bring great convenience to our lives. Due to the lack of unified planning for the construction of government, enterprise and industry information systems, and the lack of unified standards between systems, many "information islands" have been formed, and limited by administrative monopoly and commercial interests, the degree of data openness is low, which gives data to data. Utilization creates great obstacles. Another important factor restricting the opening and sharing of data resources in my country is that the policies and regulations are not perfect, and the big data mining lacks corresponding legislation. There is no way to both guarantee sharing and prevent abuse. Therefore, the establishment of a healthy data sharing ecosystem is the first step in the development of big data in my country. At the same time, how to balance openness and privacy is also the biggest problem faced in the process of big data opening. How to effectively protect the privacy of citizens and enterprises while promoting the comprehensive opening, application and sharing of data, and gradually strengthen privacy legislation, will be a major challenge in the era of big data.