Big Data challenges and opportunities, the past few years, big data expected expansion phase, the hype stage into the development stage of rational development in the coming years, landing stage applications, big data in the coming years will gradually into the rational development period. The next big data development still exist many challenges, but the outlook remains very positive.
Big Data challenges of development
Currently the development of big data are still many challenges, including seven aspects: there is no clear division of large data requirements result in a gradual loss of data assets; severe internal data silos, resulting in data value can not be fully tapped; low data availability, data quality poor, resulting in data can not be used; data related to the management of technology and architecture behind, leading to not have large data processing capabilities; poor data security capabilities and awareness, resulting in data loss; big data talent deficiency leads to big data work is not carried out; the more open big data more valuable, but the lack of policies and regulations related to big data, data makes it difficult to balance between openness and privacy, it is difficult to open better.
40 + annual salary of big data development [W] tutorial, all here!
Challenge one: there is no clear division of large data requirements
Many companies do not understand the big data business, do not understand the application scenario and the value of big data, it is difficult to raise large demand for accurate data. Due to business demand is not clear, big data sector and non-profit sector, business decision-makers are more worried about the cost of investment, led to many companies hesitant to build large data sector, many companies are in a wait or try attitude, from fundamentally affect the development of enterprises in the direction of big data, but also hindered the accumulation and tap their own enterprise data assets, even as data is not application scenarios, deleted a lot of valuable historical data, resulting in the loss of corporate data assets. Therefore, this aspect requires big data experts and practitioners together to promote and share large data scenarios, so that more business people understand the value of big data.
Challenge: enterprise data silos serious
The most important challenge for companies to start big data is fragmented data. In many enterprises, especially large enterprises, data is often scattered in different departments, and these data have different data warehouse, data technology in different sectors may also be different, which leads to its own internal data could not get through law. If you do not get through these data, the value of big data mining is very difficult. Big Data and the associated need to integrate different data play better understand customers and understand the business advantages. How to get through data from different departments, and implementation techniques and tools to share, in order to better play the enterprise value of big data.
Three challenges: data availability is low, poor data quality
Many medium-sized and large enterprises, all the time are also produced in large amounts of data, but many companies pay attention to the very large data pre-processing stage, resulting in data processing very irregular. Large data need to extract data preprocessing stage to facilitate the processing of the data into the data types, data and denoising washing, to extract the valid data and other operations. Even many companies in many cases are not reported data specification unreasonable appeared. For these reasons, resulting in the availability of business data is poor, poor data quality, data is not accurate. The significance of Big Data is not just to collect large-scale data, and the data collected be well pre-treatment, will it be possible to make data analysis and data mining personnel to extract data from large high availability has the value of information. Sybase data show that high-quality data data applications can significantly enhance the company's business performance, increase data availability by 10%, improve the performance of enterprises at least 10%.
Challenge IV: data management technology and related infrastructure
Technical architecture challenges include the following aspects: (1) traditional database deployments can not handle TB-level data, the fast-growing volume of data management capabilities beyond traditional database. How to build a distributed data warehouse, and can facilitate the expansion of a large number of servers is a challenge many traditional enterprises; (2) a lot of companies using the traditional database technology, did not consider the diversity of data types at the beginning of the design, especially for construction compatible data, semi-structured and unstructured data; (3) traditional enterprise database, data processing time is less demanding, the results of these statistical data often lag a day or two days to count out. But big data requires real-time processing data, the order of minutes or even seconds-level computing. Traditional database architects lack the ability to real-time data processing; (4) the mass of data requires good network infrastructure, need powerful data centers to support, operation and maintenance of the data center will also be a challenge. How to ensure data stability, support high concurrency while reducing server load is low, became a focus of massive data center operation and maintenance.
Challenge Five: data security
Networking makes life easier for criminals to obtain information about people, also have more easily be tracked and prevention of criminal means, there may be even more brilliant scam. How to ensure that users of information security has become a very important issue big data era. More and more data online, hackers are motivated crimes to the intense than ever, some well-known sites password leak, resulting in vulnerabilities user data theft and other sensitive personal information leakage events have alert us to strengthen the security of a large data network construction. In addition, increasing big data, is increasing the physical security requirements of data storage, so that multiple copies of data and disaster recovery mechanism also put forward higher requirements. Currently many traditional enterprise data security concern.
Challenge Six: Big Data shortage of talent
Every aspect of the construction of big data will need to rely on professionals to complete, therefore, must cultivate and foster a master big data technology, understanding of management, large data application experience building big data professional teams. At present large data relating to the lack of talent will hinder the big data market. According to Gartner predicts that by 2015, the world will add 4.4 million related to big data jobs, and there will be 25% of the organizations established chief data officer positions. Related posts big data requires that compound talents, able to integrate control of the multifaceted knowledge of mathematics, statistics, data analysis, machine learning and natural language processing. The future, there will be big data talent gap of about 1 million, in large data various industries top talent will become one of the hottest talents, covering data development engineer big data, big data analysts, data architects, large data background develop multiple directions engineers, algorithm engineers. Universities and companies therefore need to work together to training and mining. The biggest problem is the lack of large data many colleges and universities, therefore, have big data companies should joint training with the school.
Challenge Seven: Openness and privacy of data weigh
In the big data applications increasingly important today, open sharing of data resources has become critical to maintain our competitive edge in the data war. Share business data applications and personal data, not only promote the development of related industries, but also give our lives to bring great convenience. Due to government, business and industry information system construction is often a lack of unified planning, lack of uniform standards between systems, the formation of a large number of "islands of information", and administrative monopoly and business interests are limited by the low level of open data, which gives data use caused great obstacles. Another important factor restricting China's opening up and sharing of data resources policies and regulations are not perfect, big data mining the lack of appropriate legislation. Not only ensure shared and prevent abuse. Therefore, the establishment of a sound development of eco-system data sharing, is the development of China's big data requires a step past a cut. At the same time, how to balance openness and privacy, the biggest problem is the large data open face in the process. How to promote the full liberalization of data, applications and shared while effectively protecting citizens, private enterprises, and gradually strengthen privacy legislation, it will be a major challenge of big data era.
61. Big Data predictive
62. Big Data other applications
63. Large data which industries can be applied in
64. In the financial industry, large application data
65. Big Data applications in the Internet industry
66. The application of big data in the logistics industry