In 2008, Victor Meyer - written in Schoenberg "big data era" opened a new chapter in science and technology, so that "big data" word has become a household name, everyone is talking about big data, each technology companies have also introduced big data technologies. So, in the end what is big data?
Large data (Big Data), refers to the collection of data can not be captured, managed and treated with conventional software tools within a certain time frame, is the need for new treatment model to get more decision-making power, insight and process optimization capabilities massive, high growth rates and diverse information assets.
In the "era of big data," in Victor proposed a shortcut without random analysis (sampling), but the analysis of all data processing big data analytics new concept. After, IBM also made a 5V characteristics of big data:
Volume (a lot)
Velocity (high-speed)
Variety (diverse)
Value (cheap)
Veracity (authenticity)
So perhaps somewhat abstract, for example, every day we browse the Web, online shopping, take-away point, brush vibrato, chase drama network, see the headlines, all your finger across the place, would leave the data. These data chaotic complex, but implies your behavior, propensity to buy, the spending power of information and so on. How to find out the hidden association, dig out useful information to form a precise picture of the business users, it is very critical step, on this basis, derived algorithm technology, we are often called Big Data technologies, including data collection , analysis, mining, derivative, and so on.
Suddenly, big data technology has gone through 10 years, the heat continues unabated, and is now with the powerful combination of artificial intelligence, setting off a new wave of fourth industrial revolution. After the big data technology has matured, we come back to look, in the end what is big data?
Big Data is a data-centric industries, is around a large data lifecycle continue the cycle of the production process, but also a complex division of labor and the high degree of collaboration with industry resulting from a variety of industries.
Production process large data from the data transfer and the evolution of the life cycle can be divided into the following sections: data collection, data storage, data modeling, data analysis, data cash.
大数据有多重要？吴军在《智能时代：大数据与智能革命重新定义未来》一书中提到：“在这个大数据时代，谁从冗杂的大数据中摸清了蕴藏其下的规律，谁就掌握了财富。”这是对大数据价值的高度概括，也是对未来大数据应用的极目前瞻。
随着对大数据技术的了解和应用，我们开始通过各种软件收集数据，通过网络进行传输，通过云数据中心进行存储，通过数据科学家或者行业专家进行建模和加工，最后通过数据分析得到某种知识或者结论，获得了一种通过数据洞悉世界的能力。
于是，原本错综复杂的数据之间的潜在关系渐渐清晰，大量孤立、多源数据交织融合后显得更为有趣，大量看似无关的事情在经过分析后呈现出更多的因果，这些因果联系能够让我们在更多方面推测未来趋势，减少试错机会，降低成本和风险，从而提高劳动生产力。这是大数据技术带给我们最根本的价值和意义。
已经为大家精心准备了大数据的系统学习资料，从Linux-Hadoop-spark-......，需要的小伙伴可以点击进入
我们经常会发现，在网上购物时，电商推荐的商品往往是我们非常感兴趣的，仿佛电商比我们自己更了解自己。浏览新闻时，首页内容也往往会是自己比较倾向于阅读的，每个人似乎都拥有了为自己量身打造的头条。出现这些“神奇”现象的原因，都是这些App记住了我们的浏览历史，它们根据这些信息了解我们的偏好，为我们推荐最匹配的内容。
了解了大数据技术的原理，我们再来看看成为一名大数据工程师究竟需要哪些技能。一般而言，大数据技术包括三个方向，大数据架构、大数据分析和大数据开发。
Big Data architecture focuses on implementation principle Hadoop, Spark, Storm and other large data framework, deployment, tuning and stability problems, and how they relate Flume, Kafka and other traffic tools and visualization tools combined skills, then there is a number of tools commercial application problems, such as Hive, Cassandra, HBase, PrestoDB and so on. Carried out by a technical standpoint dialectical combination, to maximize software / hardware resource utilization, providing stable service, which is the target architecture big data talent. Big Data Architect focuses on content architecture theory, data streaming applications, storage applications, software applications and visualization applications.
Big Data analysis is concerned that the establishment of data indicators, the link between the statistics, data and depth data mining and machine learning, and get more use patterns law exploratory data analysis, knowledge, or to gain future the ability to predict things and pre-judgment. The main database applications, data processing, data analysis, statistical data and big data analytics.
Large data development concerned the development server, database development, and visual presentation, and other interactive data adapter and the data processing of each carrier unit, the user ultimately function floor. The main big data development database development, data flow tool development, data front-end development, data acquisition development.
PS: If you want to be a big data engineer, now wish to move in these directions now!