Transformation of big data for the crowd
1、Java
Java programming is the basis for the development of Big Data, Big Data technology, many are written in Java, such as Hadoop, Spark, mapreduce, etc. Therefore, you want to learn big data, is an essential Java programming skills!
2、Python
Python is often supported in large data processing framework.
3, Linux operation and maintenance
Large enterprise data development is often done in the Linux operating system, therefore, I want to engage in large-data-related work, need to have Linux operating systems, and command.
In this case still have to recommend my own build Big Data learning exchange group: 529 867 072, the group is big data science development, big data if you are learning, you are welcome to join small series, we are all party software development, from time to time Share dry (only the big data-related software development), including a copy of the latest big data and advanced data advanced development course my own sort of welcome advanced and want to delve into the big data small partners to join.
4、Zookeeper
ZooKeeper is a distributed, open-source coordination service for distributed applications, is an open source implementation of Google's Chubby, is a key component of Hadoop and Hbase. It is to provide a consistent service for distributed applications, provides features include: configuration maintenance, domain name service, distributed synchronization, group services.
5、R
R language has become the darling of the scientific data.
6、Hadoop和Hive
Hadoop slower than some other processing tools, but it is surprisingly accurate, it is widely used in back-end analysis. It Hive-- and a frame-based query may be run on top of the work well junction.
7.
Kafka
Kafka is a high throughput of distributed publish-subscribe messaging system that can handle all the action streaming data scale consumer website by Hadoop parallel loading mechanism to unify online and offline messaging, provided by Cluster real-time messaging.
8.
Spark
Spark is designed for large-scale data processing designed for fast general-purpose computing engine that has Hadoop MapReduce has advantages, but is different from the MapReduce Job intermediate output can be stored in memory, eliminating the need to read and write the HDFS, so Spark better suited for data mining and machine learning requires iterative MapReduce algorithm. Big Data learning exchange group: 529 867 072
Technically, in fact, just a little big data base programmers transformation has advanced natural advantage, even if you did not learn more than one language is also possible to use Hadoop.
Third, the big data development direction of employment
Big data as a basis for science, both in data development and analysis, networking and training areas of artificial intelligence algorithms, have core technology and job demands, then what are their counterparts in the development direction of the jobs it?
Direction ❶: Big Data Engineer, Development Engineer Big Data, Big Data maintenance engineer, Big Data R & D engineers, architects and other large data;
Direction ❷: big data analysts, senior big data, big data expert analysts, big data mining division, division, and other large data algorithm;
Direction ❸: big data operation and maintenance engineers.
This age of big data development impact is relatively small, because the data is not simply engage in large programming, programming the weight of even 1/6 not to, you need a lot of time and other aspects from the server, storage, computing, operation and maintenance to analyze the problem Solve the problem.
Survival time in the Big Data industry longer, their experience will be more to be sure, this is the most senior IT analysts will bring big data or reason "old" 50, 60-year-old expert.
Faced with such a bright and attractive prospect, visionary, he had to make their own arrangements posterior.