Spark is originated from the University of California, Berkeley AMPLab cluster computing platform, which is based on in-memory computing, Hadoop performance over a hundred times, from multiple iterative batch processing, data warehousing eclectic, stream processing and computing and other computing paradigm, is rare the all-rounder. Spark adopt a unified technology stack to resolve all the core issues, such as stream processing, drawing techniques, machine learning, NoSQL Query cloud computing big data, with a sound ecological system, which directly lays its unified cloud big data field the dominance.
Spark along with the popularization of technology, the demand for professionals is growing. Spark professionals is also sought in the future, you can easily get a million salary. And in order to become a master Spark also need one by one, from practicing internal strength: Generally need to go through the following stages:
50W annual salary of Java programmers to turn Big Data learning route poke me read
Artificial intelligence, big data trends and prospects poke me read
The latest and most complete big data exchange system path! ! Poke me read
2019 latest! Big Data Engineer jobs salary, it was amazing ! I read poke
First stage: a skilled master Scala language
Spark framework is the use of Scala language, refined and elegant. Spark To become a master, you have to read the Spark source code, you must master Scala ,;
Although the Spark can now multi-language Java, Python and other application development, but the fastest and best development API support is still and always will be the way of the Scala API, so you have to master complex and high Scala to write Spark performance distributed applications;
In particular, the master Scala trait, apply, function programming, generic, and inverter covariant and the like;
Phase II: Spark proficient platform itself available to the developer API
RDD master for the development model Spark track the action and various functions using the transformation;
Wide and narrow control dependency and lineage-dependent mechanism of Spark;
RDD master the calculation process, such as Stage division, Spark application submitted to the basic process of cluster and node-based works Worker, etc.
Phase III: Spark-depth core
This stage is the core part of the Spark to in-depth study by Spark source framework:
Spark source code through the submission process control tasks;
Spark source code control task scheduling by the cluster;
Especially proficient DAGScheduler, details of the work of the internal TaskScheduler and Worker nodes each step;
Fourth Estate: grasp the core framework based on the use Spark
Spark as a master of the big data era of cloud computing, real-time stream processing, drawing techniques, machine learning, NoSQL query and so has a significant advantage, when we use the Spark in use most of the time frame on which such as Shark, Spark Streaming and so on:
Spark Streaming is a very good real-time stream processing framework, to master its DStream, transformation and checkpoint etc;
Spark off-line statistical analysis, Spark 1.0.0 version launched on the basis of Shark Spark SQL, off-line statistical analysis of functional efficiency has significantly improved, it is important to grasp;
For the Spark such as machine learning and GraphX to master its principles and usage;
Fifth class: do business level Spark project
Spark project through a complete representative to run through all aspects of Spark, including architectural design of the project, the analysis used technology to achieve development, operation and maintenance, a complete grasp at every stage and the details, so you can make You'll be able to calmly face the vast majority of Spark project.
Sixth class: Provides Spark Solutions
Thorough grasp every detail Spark framework source;
Spark provide solutions under different scenarios according to the needs of different business scenarios;
According to actual needs, secondary development framework on the basis of Spark, Spark build their own framework;
The previously described six stages Spark becomes the master in the first and second stages can step through learning, a subsequent stage is preferably carried out by three expert or expert guidance next step, a final stage, the basic that is, to the "no stroke win a trick," the period, much to comprehend the intentions to complete.