On Windows IDEA build the latest source code debugging development environment Spark2.4.3

I believe that many students want to read through some source framework, to improve their ability to code, but often the first step, when it set up the environment hit a wall.

Under Benpian on to describe how in Windows, the latest version of the compiler will Spark2.4.3, and imported into IDEA compiler. Finally, at the end of IDEA run Spark comes with a total of Example code.

1. Spark ready to build the required source environment

Spark source environment to build something needed to prepare the need to be ready. There's some pit I stepped in, I will also be introduced.

First you need to download maven, maven and modify the source address of the source Ali, I believe that many students have been configured maven, here is the simple mention, maven source address in a foreign country, does not change the source Ali, download it at the time-dependent , that speed makes you wonder of life.

Then download the next scala2.11.8 / scala2.11.12, because there did not seem to spark support 2.12+, so it is necessary scala2.11 +, 2.11.8 or 2.11.12 As should be no impact, but on my own machine is 2.11.8 It is.

Download spark source, but not directly to spark the github repository direct download, currently inside the warehouse is 3 + beta, which should go to the releases.

But I have to help you to find the link, copy the address into the browser will have to download the page.

https://github.com/apache/spark/releases/tag/v2.4.3

Once downloaded decompress, even if the preparatory work done.

2. Compile Spark source code

Directly into the project does not work, because some of the files need to be compiled will produce, but the environment is not difficult to compile, had previously not been configured maven thing. Unzip the downloaded spark2.4.3, into its directory, open cmd, run the following command on the line

mvn -T 4 -DskipTests clean package

Here we must note, pom.xml file under spark2.4.3 directory, there are two areas you need to be modified. The first one is inside the pom.xml Property, which defaults to 3.5.4, need to change yourself maven version, such as my computer is 3.5.2.

Maven version on the computer can pass mvn --version this command.

For example, my computer is 3.5.2, then the corresponding pom.xml inside It changed to 3.5.2.

If not modified, then the compiler will complain of the time is.

The second place is to be modified, modify pom.xml file <scala.version> Default is 2.11.12, need to modify their computer into a scala version of this download time will be selected, not much introduction.

If <scala.version> and you scala version on the computer is inconsistent, then the following error:

Error:scalac: Scala compiler JARs not found

Also to say, spark source file has multiple pom.xml, only need to change the root pom.xml files in a directory on it. After editing, you can start the build.

Yes, compile time will be slower, but compile time will take up more computer resources, it is recommended to compile before going to sleep :)

3. Import

After compilation, you can now import into IDEA inside, directly

open => ${spark2.4.3}_path/pom.xml => Open as Project

After importing First Do not worry, we need to change it maven configuration of IDEA,

File => Setting => Direct search maven => Change Maven home directory

I.e. change place as shown in FIG,

Because there comes IDEA maven, need to change their own installed on the computer, otherwise because you are using your own computer maven compilation, this time with maven IDEA to deal with the case, some will rely reported the following error:

Library Maven: has broken classes path:

In fact, I could not find it.

OK, after the import was successful, wait, there's finally a place needs to be modified.

Menu -> Run -> Edit Configurations -> upper left point of + -> Application

In VM options: inside, plus -Dspark.master = local . This role is to spark started directly in the machine.

Configured to end here, you can run a look at the example.

To examples -> src -> scala -> LogQuery -> Run Right

You will be able to run successfully see the first example spark procedure.

4. Summary

To sum up, it is the first to compile, and then import the final run.

This process seems simple, but in fact, some pit is very easy to give up halfway, I have stepped on some pit, and I hope to help you :)

If any problems occur you can not be solved in this process are also welcome at Benpian message, the time, then I will help us to see the ~

the above!