Intellij IDEA development environment construction, scala configuration and packaging, jar package running in spark

1. Intellij IDEA development environment construction

Recently I am learning scala. In addition to writing scala programs, I also need to create maven projects and package them into Jar packages. Eclipse does not seem to be used in this respect. At the same time, because Intellij IDEA provides a very intelligent prompt function, I choose IDEA as a development environment. Intellij IDEA now also provides a community version for developers to use for free, so many original Eclipse users are gradually turning to IDEA. On the one hand, intelligent prompts and shortcut keys can improve development efficiency, on the other hand, its clean interface design and rich The plug-in integration function facilitates the integrated development and use of the program. Here is the official website download address https://www.jetbrains.com/idea/download/download-thanks.html?platform=windows&code=IIC, for the convenience of development, you can download the Windows version and install it directly.

The installation process can be continued all the way to the end, don't worry about too many problems on the plug-in, and then start the setting when the project is created. Here are some common settings:

1. When creating a new JAVA project for the first time, specify the directory where the JDK is located
2. Set font size and spacing:
3. Font size and spacing for console:
4. Display line numbers:
5. Display ToolBar and Tool Buttons under [View], and [setting] is in ToolBar:
6. If it cannot be modified, save as first, this is the design concept of Intellij

 

In addition, in addition to the free Community Edition, there is also a cracked Ultimate Enterprise Edition, which is more powerful than Community, so you can use the following methods to crack.

IDEA is very easy to use compared to other compilers, but the enterprise version needs to be activated (students in school can apply for free), and the activation page will pop up when it starts. There are two ways to activate it. I prefer the second one, and the first one occasionally fail...

1. Go to the website http://idea.lanyus.com/ to get the registration code.

2. Fill in the following license server:

　　http://intellij.mandroid.cn/
　　http://idea.imsxm.com/
　　http://idea.iteblog.com/key.php

/***********************update 2017/12/14*************************/

Suddenly found that the three activation servers can not be used, I don't know which God built the server http://idea.java.sx/, available

/***********************update 2018/1/15*************************/

Select the second copy the following code into the text box (available)

[html] view plain copy

1. C0FHYYCJ22-eyJsaWNlbnNlSWQiOiJDMEZIWVlDSjIyIiwibGljZW5zZWVOYW1lIjoid2VpIGh1YW5nIiwiYXNzaWduZWVOYW1lIjoiIiwiYXNzaWduZWVFbWFpbCI6IiIsImxpY2Vuc2VSZXN0cmljdGlvbiI6IiIsImNoZWNrQ29uY3VycmVudFVzZSI6ZmFsc2UsInByb2R1Y3RzIjpbeyJjb2RlIjoiSUkiLCJmYWxsYmFja0RhdGUiOiIyMDE3LTAyLTI2IiwicGFpZFVwVG8iOiIyMDE4LTAyLTI1In1dLCJoYXNoIjoiNTA2NDQ3MS8wIiwiZ3JhY2VQZXJpb2REYXlzIjo3LCJhdXRvUHJvbG9uZ2F0ZWQiOmZhbHNlLCJpc0F1dG9Qcm9sb25nYXRlZCI6ZmFsc2V9-pujnwQxf3C8yHf0amTTAId+Oj1LzpvTsn71x0pvgjvXtx7MWs0AAbAX2L5frRvM/0AzmF7aQuayU5EEcew+J5j0O6RsCxMUilbeeQuCBDZK1bm9NKlAYf90RSKqQ9ViMyjsBVTHoABM0XHNi++XASeBB9g1OUZdzseLZ6pit0G0BPmunKIuX7bqjWewl0okbNIl+bW2y3eD4un9jDODbjMmdlB6T42s6mQ/lDkjD65C5OY1nY2fsVyd4sa4i0iCYwNvzFlDVG/AzziLYNDAdjipE+lN5McMXMfPZuGKkvTJMJk56l5t+jPjdjPRDId+lLqa0GU3zaNP251mm8vmOew==-MIIEPjCCAiagAwIBAgIBBTANBgkqhkiG9w0BAQsFADAYMRYwFAYDVQQDDA1KZXRQcm9maWxlIENBMB4XDTE1MTEwMjA4MjE0OFoXDTE4MTEwMTA4MjE0OFowETEPMA0GA1UEAwwGcHJvZDN5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxcQkq+zdxlR2mmRYBPzGbUNdMN6OaXiXzxIWtMEkrJMO/5oUfQJbLLuMSMK0QHFmaI37WShyxZcfRCidwXjot4zmNBKnlyHodDij/78TmVqFl8nOeD5+07B8VEaIu7c3E1N+e1doC6wht4I4+IEmtsPAdoaj5WCQVQbrI8KeT8M9VcBIWX7fD0fhexfg3ZRt0xqwMcXGNp3DdJHiO0rCdU+Itv7EmtnSVq9jBG1usMSFvMowR25mju2JcPFp1+I4ZI+FqgR8gyG8oiNDyNEoAbsR3lOpI7grUYSvkB/xVy/VoklPCK2h0f0GJxFjnye8NT1PAywoyl7RmiAVRE/EKwIDAQABo4GZMIGWMAkGA1UdEwQCMAAwHQYDVR0OBBYEFGEpG9oZGcfLMGNBkY7SgHiMGgTcMEgGA1UdIwRBMD+AFKOetkhnQhI2Qb1t4Lm0oFKLl/GzoRykGjAYMRYwFAYDVQQDDA1KZXRQcm9maWxlIENBggkA0myxg7KDeeEwEwYDVR0lBAwwCgYIKwYBBQUHAwEwCwYDVR0PBAQDAgWgMA0GCSqGSIb3DQEBCwUAA4ICAQC9WZuYgQedSuOc5TOUSrRigMw4/+wuC5EtZBfvdl4HT/8vzMW/oUlIP4YCvA0XKyBaCJ2iX+ZCDKoPfiYXiaSiH+HxAPV6J79vvouxKrWg2XV6ShFtPLP+0gPdGq3x9R3+kJbmAm8w+FOdlWqAfJrLvpzMGNeDU14YGXiZ9bVzmIQbwrBA+c/F4tlK/DV07dsNExihqFoibnqDiVNTGombaU2dDup2gwKdL81ua8EIcGNExHe82kjF4zwfadHk3bQVvbfdAwxcDy4xBjs3L4raPLU3yenSzr/OEur1+jfOxnQSmEcMXKXgrAQ9U55gwjcOFKrgOxEdek/Sk1VfOjvS+nuM4eyEruFMfaZHzoQiuw4IqgGc45ohFH0UUyjYcuFxxDSU9lMCv8qdHKm+wnPRb0l9l5vXsCBDuhAGYD6ss+Ga+aDY6f/qXZuUCEUOH3QUNbbCUlviSz6+GiRnt1kA9N2Qachl+2yBfaqUqr8h7Z2gsx5LcIf5kYNsqJ0GavXTVyWh7PYiKX4bs354ZQLUwwa/cG++2+wNWP+HtBhVxMRNTdVhSm38AknZlD+PTAsWGu9GyLmhti2EnVwGybSD2Dxmhxk3IPCkhKAK+hAGYD6ss+Ga+aDY6f/qXZuUCEUOH3QUNbbCUlviSz6+GiRnt1kA9N2Qachl+2yBfaqUqr8h7Z2gsx5LcIf5kYNsqJ0GavXTVyWh7PYiKX4bs354ZQLUwwa/cG++2+wNWP+HtBhVxMRNTdVhSm38AknZlD+PTAsWGu9GyLmhti2EnVwGybSD2Dxmhxk3IPCkhKAK+hAGYD6ss+Ga+aDY6f/qXZuUCEUOH3QUNbbCUlviSz6+GiRnt1kA9N2Qachl+2yBfaqUqr8h7Z2gsx5LcIf5kYNsqJ0GavXTVyWh7PYiKX4bs354ZQLUwwa/cG++2+wNWP+HtBhVxMRNTdVhSm38AknZlD+PTAsWGu9GyLmhti2EnVwGybSD2Dxmhxk3IPCkhKAK+pl0eWYGZWG3tJ9mZ7SowcXLWDFAk0lRJnKGFMTggrWjV8GYpw5bq23VmIqqDLgkNzuoog==  

2.scala configuration

There is no Scala plugin in the default configuration of Intellij IDEA , which needs to be installed manually. Select configure plugins when Intellij IDEA runs for the first time. After selecting the default, configure it on the main interface of the software. However, the test found that this setting method often fails to configure due to network problems. Go directly to the link https://plugins.jetbrains.com/plugin/1347-scala to download. After the download is complete, unzip it to the plugins in the Intellij IDEA installation directory, and then start Intellij IDEA. Then you can create the creation of your Scala development environment:

1 File->New Project 

Select scala->IDEA, fill in the project name, select the path where the project is saved and the JDK and Scala SDK versions, and then the next step: get the following interface:

Click Finish, then click File-->Project Structure, as shown in the figure below, select the installed jdk directory, and the path of the created project file

On the right side of Project->Project SDK, click new to get: 

Select JDK, and select the installation directory of JDK in the pop-up box 

Click the green "+" sign on the right side of Libraries, select Scala SDK, and the option to select the scala version appears as shown in the following figure:

In the previous step, you can choose the Scala that comes with Intellij IDEA, or you can use the Scala you installed yourself, click "Browse" in the figure above, and then select the Scala installation directory. After the configuration is complete, finish directly and get the following Project directory 

点击File->Project Structure->Modules， 

在src目录上，右键然后点击new fold 命名为main，再右鍵main，同样new fold，命名为scala，并设置为sources，如下图，注意，src、main均不要设置sources，否则后面编译helloworld会报错 XX is already defined as object xx 

整体的工程文件已经创建好了，如下图： 在scala源文件目录上，直接右鍵，new->scala class ，选择Object，然后输入名称:HelloWorld 

输入的代码如下：

1 object HelloWorld  {
2   def main(args: Array[String]) {
3     println("Hello World")
4   }
5 }

点击HelloWorld.scala文件，右键或直接ctr+shift+F10运行程序

 

3.程序打包

File->Project Structure

 

Next step：

选择主类：(点击省略号会自动出现要使用的主类）

 

Next step：

点击应用、OK

 

 

Next step：

 

 

 

Next step：

 

Next step：

本地打好的包：

程序运行过程出现的异常及解决方式

编写完scala程序，在运行时出现Error:scalac: Error: org.jetbrains.jps.incremental.scala.remote.ServerException类型的异常。

折腾一两个小时，最后发现安装了多个版本的Scala SDK，前后设置的版本不一致。另外，也要注意scala2.1以上的版本尽量使用JDK1.8。还有，如果要在spark集群环境下运行scala程序是，要注意使用的scala版本和spark的版本的兼容性问题，从Spark2.0版开始，Spark默认使用Scala 2.11构建，Scala 2.10用户应该下载Spark源码包，并用Scala 2.10支持构建 。

Jar包在Spark中运行

将jar包通过XShell上传到服务器的摸个目录下，我这里将HelloScala的jar包放在/home/hadoop/myProject/myJar/目录下，进入spark的安装目录，启动spark(./start-all.sh)，并通过如下的名利运行jar包：

/usr/local/spark/bin/spark-submit --class "HelloScala" /home/hadoop/myProject/myJar/HelloScala.jar 

Local模式运行spark应用程序，通过spark-submit提交应用程序，该命令的格式如下：
/usr/local/spark/bin/spark-submit       //提交Spark任务的入口，其中/usr/local/spark为spark的安装目录，spark-submit为提交Spark 任务的脚本
  --class <main-class>     //需要运行的程序的主类，应用程序的入口点
  --master <master-url>    //Master URL，下面会有具体解释
  --deploy-mode <deploy-mode>      //部署模式
  ... # other options  //其他参数
  <application-jar>  //应用程序JAR包
  [application-arguments] //传递给主类的主方法的参数
The deploy-mode parameter is used to specify the deployment mode of the application. There are two deployment modes: client and cluster. The default is client. When the client deployment mode is used, the Driver Program is directly run locally. When the cluster mode is used, the Driver Program is run on the Worker node. A more common deployment strategy is to submit your application from a gateway machine that collaborates with your worker cluster. In this setting, it is more suitable to use the client mode. In the client mode, the Driver is started directly in the spark-submit process. This process is directly used as the client of the cluster, and the input and output of the application are connected to the console. Therefore, this mode is especially suitable for applications involving REPL. Alternatively, if your application is submitted from a machine that is far away from the worker machine, cluster mode is more appropriate, which reduces network latency between the driver and the executor.

Intellij IDEA common shortcut keys

hot key	Usage description
Ctrl+Shift+A	Find operations by name, e.g. find create scala class operations
Alt + F1	view switch
Ctrl+Tab	Tool window, editing code file switching
Alt + Home	show navbar
Ctrl+J	Insert code templates, such as the main method
Ctrl+Alt+J	Wrap the selected code with a code template
F4	Open the project configuration window
Ctrl+Slash	comment code //
Ctrl+Shift+Slash	comment code /**/
Ctrl+N/Ctrl+Shift+N	Enter a name to query a class or file
Ctrl+D	Copy selected code
Ctrl+W / Ctrl+Shift+W	Smart code selection
Ctrl+F	Search for text content in the current file
Shift+Shift	search everywhere
Ctrl+Shift+F7	Highlight the usage of the first method or symbol in the current file
Ctrl+Space	code auto-suggestion
Ctrl+Shift+Enter	Statement auto-completion
Alt + enter	Automatic code correction

refer to:

         http://www.mamicode.com/info-detail-1192175.html

 http://blog.csdn.net/baolibin528/article/details/50315863   

 http://blog.csdn.net/qq_38663729/article/details/78757039

Scala configures the intellij IDEA15.0.3 environment and hello world!