Chapter Summary

Spark is a fast and general-purpose computing engine designed for large-scale data processing. It is developed and implemented by the Scala language. Regarding big data technology, it is computing data itself, and Scala has both object-oriented organization project engineering capabilities and The function of calculating data, and the tight integration of Spark and Scala, this book will use Scala language to develop Spark programs, so learning Scala well will help us better grasp the Spark framework.

1.1 Getting to know Scala first

1.1.1 Overview of Scala

Scala was developed in 2001 by the Programming Methods Laboratory of the Federal Institute of Technology in Lausanne (EPFL), and it was created by Martin Odersky (Martin Odersky). Currently, many companies that rely on Java for mission-critical business applications are moving or are moving to Scala to improve their development efficiency, application scalability, and overall reliability.

Scala is the abbreviation of Scalable Language. It is a multi-paradigm programming language. Its original design intention is to realize a scalable language and integrate various features of object-oriented programming and functional programming. details as follows:

Scala is object-oriented
Scala is a pure object-oriented language where every value is an object. The data types and behavior of objects are described by class characteristics.
Scala is functional programming
Scala is also a functional language where functions are available as values.
Scala is statically typed.
Scala has a type system that ensures code security and consistency through compile-time checks.
Scala is extensible
Scala provides many unique language mechanisms, and it can easily and seamlessly add new language structures in the form of libraries.
Scala is interoperable
Scala is designed to interoperate well with the popular Java. Scala uses the Scala compiler to compile source files into Java class files. We can call Java class libraries from it, and we can also call Scala code from Java applications.

1.1.2 Download and install Scala

Scala language can run on Windows, Linux, Mac OS and other systems. Since Scala runs on the JVM platform, the JDK environment must be configured before installing Scala (the JDK version must not be lower than 1.5). The JDK version used in this book is jdk1.8.

Install Scala on Windows

Download the Scala installation package.
Visit the Scala official website The Scala Programming Language , click [Download] to enter the download page, where you can download the latest Scala version. Considering the stability of Scala and the compatibility of Spark, we choose to download Scala2.11.8 (download address: Scala 2.11.8 | The Scala Programming Language ), as shown in the following figure:
Unzip the Scala installation package and configure the environment variables under the windows system
After the download is successful, unzip the Scala installation package scala-2.11.8.zip to the location you need, and configure the environment variables under the Windows system, click [This computer] > [ Properties] > [Advanced System Settings] > [Environment Variables], click [New] in the system variables, the variable value is the location where you decompressed, as shown in the figure below:

Add the bin directory of Scala to the path, as shown in the figure below:
Enter the Windows command line and enter the " scala" command to test whether the Scala environment is successfully installed. As shown below:

Install Scala under Linux

Download the Scala installation package and upload the installation package to the Linux system/export/software
Execute tar -zxvf scala-2.11.8.tgz -C /export/servers/the command to decompress and install
Add Scala environment variables under Linux system

Install Scala on Mac

Download the Scala installation package and unzip it to the main directory (if you can’t find the main directory, go back to the desktop and press ⇧⌘H to enter the computer’s main directory).
Execute " open ~/.bash_profile" to add Scala environment variables.
Enter the " scala -version" command to check the installation status of Scala. If the Scala version number is displayed on the terminal, it means that Scala is installed successfully.

1.1.3 Download and install the Scala plug-in in the IDEA development tool

At present, there are mainly two mainstream development tools for Scala: Eclipse tools and IDEA tools. We can install the corresponding Scala plug-ins in these two development tools for Scala development.

Since the IDEA tool can automatically identify code errors and perform simple repairs, and the IDEA tool has many excellent built-in plug-ins, most Scala development programmers now choose IDEA as a tool for developing Scala.

This book will take the Windows operating system as an example to explain step by step how to download and install the Scala plug-in on the IDEA tool. The specific steps are as follows.

Download the IDEA installation package, open the installation package and click the [Next] button to install until the "Welcome to IntelliJ IDEA" interface appears, then the installation ends.
Visit " https://plugins.jetbrains.com/plugin/1347-scala " to download the Scala plugin.
Click [Configure] → [Plugins] → [Install plugin from disk] → [OK] → [OK] in the lower right corner of the IDEA main interface, click the [Restart] button to restart the IDEA tool, and the installation is complete.

Note: For the better use of the computer, do not install all the software on the C drive, it can be installed on other drive letters, the reader chooses by himself

1.1.4 Developing the first Scala program

Create a project. In the main interface of the IDEA tool, click [create New Project] to create a project, as shown in the following figure:
Create packages and Scala classes.

Create a package
Select the "src" folder, right click and select [New]→[Package] button, enter the package name, the effect is shown in the figure
Create a class
Select the package name, right-click [New]→[Scala Class], and select [Object] type to create a Scala class, the class name is "HelloWorld", the effect is shown in the following figure:

Write the code in the HelloWorld.scala file, the code is as follows.

object HelloWorld {
    def main(args: Array[String]) {
        println("Hello, world!")
    }
}

Run the code, and the console output is shown in the figure below.

Note: Different versions of IntelliJ IDEA have slightly different interfaces, and readers need to determine by themselves according to the version.

Reprinted from: https://blog.csdn.net/u014727709/article/details/132032236
welcome to start, welcome to comment, welcome to correct

Spark Big Data Analysis and Practical Notes (Chapter 1 Scala Language Basics-1)

Article directory