Spark in the Win10 environment to build

This tutorial is completed with reference to another blog, middle solve some other problems, I'll just copy the same part of the reference Bowen, most of the text in accordance with the map is not my version, need not correspond to the version number.

Reference blog: https://blog.csdn.net/songhaifengshuaige/article/details/79480491

Introduction
This chapter explains how to set up the environment to achieve spark under WIN10.

Chapter Summary
1. Release Notes
2. Preparing the environment:
the JDK configuration;
Scala installation and configuration;
the Spark installation and configuration;
hadoop installation and configuration;

Imprint
the JDK: 1.8
Scala: 2.11.8
the Spark: 2.3.0
hadoop: 2.8.3

Environmental ready
jdk configuration
1, configuration JAVA_HOME and Path:

Path


2, verify the configuration:


scala Installation and Configuration
1, scala Download:
Visit the official address http://www.scala-lang.org/download/2.11.8.html


2, the configuration Path:


3, verification configuration:


Spark installation and configuration
1, spark Downloads:
Visit the official address http://spark.apache.org/downloads.html

Select the address to download the official recommendation, other addresses can also be used (recommended Thunder download tools download speed is much faster)


2, to extract the disk D;


3, Configuration Path:


4、通过spark-shell进入Spark的交互式命令行模式:

如上,可以看到对应的spark、scala、java版本,同时也看到了异常信息,该异常信息是由于hadoop导致的,下面来配置hadoop。


Hadoop安装与配置
1、hadoop下载
访问官方http://hadoop.apache.org/releases.html

进入下载页,如下建议地址(同样建议用迅雷等下载工具下载,速度更快)


2、解压至


3、配置HADOOP_HOME&Path:

path:


4、winutils下载:https://github.com/cdarlint/winutils

下载对应版本的bin目录直接替换本地bin目录即可。

5、此时继续spark-shell进入spark命令行交互模式:

此时即不会出现上述的异常提示。

6、访问控制台中的webui如下:

发布了69 篇原创文章 · 获赞 28 · 访问量 3万+

Guess you like

Origin blog.csdn.net/qq_24852439/article/details/103064668
Recommended