Spark On YARN 环境搭建

企业开发 2018-05-10 10:09:31 阅读次数: 1
==================================================================================
一、基础环境
==================================================================================
1、服务器分布
192.168.10.84  主名字节点
192.168.10.84  备名字节点
192.168.10.83  数据节点1
192.168.10.85  数据节点2


----------------------------------------------------------------------------------
2、HOSTS 设置
在每台服务器的“/etc/hosts”文件
192.168.10.83  192-168-10-83
192.168.10.84  192-168-10-84
192.168.10.85  192-168-10-85

----------------------------------------------------------------------------------

3、SSH 免密码登录
可参考文章：

http://blog.csdn.net/codepeak/article/details/14447627

==================================================================================
二、Hadoop 2.6.2 编译安装【官方提供的二进制版本native java为32位版本，64位环境需重新编译】
==================================================================================
1、JDK 安装
http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.tar.gz

# tar xvzf jdk-7u45-linux-x64.tar.gz -C /usr/local
# cd /usr/local
# ln -s jdk1.7.0_45 jdk

# vim /etc/profile
export JAVA_HOME=/usr/local/java
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin

# source /etc/profile

----------------------------------------------------------------------------------

2、MAVEN 安装
http://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz

# tar xvzf apache-maven-3.1.1-bin.tar.gz -C /usr/local
# cd /usr/local
# ln -s apache-maven-3.1.1 maven


# vim /etc/profile
export MAVEN_HOME=/usr/local/maven
export PATH=$PATH:$MAVEN_HOME/bin

# source /etc/profile


# mvn -v



3、PROTOBUF 安装
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

# tar xvzf protobuf-2.5.0.tar.gz
# ./configure --prefix=/usr/local/protobuf
# make && make install


# vim /etc/profile
export PROTO_HOME=/usr/local/protobuf
export PATH=$PATH:$PROTO_HOME/bin

# source /etc/profile

# vim /etc/ld.so.conf
/usr/local/protobuf/lib


# /sbin/ldconfig

4、其他依赖库安装
http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz
http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.9.tar.gz
http://www.openssl.org/source/openssl-1.0.1e.tar.gz

# tar xvzf cmake-2.8.12.1.tar.gz
# cd cmake-2.8.12.1
# ./bootstrap --prefix=/usr/local
# gmake && gmake install


# tar xvzf ncurses-5.9.tar.gz
# cd ncurses-5.9
# ./configure --prefix=/usr/local
# make && make install


# tar xvzf openssl-1.0.1e.tar.gz
# cd openssl-1.0.1e
# ./config shared --prefix=/usr/local
# make && make install


# /sbin/ldconfig

----------------------------------------------------------------------------------

5、编译 Hadoop
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.6.2/hadoop-2.6.2-src.tar.gz

(1)、maven源设置【在<mirrors></mirros>里添加】
# vim /usr/local/maven/conf/settings.xml
<mirror>
    <id>nexus-osc</id>
    <mirrorOf>*</mirrorOf>
    <name>Nexusosc</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
</mirror>


(2)、编译Hadoop
# tar xvzf hadoop-2.6.2-src.tar.gz
# cd hadoop-2.6.2-src
# mvn clean install -DskipTests
# mvn package -Pdist,native -DskipTests -Dtar


## 编译成功后，生成的二进制包所在路径
hadoop-dist/target/hadoop-2.6.2

创建hadoop用户
useradd hadoop

# cp -a hadoop-dist/target/hadoop-2.6.2 /home/hadoop/source
# cd /home/hadoop
# ln -s /home/hadoop/source/hadoop-2.6.2 ./hadoop


【注意：编译过程中，可能会失败，需要多尝试几次】


========================================================================================
三、Hadoop YARN 分布式集群配置【注：所有节点都做同样配置】
========================================================================================
1、环境变量设置
# vim /etc/profile  或者 .bash_profile



export HADOOP_DEV_HOME=/home/hadoop/hadoop
export PATH=$PATH:$HADOOP_DEV_HOME/bin
export PATH=$PATH:$HADOOP_DEV_HOME/sbin
export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export SPARK_HOME=/home/hadoop/spark
export PATH=$PATH:$SPARK_HOME/bin
export PATH


# source /etc/profile


3、配置 core-site.xml
# vim /home/hadoop/hadoop/etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/hadoop/tmp</value>
                <description>A base for other temporary directories.</description>
        </property>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://192.168.10.84:9000</value>
        </property>

        <property>
                <name>hadoop.proxyuser.root.hosts</name>
                <value>192.168.10.84</value>
        </property>
        <property>
                <name>hadoop.proxyuser.root.groups</name>
                <value>*</value>
        </property>
       </configuration>


4、配置 hdfs-site.xml
# vim /home/hadoop/hadoop/etc/hadoop/hdfs-site.xml

<property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/hadoop/hdfs/name</value>
                <final>true</final>
        </property>

        <property>
                <name>dfs.federation.nameservice.id</name>
                <value>ns1</value>
        </property>

        <property>
                <name>dfs.namenode.backup.address.ns1</name>
                <value>192.168.10.84:50100</value>
        </property>

        <property>
                <name>dfs.namenode.backup.http-address.ns1</name>
                <value>192.168.10.84:50105</value>
        </property>

        <property>
                <name>dfs.federation.nameservices</name>
                <value>ns1</value>
        </property>

        <property>
                <name>dfs.namenode.rpc-address.ns1</name>
                <value>192.168.10.84:9000</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.ns2</name>
                <value>192.168.10.84:9000</value>
        </property>

        <property>
                <name>dfs.namenode.http-address.ns1</name>
                <value>192.168.10.84:23001</value>
        </property>

        <property>
                <name>dfs.namenode.http-address.ns2</name>
                <value>192.168.10.84:13001</value>
        </property>

        <property>
                <name>dfs.dataname.data.dir</name>
                <value>file:/hadoop/hdfs/data</value>
                <final>true</final>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address.ns1</name>
                <value>192.168.10.84:23002</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address.ns2</name>
                <value>192.168.10.84:23002</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address.ns1</name>
                <value>192.168.10.84:23003</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address.ns2</name>
                <value>192.168.10.84:23003</value>
        </property>




------------------------------------------------------------------------------------------------------------------------------------------

6、配置 yarn-site.xml
# vim /home/hadoop/hadoop/etc/hadoop/yarn-site.xml


<property>
                <name>yarn.resourcemanager.address</name>
                <value>192.168.10.84:18040</value>
        </property>

        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>192.168.10.84:18030</value>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>192.168.10.84:18088</value>
        </property>

        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>192.168.10.84:18025</value>
        </property>

        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>192.168.10.84:18141</value>
        </property>

        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>



7、配置 hadoop-env.sh、mapred-env.sh、yarn-env.sh【在开头添加】
文件路径：
/home/hadoop/hadoop/etc/hadoop/hadoop-env.sh
/home/hadoop/hadoop/etc/hadoop/mapred-env.sh
/home/hadoop/hadoop/etc/hadoop/yarn-env.sh


添加内容：
export JAVA_HOME=/usr/local/java
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib


export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_PID_DIR=/home/hadoop/data/hadoop/pids
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"


export HADOOP_PREFIX=$HADOOP_HOME


export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME


export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop


export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native


export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

------------------------------------------------------------------------------------------------------------------------------------------

8、数据节点配置
# vim /home/hadoop/hadoop/etc/hadoop/slaves
192.168.10.83
192.168.10.84
192.168.10.85

------------------------------------------------------------------------------------------------------------------------------------------

9、Hadoop 简单测试
# cd /home/hadoop/hadoop


## 首次启动集群时，做如下操作【主名字节点上执行】
# hdfs namenode -format
# sbin/start-dfs.sh


## 检查进程是否正常启动
# jps

主名字节点：
NodeManager

备名字节点：
SecondaryNameNode


数据节点：

DataNode

## hdfs与mapreduce测试
# hdfs dfs -mkdir -p /user/test
# hdfs dfs -put bin/hdfs.cmd /user/test
# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/test  /user/out
# hdfs dfs -ls /user/out


## hdfs信息查看
# hdfs dfsadmin -report
# hdfs fsck / -files -blocks


## 集群的后续维护
# sbin/start-all.sh
# sbin/stop-all.sh


## 监控页面URL
http://192.168.10.84:80/

========================================================================================
四、Spark 分布式集群配置【注：所有节点都做同样配置】
========================================================================================
1、Scala 安装
http://www.scala-lang.org/files/archive/scala-2.9.3.tgz

# tar xvzf scala-2.9.3.tgz -C /usr/local
# cd /usr/local
# ln -s scala-2.9.3 scala


# vim /etc/profile
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
export SPARK_CLASSPATH=/home/hadoop/spark/lib


# source /etc/profile

------------------------------------------------------------------------------------------------------------------------------------------

2、Spark 安装
http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-incubating-bin-hadoop2.6tgz

# tar xvzf spark-1.6.1-incubating-bin-hadoop2.6.tgz -C /home/hadoop/source
# cd /home/hadoop
# ln -s spark-1.6.1-incubating-bin-hadoop2.6  ./spark


# vim /etc/profile
export SPARK_HOME=/home/hadoop/spark
export PATH=$PATH:$SPARK_HOME/bin


# source /etc/profile


# cd /home/hadoop/spark/conf
# mv spark-env.sh.template spark-env.sh

# vim spark-env.sh
export JAVA_HOME=/usr/local/java
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/home/hadoop/hadoop


## worker节点的主机名列表
# vim slaves
192.168.10.83
192.168.10.84
192.168.10.85


# mv log4j.properties.template log4j.properties

## 在Master节点上执行
# .sbin/start-all.sh


## 检查进程是否启动【在master节点上出现“Master”，在slave节点上出现“Worker”】
# jps

Master节点：

Master


Slave节点：

worker


3、相关测试
## 监控页面URL
http://192.168.10.84:8080/



## 先切换到“/home/hadoop/spark”目录
(1)、本地模式
# ./run-example org.apache.spark.examples.SparkPi local


(2)、普通集群模式
# ./run-example org.apache.spark.examples.SparkPi spark://namenode1:7077
# ./run-example org.apache.spark.examples.SparkLR spark://namenode1:7077
# ./run-example org.apache.spark.examples.SparkKMeans spark://namenode1:7077 file:/home/hadoop/spark/kmeans_data.txt 2 1


(3)、结合HDFS的集群模式
# hadoop fs -put README.md .
# MASTER=spark://namenode1:7077 ./spark-shell
scala> val file = sc.textFile("hdfs://namenode1:9000/user/root/README.md")
scala> val count = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)
scala> count.collect()
scala> :quit


(4)、基于YARN模式
# SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-1.6.1-incubating-hadoop2.6.2.jar \
./spark-class org.apache.spark.deploy.yarn.Client \
--jar examples/target/scala-2.9.3/spark-examples-assembly-1.6.1-incubating.jar \
--class org.apache.spark.examples.SparkPi \
--args yarn-standalone \
--num-workers 3 \
--master-memory 4g \
--worker-memory 2g \
--worker-cores 1


执行结果：
/usr/local/hadoop/logs/userlogs/application_*/container*_000001/stdout


(5)、其他一些样例程序
examples/src/main/scala/org/apache/spark/examples/


(6)、问题定位【数据节点上的日志】
/home/hadoop/hadoop/logs


(7)、一些优化
# vim /home/hadoop/spark/conf/spark-env.sh
export SPARK_WORKER_MEMORY=16g  【根据内存大小进行实际配置】
......


(8)、最终的目录结构




猜你喜欢

转载自ximeng1234.iteye.com/blog/2285142
今日推荐
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
“百模大战”必有一战 | 2024中国“百模大战”竞争格局分析
最强开源大模型 Llama 3 上架 Gitee AI
周排行
自媒体文章如何提高原创度以及如何检测原创度
开启qq邮箱的smtp服务
Qt程序单次启动（QSingleApplication类）
国外的外包网站
更新IDEA主题——放飞代码风格
cocos2dx 实现搓牌效果（翻牌效果），包括铺平动画
dict和json之间的互相转换
angular的一些思考
. Fibonacci数列是这样定义的： F[0] = 0 F[1] = 1 for each i ≥ 2: F[i] = F[i-1] + F[i-2] 因此，Fibonacci数列就形如：0, 1
洛谷P1064 金明的预算方案
每日归档
更多
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022