Spark on Mesos集群部署
1.安装Mesos
- 安装最新版的Mesos
rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
yum install mesos
安装指定版本Mesos
- 找到指定Mesos版本的RPM包地址
- 下载安装Mesos
wget http://repos.mesosphere.com/el/7/x86_64/RPMS/mesos-0.28.2-2.0.27.centos701406.x86_64.rpm
rpm -ivh mesos-0.28.2-2.0.27.centos701406.x86_64.rpm
2.Mesos Master配置
/etc/mesos/zk
zk://zookeeper1001.com.cn:2181/mesos
/etc/mesos-master/hostname
/etc/mesos-master/quorum
2
/etc/mesos-master/zk_session_timeout
60secs
3.Mesos Slave配置
/etc/mesos-slave/executor_registration_timeout
5mins
/etc/mesos-slave/work_dir
/data0/mesos/slave
/etc/mesos-slave/hostname
- /etc/mesos-slave/ip
4.Spark配置
spark-env.sh
MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
spark-defaults.conf
spark.master mesos://ip:5050
spark.executor.url http://hostname/packages/spark/spark-1.4.1-bin-hadoop2.4.tgz