Experimental environment: mesos version: 0.21.0, operating system: Red Hat 6
In order to understand the three deployments of Spark, standalone, mesos and YARN. So I want to try to deploy a mesos cluster. According to my own needs, I refer to the following steps in the official documentation:
Downloading Mesos
There are different ways you can get Mesos:
Download the latest stable release from Apache (Recommended)
$ wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz $ tar -zxf mesos-0.22.1.tar.gz
CentOS 6.5
Following are the instructions for stock CentOS 6.5. If you are using a different OS, please install the packages accordingly.
Mesos 0.21.0+ requires subversion 1.8+ devel package which is not available by default by yum.
Add one of the repo that has subversion-devel 1.8 available, i.e:
Add new repo /etc/yum.repos.d/wandisco-svn.repo, with:
[WandiscoSVN]
name=Wandisco SVN Repo
baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/
enabled=1
gpgcheck=0
$ sudo yum groupinstall -y "Development Tools"
$ sudo yum install -y python-devel java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
# Install maven.
$ wget http://mirror.nexcess.net/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
ud sudo tar -zxf apache-maven-3.0.5-bin.tar.gz -C / opt /
$ sudo ln -s /opt/apache-maven-3.0.5/bin/mvn /usr/bin/mvn
Building Mesos
# Change working directory.
$ cd months
# Bootstrap (***Skip this if you are not building from git repo***).
$ ./bootstrap
# Configure and build.
$ mkdir build
$ cd build
$ ../configure
$ make
# Run test suite.
$ make check
# Install (***Optional***).
$ make install
Examples
Mesos comes bundled with example frameworks written in C++, Java and Python.
# Change into build directory. $ cd build # Start mesos master (***Ensure work directory exists and has proper permissions***). $ ./bin/mesos-master.sh --ip = 127.0.0.1 --work_dir = / var / lib / mesos # Start mesos slave. $ ./bin/mesos-slave.sh --master=127.0.0.1:5050 # Visit the mesos web page. $ http://127.0.0.1:5050 # Run C++ framework (***Exits after successfully running some tasks.***). $ ./src/test-framework --master=127.0.0.1:5050 # Run Java framework (***Exits after successfully running some tasks.***). $ ./src/examples/java/test-framework 127.0.0.1:5050 # Run Python framework (***Exits after successfully running some tasks.***). $ ./src/examples/python/test-framework 127.0.0.1:5050 NOTE: To build the example frameworks, make sure you build the test suite by doingmake check.
But building mesos is a lot harder than I thought. It took me about a day and a half to download it, prepare the environment for installation, and simply test it. Almost every step will encounter problems, so it is specially recorded for future reference:
Download Mesos:
There are two ways to obtain Mesos on the Apache Mesos official document:
1. Download from the Apache official website (recommended method).
$ wget http://www.apache.org/dist/mesos/0.21.0/mesos-0.21.0.tar.gz $ tar -zxf mesos-0.21.0.tar.gz 2. Get it from the Mesos git repository (only for the way of the great gods) $ git clone https://git-wip-us.apache.org/repos/asf/mesos.git
Because Spark1.3.1 supports mesos 0.21.0, I did not use the latest mesos version, so I used version 0.21.0. I used the first method to download Mesos. The downloaded Mesos is only the source code, not the binary package. So we need to compile according to our own operating system. Before compiling, first confirm the environment required to compile mesos.
System requirements:
Mesos runs on Linux (64 Bit) and Mac OSX (64 Bit):
CentOS 6.5
1. Make sure the source under /etc/yum.repos.d is CentOS, and the version is the same as Linux.
My Red Hat 6 uses the CentOS 6 source. If Ubuntu, you can refer to the official documentation to do it.
I used the source of CentOS5 in Red Hat 6 before, which caused problems when updating the environment later, so I also recorded the source of CentOS6:
In the directory /etc/yum.repos.d, the file name can be taken by myself, I use The name of the file is CentOS6.repo, and its content is as follows:
[base]
name=CentOS-6 - Base - 163.com baseurl=http://mirrors.163.com/centos/6/os/$basearch/ #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os #gpgcheck=1 #gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 ##released updates #[updates] #name=CentOS-6 - Updates - 163.com #baseurl=http://mirrors.163.com/centos/6/updates/$basearch/ ##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates #gpgcheck=1 #gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 ##additional packages that may be useful #[extras] #name=CentOS-6 - Extras - 163.com #baseurl=http://mirrors.163.com/centos/6/extras/$basearch/ ##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras #gpgcheck=1 #gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 ##additional packages that extend functionality of existing packages # [centosplus] #name=CentOS-6 .2- Plus - 163.com #baseurl = http: //mirrors.163.com/centos/6/centosplus/$basearch/ ##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus #gpgcheck=1 #enabled=0 #gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 ##contrib - packages by Centos Users #[contrib] #name=CentOS-6 - Contrib - 163.com #baseurl=http://mirrors.163.com/centos/6/contrib/$basearch/ ##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib #gpgcheck=1 #enabled=0 #gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 Then execute: yum clean all && yum clean metadata && yum clean dbcache && yum makecache && yum update
2. Mesos 0.21.0+ requires subversion 1.8+ deve package, which is not available in the default yum. A new repo can be added, /etc/yum.repos.d/wandisco-svn.repo whose content is:
[WandiscoSVN]
name=Wandisco SVN Repo
baseurl=http://opensource.wandisco.com/centos/6/ svn-1.8/RPMS/$basearch/
enabled=1
gpgcheck=0
I'm not sure if the file /etc/yum.repos.d/wandisco-svn.repo has any effect, add the file wandisco-svn.repo to / etc/yum.repos.d.
3. Execute sudo yum groupinstall -y "Development Tools" and encounter the following problems:
public key for systemtap-client-2.5-5.el6.x86_64.rpm is not installed The
reason is that systemtap-client-2.5-5.el6 is missing. x86_64.rpm. Just download this rpm and execute rpm -ihv systemtap-client-2.5-5.el6.x86_64.rpm to install it. If other dependencies are missing, install other dependencies first.
4、执行sudo yum install -y python-devel java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
5、安装maven
构建Mesos：
# Change working directory. $ cd months # Configure and build. $ mkdir build $ cd build
$ ../configure
encountered the following problems when executing to ../configure:
1) configure: error: cannot find libz. libz is required for mesos to build.
The reason is the lack of zlib-devel-1.2.3-25.el6 .x86_64
needs to download the corresponding rpm and install:
rpm -ihv zlib-devel-1.2.3-25.el6.x86_64.rpm
2) configure: error: cannot find libcurl. libcurl is required for mesos to build. The
reason is the lack of libcurl-devel-7.19.7-16.el6.x86_64
needs to download the corresponding rpm and install it:
rpm -ihv libidn-devel-1.18-2.el6.x86_64.rpm
rpm -ihv libcurl-devel-7.19.7-16 .el6.x86_64.rpm
3) configure: error: cannot find libapr-1. libapr-1 is required for mesos to build. The
reason is that apr-devel-1.3.9-3.el6.x86_64
needs to download the corresponding rpm, and install:
rpm -ihv apr-1.3.9-3.el6.x86_64.rpm rpm -ihv apr-devel-1.3.9-3.el6.x86_64.rpm rpm -ihv apr-util-1.3.9-3.el6.x86_64.rpm 4) configure：error：cannot find libsvn_subr-1 headers. libsubversion-1 is required for mesos to build. Find the following packages from http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/ and install: rpm -ihv serf-1.2.0-1.x86_64.rpm rpm -ihv subversion-debuginfo-1.8.0-1.x86_64.rpm rpm -ihv subversion-1.8.0-1.x86_64.rpm rpm -ihv subversion-javahl-1.8.0-1.x86_64.rpm rpm -ihv subversion-devel-1.8.0-1.x86_64.rpm
5) configure：error：cannot find libsasl2. We need libsasl2 for authentication!
安装 rpm -ihv cyrus-sasl-devel-2.1.23-8.el6.x86_64.rpm
6) 如下图的错误：
libcurl is required for mesos to build.
安装 libcurl-devel
libapr-1 is required for mesos to build
安装apr-devel-1.3.9-3.el6.x86_64
libsubversion-1 is required for mesos to build.
# 处理方法
yum install subversion
yum install subversion-javahl
yum install subversion-devel
We need CRAM-MD5 support for SASL authentication.
yum install cyrus-sasl-md5
configure: error: in `/data/mesos/mesos-0.28.0/build':
configure: error:
Could not link test program to Python. Maybe the main Python library has been
installed in some non-standard library path. If so, pass it to configure,
via the LDFLAGS environment variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
============================================================================
ERROR!
You probably have to install the development version of the Python package
for your distribution. The exact name of this package varies among them.
============================================================================
See `config.log' for more details
yum install Python-devel
$ make
遇到如下问题：
In order to solve the above problem, the following packages need to be installed:
rpm -ihv apr-util-1.3.9-3.el6.x86_64.rpm rpm -ihv apr-util-ldap-1.3.9-3.el6.x86_64.rpm rpm -ihv db4-cxx-4.7.25-16.el6.x86_64.rpm rpm -ihv db4-devel-4.7.25-16.el6.x86_64.rpm rpm -ihv expat-devel-2.0.1-9.1.el6.x86_64.rpm rpm -ihv openldap-devel-2.4.19-15.el6.x86_64.rpm rpm -ihv apr-util-devel-1.3.9-3.el6.x86_64.rpm
# Run test suite.
$ make check
encountered the following problem:
It means that the Linux kernel is too old, I did not bird this error, and skipped directly to the next step.
# Install (***Optional***).
$ make install
We need CRAM-MD5 support for SASL authentication.
yum install cyrus-sasl-md5
test:
# Change into build directory.
$ cd build # Start mesos master (***Ensure work directory exists and has proper permissions***). $ ./bin/mesos-master.sh --ip = 127.0.0.1 --work_dir = / var / lib / mesos # Start mesos slave. $ ./bin/mesos-slave.sh --master=127.0.0.1:5050 (you can add an & at the end) # Visit the mesos web page. $ http://127.0.0.1:5050 # Run Java framework (***Exits after successfully running some tasks.***). $ ./src/examples/java/test-framework 127.0.0.1:5050
There is no big problem in the test steps, you can access the page, it should be noted that the process of mesos cannot be found using jps.