hadoop(6)2.6.0 and Example
1. Prepare
ProtocolBuffer Installation
>
git clone https://github.com/google/protobuf.git
>
./autogen.sh
>
./configure --prefix=/home/carl/tool/protobuf-3.0.0
>
make
>
make check
>
make install
Add that to path.
>
protoc --version
libprotoc 3.0.0
Need to switch to 2.5.0 version
JDK Installation
>
java -version
java version "1.6.0_45"
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_
MAVEN Installation
Unzip and place in the right directory, adding to path
>
mvn --version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)
Maven home: /opt/maven
Add this to stop the out of memory error
export
MAVEN_OPTS=
"
-Xmx512m -XX:MaxPermSize=128m
"
Hadoop Src Build
>
mvn package -Pdist -DskipTests -Dtar
Find the release dist file
hadoop-2.6.0.tar.gz
I get “JAVA_HOME not set” Error message, I already have the JAVA_HOME in .profile, but I added it in
/opt/hadoop/etc/hadoop/hadoop-env.sh as well.
export JAVA_HOME="/usr/lib/jvm/java-8-oracle"
Configure the SSH on master
>
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Format file system
Format the filesystem
>
hdfs namenode -format
All the configuration should take reference from
It works. Change the etc/hadoop/slaves files to have a machine list there.
Command line to start the HDFS and YARN
>
sbin/start-dfs.sh
>
sbin/start-yarn.sh
Tips
Error Message:
autoreconf: not found
Solution:
>
sudo apt-get install autoconf
Error Message:
Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: libtoolize is needed because this package uses Libtool
Solution:
>
sudo apt-get install libtool
Error Message:
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/carl/install/protobuf':
configure: error: in `/home/carl/install/protobuf':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
Solution:
>
sudo apt-get install g++
Error Message:
protoc version is 'libprotoc 3.0.0', expected version is '2.5.0'
Solution:
Switch to 2.5.0
References:
http://sillycat.iteye.com/blog/1556106 Hadoop doc and Installation on ubuntu with an old version
http://sillycat.iteye.com/blog/1556107 cluster of the old version
http://sillycat.iteye.com/blog/2084169 yet another resource negotiator 2.4.0
http://sillycat.iteye.com/blog/2090186 cluster 2.4.1
http://sillycat.iteye.com/blog/2090937 eclipse and example