1. prepare the virtual environment for the hadoop cluster.
you can choose Virtual Box or VM Wave. There are some issues with Vitural box in my labtop. So i choose VM 10.0.
then you need use the following software tools. you'd better prepare them well before you start to install the env.
maven linux version 3.11
jdk 1.7.72
protoc 2.5.0 re-compile hadoop 2.5.2
https://code.google.com/p/protobuf/downloads/list
tar -xvf protobuf-2.5.0.tar.bz2
cd protobuf-2.5.0
./configure --prefix=/opt/xxxxx/protoc/
make && make install
yum install gcc
yum intall gcc-c++
yum install make
yum install cmake
yum install openssl-devel
yum install ncurses-devel
if you haven't these tools, you more or less meet some compile problem.
2. install jdk, maven, and config maven.
I am in China, so the forgien maven central repository some time is not available, ro too slow for me. So i config a mirror maven server in China.
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
When every thing in above are ready. Next, you will download hadoop from apache offical site. be noted: download the src version. I use the version 2.5.2
mvn clean package -Pdist,native -DskipTests -Dtar
the build process will last 30-60 mins based on you PC.
if one of the maven task fails, you need build it manually to save time.
at last,you will see the sucessfull screen liking below.
main:
[exec] $ tar cf hadoop-2.5.2.tar hadoop-2.5.2
[exec] $ gzip -f hadoop-2.5.2.tar
[exec]
[exec] Hadoop dist tar available at: /root/hadoopsrc/srcdir/hadoop-2.5.2-src/hadoop-dist/target/hadoop-2.5.2.tar.gz
[exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /root/hadoopsrc/srcdir/hadoop-2.5.2-src/hadoop-dist/target/hadoop-dist-2.5.2-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [2.414s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.719s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [5.243s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.433s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [3.172s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.075s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [5.361s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [6.530s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [5.012s]
[INFO] Apache Hadoop Common .............................. SUCCESS [4:47.964s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [12.655s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.097s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [8:59.599s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [53.998s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [11.246s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [7.457s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.161s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.140s]
[INFO] hadoop-yarn-api ................................... SUCCESS [3:22.369s]
[INFO] hadoop-yarn-common ................................ SUCCESS [53.995s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.176s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [13.378s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [31.324s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.596s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [7.033s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [24.992s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.576s]
[INFO] hadoop-yarn-client ................................ SUCCESS [6.709s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.213s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [3.840s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.157s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.153s]
[INFO] hadoop-yarn-project ............................... SUCCESS [15.632s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.152s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [38.670s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [33.585s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [6.307s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [15.549s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [11.430s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [34.442s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.081s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [8.559s]
[INFO] hadoop-mapreduce .................................. SUCCESS [11.834s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [1:07.545s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [1:19.210s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [4.697s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [8.833s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [7.416s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [4.417s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [4.287s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [24.609s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [8.762s]
[INFO] Apache Hadoop Client .............................. SUCCESS [19.307s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.386s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [11.350s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [14.692s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.130s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:48.893s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:50.549s
[INFO] Finished at: Tue Dec 09 07:31:56 PST 2014
[INFO] Final Memory: 81M/243M
[INFO] ------------------------------------------------------------------------
[root@master hadoop-2.5.2-src]#
----------------------------------------------
next
install hadoop
tar hadoopxxxxx -C /opt/hadoop
create hadoop user
you must grant the follow folder authority to hadoop user
chown -R hadoop:hadoop /hadoop /opt/hadoop
switch to hadoop user
config the following 7 files for hadoop cluster
create folder
tmp
dfs/name
dfs/data
be noted:
the three folders must be mapped to the config files
next
shutdown the master server. and clone it to slave1 and slave2.
next start three servers.
config the host name and network.
configurate the ssh login from master to two slaves.
master---slave1
master---slave2
next make sure the iptables are shutdown.
for my test env.
execute the below command by root user
chkconfig iptables off
it will close iptales for ever.
chkconfig iptables on (open)
./hdfs namenode -format
then test the installation
master:
[hadoop@master sbin]$ ./start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-namenode-master.out
slave1: starting datanode, logging to /opt/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-datanode-slave2.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-secondarynamenode-master.out
[hadoop@master sbin]$ jps
2440 SecondaryNameNode
2539 Jps
2274 NameNode
[hadoop@master sbin]$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.5.2/logs/yarn-hadoop-resourcemanager-master.out
slave1: starting nodemanager, logging to /opt/hadoop/hadoop-2.5.2/logs/yarn-hadoop-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop/hadoop-2.5.2/logs/yarn-hadoop-nodemanager-slave2.out
[hadoop@master sbin]$ jps
2440 SecondaryNameNode
2660 Jps
2274 NameNode
2584 ResourceManager
[hadoop@master sbin]$ pwd
/opt/hadoop/hadoop-2.5.2/sbin
[hadoop@master sbin]$ cd ..
[hadoop@master hadoop-2.5.2]$
slave1:
[hadoop@slave1 hadoop-2.5.2]$ ls
bin dfs etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share tmp
[hadoop@slave1 hadoop-2.5.2]$ rm -rf tmp/
[hadoop@slave1 hadoop-2.5.2]$ rm -rf dfs/
[hadoop@slave1 hadoop-2.5.2]$ ls
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[hadoop@slave1 hadoop-2.5.2]$ jps
2146 Jps
2079 DataNode
[hadoop@slave1 hadoop-2.5.2]$ jps
2213 Jps
2079 DataNode
2182 NodeManager
[hadoop@slave1 hadoop-2.5.2]$
slave2:
[hadoop@slave2 hadoop-2.5.2]$ jps
2080 DataNode
2147 Jps
[hadoop@slave2 hadoop-2.5.2]$ jps
2270 Jps
2080 DataNode
2183 NodeManager
[hadoop@slave2 hadoop-2.5.2]$
check cluster nodes:
http://192.168.23.129:8088/cluster/nodes
check the status of every node
http://192.168.23.129:50070/dfshealth.html 
./stop-dfs.sh
./stop-yarn.sh
[root@localhost ~]# groupadd hadoop
[root@localhost ~]# useradd -g hadoop hadoop
[root@localhost ~]# passwd hadoop
cat id_rsa.pub > authorized_keys
chmod go-rw ~/.ssh/authorized_keys
scp * hadoop@slave1:/opt/hadoop/xxxxx
[hadoop@slave1 ~]$ chmod 700 .ssh
[hadoop@slave2 ~]$ mkdir ~/.ssh
[hadoop@slave2 ~]$ chmod 700 .ssh
done!
My QQ: 735028566
http://www.iteye.com/topic/1136809
http://my.oschina.net/fhzsy/blog/363045
http://www.iteye.com/topic/1136947
run jar file
[hadoop@hadoopmaster sbin]$ hadoop jar /opt/jack.jar org.apache.hadoop.t1.WordCount /jackdemodir/wordcount/input /jackdemodir/wordcount/output1
15/08/01 22:44:35 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/192.168.1.50:8032
15/08/01 22:44:37 INFO input.FileInputFormat: Total input paths to process : 1
15/08/01 22:44:37 INFO mapreduce.JobSubmitter: number of splits:1
15/08/01 22:44:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1438494222950_0001
15/08/01 22:44:38 INFO impl.YarnClientImpl: Submitted application application_1438494222950_0001
15/08/01 22:44:38 INFO mapreduce.Job: The url to track the job: http://hadoopmaster:8088/proxy/application_1438494222950_0001/
15/08/01 22:44:38 INFO mapreduce.Job: Running job: job_1438494222950_0001
15/08/01 22:44:47 INFO mapreduce.Job: Job job_1438494222950_0001 running in uber mode : false
15/08/01 22:44:47 INFO mapreduce.Job: map 0% reduce 0%
15/08/01 22:44:55 INFO mapreduce.Job: map 100% reduce 0%
15/08/01 22:45:01 INFO mapreduce.Job: map 100% reduce 100%
15/08/01 22:45:02 INFO mapreduce.Job: Job job_1438494222950_0001 completed successfully
15/08/01 22:45:02 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=571
FILE: Number of bytes written=212507
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=463
HDFS: Number of bytes written=385
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5427
Total time spent by all reduces in occupied slots (ms)=4297
Total time spent by all map tasks (ms)=5427
Total time spent by all reduce tasks (ms)=4297
Total vcore-seconds taken by all map tasks=5427
Total vcore-seconds taken by all reduce tasks=4297
Total megabyte-seconds taken by all map tasks=5557248
Total megabyte-seconds taken by all reduce tasks=4400128
Map-Reduce Framework
Map input records=1
Map output records=55
Map output bytes=556
Map output materialized bytes=571
Input split bytes=128
Combine input records=55
Combine output records=45
Reduce input groups=45
Reduce shuffle bytes=571
Reduce input records=45
Reduce output records=45
Spilled Records=90
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=131
CPU time spent (ms)=1910
Physical memory (bytes) snapshot=462319616
Virtual memory (bytes) snapshot=1765044224
Total committed heap usage (bytes)=275251200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=335
File Output Format Counters
Bytes Written=385
check the result
[hadoop@hadoopmaster sbin]$ hadoop fs -cat /jackdemodir/wordcount/output1/part-r-00000
350 1
ASF 1
Abdera 1
Apache? 1
Are 1
From 1
Open 2
Source 2
The 1
Zookeeper, 1
a 2
all-volunteer 1
and 3
are 3
by 1
chances 1
cover 1
develops, 1
experience 1
find 1
for 1
going 1
here. 1
if 1
in 1
incubates 1
industry 1
initiatives 1
it 1
leading 1
looking 1
more 1
of 1
powered 1
projects 1
range 1
rewarding 1
software, 1
stewards, 1
technologies. 1
than 1
that 1
to 2
wide 1
you 3