2.下载jdk1.6 version 1.6.0_29
3.准备4台机器
| name | ip | type |
| oplog.master | 10.125.14.205 | master |
| oplog.slave1 | 10.125.14.195 | slave |
| oplog.slave2 | 10.125.14.19 | slave |
| oplog.slave3 | 10.125.14.176 | slave |
4.修改/etc/profile,增加如下配置(四台机器均加入)
.. export JAVA_HOME=/home/jincheng.sunjc/softs/jdk1.6.0_29/ export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib: ..
[jincheng.sunjc@v014205-bja hadoop20205]$ source /etc/profile
5.修改/etc/hosts,增加如下配置(四台机器均加入)
10.125.14.205 oplog.master 10.125.14.195 oplog.slave1 10.125.14.19 oplog.slave2 10.125.14.176 oplog.slave3
[jincheng.sunjc@v014205-bja hadoop20205]$ source /etc/profile
6.修改../hadoop20205/conf/hadoop-env.sh
export JAVA_HOME=/home/jincheng.sunjc/softs/jdk1.6.0_29/ expor tHADOOP_HOME_WARN_SUPPRESS=TRUE
7.修改../hadoop20205/conf/core-site.xml
<configuration> <property> <name></name> <value>/home/jincheng.sunjc/softs/hadoop/data</value> </property> <property> <name>fs.default.name</name> <value>hdfs://10.125.14.205:9000</value> </property> </configuration>
8.修改hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>
9.修改mapred-site.xml
<configuration> <property> <name>mapred.job.tracker</name> <value>http://10.125.14.205:9001</value> </property> </configuration>
10.配置修改../hadoop20205/conf/masters
oplog.master
11.配置修改../hadoop20205/conf/slaves
oplog.slave1 oplog.slave2 oplog.slave3
12.拷贝master机器的目录到slave1 slave2 slave3
scp -r softs/ slave1:/home/jincheng.sunjc/. scp -r softs/ slave2:/home/jincheng.sunjc/. scp -r softs/ slave3:/home/jincheng.sunjc/.
13.配置master本地免登
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
14.配置master和slave1 slave2 slave3 免登
master：scp ~/.ssh/id_dsa.pub slave1:/home/jincheng.sunjc/. slave1：cat /home/jincheng.sunjc/id_dsa.pub >> ~/.ssh/authorized_keys slave2和slave3同样操作
15.启动集群
[jincheng.sunjc@v014205-bja bin]$ ./hadoop namenode -format 13/04/22 16:50:39 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = v014205-bja/10.125.14.205 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 0.20.205.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205 -r 1179940; compiled by 'hortonfo' on Fri Oct 7 06:20:32 UTC 2011 ************************************************************/ Re-format filesystem in /tmp/hadoop-jincheng.sunjc/dfs/name ? (Y or N) Y 13/04/22 16:50:41 INFO util.GSet: VM type = 64-bit 13/04/22 16:50:41 INFO util.GSet: 2% max memory = 19.33375 MB 13/04/22 16:50:41 INFO util.GSet: capacity = 2^21 = 2097152 entries 13/04/22 16:50:41 INFO util.GSet: recommended=2097152, actual=2097152 13/04/22 16:50:41 INFO namenode.FSNamesystem: fsOwner=jincheng.sunjc 13/04/22 16:50:41 INFO namenode.FSNamesystem: supergroup=supergroup 13/04/22 16:50:41 INFO namenode.FSNamesystem: isPermissionEnabled=true 13/04/22 16:50:41 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 13/04/22 16:50:41 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 13/04/22 16:50:41 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/04/22 16:50:42 INFO common.Storage: Image file of size 120 saved in 0 seconds. 13/04/22 16:50:42 INFO common.Storage: Storage directory /tmp/hadoop-jincheng.sunjc/dfs/name has been successfully formatted. 13/04/22 16:50:42 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at v014205-bja/10.125.14.205 ************************************************************/
[jincheng.sunjc@v014205-bja bin]$ ./start-all.sh starting namenode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-namenode-v014205-bja.out oplog.slave3: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014176-bja.out oplog.slave1: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014195-bja.out oplog.slave2: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014019-bja.out oplog.master: starting secondarynamenode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-secondarynamenode-v014205-bja.out starting jobtracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-jobtracker-v014205-bja.out oplog.slave1: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014195-bja.out oplog.slave2: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014019-bja.out oplog.slave3: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014176-bja.out [jincheng.sunjc@v014205-bja bin]$
16.验证集群启动情况
1）验证master,有namenode jobtracker和secondaryNameNode进程
[jincheng.sunjc@v014205-bja bin]$ jps 4559 Jps 4345 SecondaryNameNode 4443 JobTracker 4170 NameNode [jincheng.sunjc@v014205-bja bin]$
2）验证slave,有dataNode 和 tasktracker进程
[jincheng.sunjc@v014195-bja hadoop]$ jps 32618 DataNode 32718 TaskTracker 330 Jps [jincheng.sunjc@v014195-bja hadoop]$
3）命令校验