大数据：Hadoop：集群搭建(2.0+)

  安装Hadoop

•       解压                     tar -zxvf hadoop-2.7.3.tar.gz -C /cluster/
•   1.     vim hadoop-env.sh

  export JAVA_HOME=/usr/java/jdk1.8.0_231

•   2.   vim core-site.xml

  	<!-- 指定hdfs的nameservice为ns -->
       <property>
          <name>fs.defaultFS</name>
          <value>hdfs://node1:9000/</value>
      </property>
  	<!-- 指定hadoop临时目录 -->
  	<property>
  		<name>hadoop.tmp.dir</name>
  		<value>/cluster/hadoop-2.7.3/tmp</value>
  	</property>
   
  	<!-- 指定zookeeper地址 -->
  	<property>
  		<name>ha.zookeeper.quorum</name>
  		<value>node1:2181,node2:2181,node3:2181</value>
  	</property>

•   3.      vim hdfs-site.xml 

      <property>
          <name>dfs.namenode.secondary.http-address</name>
          <value>node1:9001</value>
      </property>
      <property>
          <name>dfs.namenode.name.dir</name>
          <value>/cluster/hadoop-2.7.3/dfs/name</value>
      </property>
      <property>
          <name>dfs.datanode.data.dir</name>
          <value>/cluster/hadoop-2.7.3/dfs/data</value>
      </property>
      <property>
          <name>dfs.replication</name>
          <value>3</value>
      </property>

•    4.     vim mapred-site.xml

      <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
      </property>

•   5.     vim yarn-site.xml

      <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
      </property>
      <property>
          <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
          <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
      <property>
          <name>yarn.resourcemanager.address</name>
          <value>node1:8032</value>
      </property>
      <property>
          <name>yarn.resourcemanager.scheduler.address</name>
          <value>node1:8030</value>
      </property>
      <property>
          <name>yarn.resourcemanager.resource-tracker.address</name>
          <value>node1:8035</value>
      </property>
      <property>
          <name>yarn.resourcemanager.admin.address</name>
          <value>node1:8033</value>
      </property>
      <property>
          <name>yarn.resourcemanager.webapp.address</name>
          <value>node1:8088</value>
      </property>

•   6.     vim slaves 

  node2
  node3

• 复制到集群其他节点    scp -r hadoop-2.7.3/ node3:/cluster/    scp -r hadoop-2.7.3/ node2:/cluster/

初始化Hadoop

• 格式化HDFS (node1操作)     hdfs namenode -format
• 启动hadoop             ./start-all.sh        或者分别启动(    启动HDFS：  ./start-dfs.sh    启动YARN ： ./start-yarn.sh   )                            
•  浏览器验证:
  • HDFS管理界面:   http://192.168.8.101:50070  
  • MR管理界面:       http://192.168.8.101:8088