main content
- Hadoop installation
premise
- zookeeper normal use
- JAVA_HOME environment variable
Installation package
Micro Cloud Download | tar package directory under
Hadoop 2.7.7
Division of roles
Role Assignments NN DN Shnhn cluster-master Yes no no cluster-slave1 no Yes Yes cluster-slave1 no Yes no cluster-slave1 no Yes no
First, prepare the environment
Upload image to docker
docker cp hadoop-2.7.7.tar.gz cluster-master:/root/tar
Decompression
tar xivf hadoop-2.7.7.tar.gz -C /opt/hadoop
Second, the configuration file
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://jinbill</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>cluster-master:2181</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>mr_jinbill</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>cluster-slave2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>cluster-slave3</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>192.168.11.46:12181</value>
</property>
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
hadoop-env.sh
export JAVA_HOME=/opt/jdk/jdk1.8.0_221
hdfs-site.xml
<configuration>
<property>
<name>dfs.nameservices</name>
<value>jinbill</value>
</property>
<property>
<name>dfs.ha.namenodes.jinbill</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.jinbill.nn1</name>
<value>cluster-master:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.jinbill.nn2</name>
<value>cluster-slave1:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.shsxt.nn1</name>
<value>cluster-master:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.shsxt.nn2</name>
<value>cluster-slave1:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://cluster-slave1:8485;cluster-slave2:8485;cluster-slave3:8485/jinbill</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.jinbill</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/hadoop/data</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
New slaves file, if the direct editing
cluster-slave1
cluster-slave2
cluster-slave3
Third, the initialization
Start all nodes JournalNode
hadoop-daemon.sh start journalnode
NN initialization metadata on
hdfs namenode -forma
Metadata formatted copy to SNN
scp /opt/zookeeper/dfs cluster-slave1:/opt/hadoop
NN start master node
hadoop-daemon.sh start namenode
Executed on SNN
hdfs namenode -bootstrapStandby
Start SNN
hadoop-daemon.sh start namenode
NN or initialization ZKFC on SNN
hdfs zkfc -formatZK
Stop above node
stop-dfs.sh
Fourth, start
start-dfs.sh
start-yarn.sh
Fifth, the test was successful
Because different network segments, so we have to add routes to access
- Open cmd, requires administrator privileges
- route add 172.15.0.0 mask 255.255.0.0 192.168.11.38 -p