1. First plan the service to be installed
| Ip | host | install software |
process |
| 10.10.10.5 |
master | hadoop、zookeeper | NameNode |
| DFSZKFailoverController | |||
| JournalNode |
|||
| DataNode | |||
| ResourceManager | |||
| jobHistoryServer | |||
| NodeManager | |||
| 10.10.10.6 |
slave1 | hadoop、zookeeper | NameNode |
| DFSZKFailoverController | |||
| JournalNode | |||
| dataNode | |||
| ResourceManager | |||
| NodeManager | |||
| QuoruPeerMain | |||
| 10.10.10.6 | slave2 | hadoop、zookeeper | JournalNode |
| DataNode | |||
| NodeManager | |||
| QuorumPeerMain |
Environmental preparation
Turn off firewall
systemctl stop iptables.service
systemctl disable iptables.service
systemctl disable iptables.service
1. Upload the installation package hadoop-2.6.0-cdh5.16.2.tar.gz zookeeper-3.4.5-cdh5.16.2.tar.gz to the /opt/soft directory
2. Set the host name
master:
hostname master
vi /etc/sysconfig/network
slave1:
hostname slave1
slave2:
hostname slave2
Configure the mapping relationship between ip and hostname
vim cat /etc/hosts
By sending the modified file to slave1 and slave2
scp /etc/hosts root@slave1:/etc/
scp /etc/hosts root@slave2:/etc/
I have configured mutual trust on the three services, so I can send it directly. If it can’t be sent directly, please Baidu to see how to configure mutual trust
3. Configure the jdk environment hadoop zookeeper
As shown in the figure, the decompressed files of my jdk jar package are in /usr/local/jdk, hadoop:/opt/soft2/hadoop zookeeper:/opt/soft2/zookeeper
4. Modify the zookeeper configuration
cd /opt/soft2/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
I came to zoo.cfg
Mainly modify dataDir, the path where zk stores data
mkdir /opt/soft2/zookeeper/zkData
使用 scp -r zookeeper slave1:/opt/soft2/
使用 scp -r zookeeper slave2:/opt/soft2/
Copy the entire zookeeper file to the remaining nodes
In the data directory of each node according to the configuration file
master中 echo 1 > /opt/soft2/zookeeper/zkData/myid
slave1中 echo 2 > /opt/soft2/zookeeper/zkData/myid
slave2中 echo 3 > /opt/soft2/zookeeper/zkData/myid
Install hadoop
Modify the configuration file of hadoop
cd /opt/soft2/hadoop/etc/hadoop
vim hadoop-env.sh
Configure jdk environment
Configure the core configuration of hadoop
vim core-site.xml
<configuration>
<!--Yarn needs to use fs.defaultFS to specify the NameNode URI -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<!--==============================Trash mechanism============== ======================== -->
<property>
<name>fs.trash.checkpoint.interval</name>
<value>0</value>
</property>
<property>
<!--How many minutes. The CheckPoint directory under Trash will be deleted, the configuration server setting priority is greater than the client, default: 0 do not delete-->
<name>fs.trash.interval</name>
<value>10080</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/soft2/hadoop/data</value>
</property>
<!-- Specify zookeeper address-->
<property>
<name>ha.zookeeper.quorum</name>
<value>master:2181,slave1:2181,slave2:2181</value>
</property>
<!--Specify ZooKeeper timeout interval, in milliseconds-->
<property>
<name>ha.zookeeper.session-timeout.ms</name>
<value>2000</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec
</value>
</property>
</configuration>
vim hdfs-site.xml configure hdfs
<configuration>
<!--HDFS Super User-->
<property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
<!--Open web hdfs -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/soft2/hadoop/data/dfsname</value>
<description> namenode stores name table (fsimage) local directory (need to be modified)</description>
</property>
<property>
<name>dfs.namenode.edits.dir</name>
<value>${dfs.namenode.name.dir}</value>
<description>namenode extensive transaction file(edits) local directory (need to modify)</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/soft2/hadoop/data/dfsdata</value>
<description>The local directory where the datanode stores the block (need to be modified)</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- Block size 128M (default 128M) -->
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<!--======================================================================= -->
<!--HDFS High Availability Configuration-->
<!--Specify the nameservice of hdfs as mycluster, which needs to be consistent with that in core-site.xml -->
<property>
</property>
<!-- Block size 128M (default 128M) -->
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<!--======================================================================= -->
<!--HDFS High Availability Configuration-->
<!--Specify the nameservice of hdfs as mycluster, which needs to be consistent with that in core-site.xml -->
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<!--Set NameNode IDs This version only supports two NameNodes at most -->
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<!-- Hdfs HA: dfs.namenode.rpc-address.[nameservice ID] rpc communication address-->
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>master:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>slave1:8020</value>
</property>
<!-- Hdfs HA: dfs.namenode.http-address.[nameservice ID] http communication address-->
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>master:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.ruozeclusterg10.nn2</name>
<value>ruozedata002:50070</value>
</property>
<!--==================Namenode editlog synchronization========================= =================== -->
<!--Guaranteed Data Recovery-->
<property>
<name>dfs.journalnode.http-address</name>
<value>0.0.0.0:8480</value>
</property>
<property>
<name>dfs.journalnode.rpc-address</name>
<value>0.0.0.0:8485</value>
</property>
<property>
<!--Set the JournalNode server address, QuorumJournalManager is used to store editlog -->
<!--格式：qjournal://<host1:port1>;<host2:port2>;<host3:port3>/<journalId> 端口同journalnode.rpc-address -->
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://master:8485;slave1:8485;slave2:8485/mycluster</value>
</property>
<property>
<!--JournalNode storage data address-->
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/data/dfs/jn</value>
</property>
<!--===================DataNode editlog synchronization========================= =================== -->
<property>
<!--DataNode, Client connects to Namenode to identify and select Active NameNode strategy-->
<!-- Configuration failure automatic switching implementation mode -->
<name>dfs.client.failover.proxy.provider.ruozeclusterg10</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!--==================Namenode fencing：=============================================== -->
<!-- Prevent the stopped Namenode from starting after Failover, resulting in two services -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
<!--How many milliseconds think fencing failed-->
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
<!--==================NameNode auto failover base ZKFC and Zookeeper====================== -->
<!--Enable based on Zookeeper -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!--Dynamic permission datanode connection namenode list-->
<property>
<name>dfs.hosts</name>
<value>/opt/soft2/hadoop/etc/hadoop/slaves</value>
</property>
</configuration>
Modify mapred-site.xml
The configuration does not exist in the configuration
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<configuration>
<!-- 配置 MapReduce Applications -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- JobHistory Server ============================================================== -->
<!-- Configure MapReduce JobHistory Server address, default port 10020 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<!-- Configure the MapReduce JobHistory Server web ui address, the default port is 19888 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>slave1:19888</value>
</property>
<!-- Configure the compression of Map segment output, snappy-->
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
</configuration>
vim slaves add the following
master
slave1
slave2
vim yarn-env.sh
vim yarn-site.xml
<configuration>
<!-- nodemanager configuration========================================= ===== -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
<description>Address where the localizer IPC is.</description>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
<description>NM Webapp address.</description>
</property>
<!-- HA configuration =========================================== =================== -->
<!-- Resource Manager Configs -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- Enable embedded automatic failover. Start the HA environment and work with ZKRMStateStore to handle fencing -->
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<!-- Cluster name, ensure the corresponding cluster during HA election -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!--Here RM master and backup nodes need to be specified separately, (optional)
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm2</value>
</property>
-->
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<!-ZKRMStateStore placement->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>master:2181,slave1:2181,slave2:2181</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>master:2181,slave1:2181,slave2:2181</value>
</property>
<!-- RPC address for Client to access RM (applications manager interface) -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>master:23140</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>slave1:23140</value>
</property>
<!-- AM access to RM RPC address (scheduler interface) -->
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>master:23130</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>slave1:23130</value>
</property>
<!-- RM admin interface -->
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>master:23141</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>slave1:23141</value>
</property>
<!--The RPC port of NM to access RM-->
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>master:23125</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>slave1:23125</value>
</property>
<!-- RM web application address-->
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>slave1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>master:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>slave1:23189</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
<discription>A single task can apply for the minimum memory, the default is 1024MB</discription>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
<discription>A single task can apply for the maximum memory, the default is 8192MB</discription>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>
Start zk
Run on all three computers: Start zkServer.sh start to view the status zkServer.sh status
Start hadoop (hdfs+yarn)
1. All three computers start the journal JournalNode
hadoop-daemon.sh start journalnode
2. Format hadoop
hadoop purpose -format
Send the generated metadata to each node
[root@master hadoop]# scp -r data slave1:/opt/soft2/hadoop/
fsimage_0000000000000000000 100% 317 0.3KB/s 00:00
VERSION 100% 202 0.2KB/s 00:00
fsimage_0000000000000000000.md5 100% 62 0.1KB/s 00:00
seen_txid
3. Initialize zkfc
hdfs zkfc -formatZK
4. Start the distributed file system of hdfs
start-dfs.sh
5. Start yarn
start-yarn.sh