Today I will wrap up the Hadoop part. This is the last article in the Hadoop series. Because HA requires Zookeeper, after explaining part of Zookeeper, I will come back to the HA part of Hadoop and follow the column. "Break the Cocoon and Become a Butterfly-Big Data" , see more related content~
table of Contents
2.1 HDFS HA working mechanism
2.2.1 Environmental preparation
2.2.3 Configure Zookeeper cluster
2.2.6 Configure automatic failover
One, HA overview
HA (High Available), that is, high availability (7*24 hours non-stop service). The most critical strategy for achieving high availability is to eliminate single points of failure. Strictly speaking, HA should be divided into the HA mechanism of each component: HDFS HA and YARN HA. Before Hadoop 2.0, there was a single point of failure (SPOF) for the NameNode in the HDFS cluster. NameNode mainly affects HDFS clusters in the following two aspects: (1) An accident occurs to the NameNode machine, such as a downtime, the cluster will be unavailable until the administrator restarts. (2) The NameNode machine needs to be upgraded, including software and hardware upgrades. At this time, the cluster will also be unavailable.
The HDFS HA function solves the above problems by configuring Active/Standby two NameNodes to implement hot standby of the NameNode in the cluster. If there is a failure, such as the machine crashes or the machine needs to be upgraded and maintained, then the NameNode can be quickly switched to another machine in this way.
Two, HDFS HA
2.1 HDFS HA working mechanism
2.1.1 Main points of work
1. Metadata management methods need to be changed.
(1) Each metadata is stored in the memory;
(2) Only the NameNode node in the Active state can write to the Edits log;
(3) Both NameNodes can read Edits;
(4) Shared Edits are placed in a shared Storage management (qjournal and NFS are two mainstream implementations);
2. A state management function module is required.
A zkfailover is implemented, which resides in the node where each namenode is located, and each zkfailover is responsible for monitoring its own NameNode node, and uses zk for status identification. When the state needs to be switched, zkfailover is responsible for the switch, and the brain split needs to be prevented during the switch. The occurrence of the phenomenon.
3. It is necessary to ensure that the two NameNodes can log in with SSH without password.
4. Fence, that is, only one NameNode provides external services at the same time.
2.1.2 Working mechanism
1. Automatic failover adds two new components to HDFS deployment: ZooKeeper and ZKFailoverController (ZKFC) processes. ZooKeeper is a highly available service that maintains a small amount of coordinated data, informs the client of these data changes, and monitors client failures. The automatic failover of HA relies on the following features of ZooKeeper:
(1) Failure detection: Each NameNode in the cluster maintains a persistent session in ZooKeeper. If the machine crashes, the session in ZooKeeper will be terminated, and ZooKeeper notifies the other NameNode that a failover needs to be triggered.
(2) Active NameNode selection: ZooKeeper provides a simple mechanism for only selecting a node as active. If the currently active NameNode crashes, another node may obtain a special exclusive lock from ZooKeeper to indicate that it should become the active NameNode.
2. ZKFC is another new component in automatic failover. It is the client of ZooKeeper and also monitors and manages the status of the NameNode. Each host running NameNode also runs a ZKFC process, and ZKFC is responsible for:
(1) Health monitoring: ZKFC uses a health check command to periodically ping the NameNode on the same host. As long as the NameNode returns to the health status in time, ZKFC considers the node to be healthy. If the node crashes, freezes or enters an unhealthy state, the health monitor identifies the node as unhealthy.
(2) ZooKeeper session management: When the local NameNode is healthy, ZKFC maintains a session opened in ZooKeeper. If the local NameNode is active, ZKFC also maintains a special znode lock, which uses ZooKeeper's support for transient nodes. If the session is terminated, the lock node will be automatically deleted.
(3) Selection based on ZooKeeper: If the local NameNode is healthy and ZKFC finds that no other node currently holds the znode lock, it will acquire the lock for itself. If it succeeds, it has won the choice and is responsible for running the failover process to make its local NameNode Active. The failover process is similar to the manual failover described earlier. First, if necessary, protect the previous active NameNode, and then the local NameNode is converted to the Active state.
2.2 HDFS HA configuration
2.2.1 Environmental preparation
Including: modify the IP, change the host name, host mapping, turn off the firewall, turn off the security subsystem, configure key-free login, install JDK configuration environment variables and so on. These are all mentioned in the previous article, so I won't repeat them here. You can refer to "2. Building a Hadoop Operating Environment under Linux" .
2.2.2 Cluster Node Planning
| master |
slave01 |
slave02 |
| NameNode |
NameNode |
|
| JournalNode |
JournalNode |
JournalNode |
| DataNode |
DataNode |
DataNode |
| ZK |
ZK |
ZK |
|
|
ResourceManager |
|
| NodeManager |
NodeManager |
NodeManager |
2.2.3 Configure Zookeeper cluster
The steps are the same as the steps for distributed installation of Zookeeper in "Second, Linux Installation of Zookeeper" .
2.2.4 Configure HDFS HA
1. First, create a new ha directory under the /opt/modules directory, and copy the hadoop-2.7.2 directory to the ha directory:
2. Configure JAVA_HOME in hadoop-env.sh:
3. Configure the core-site.xml file
<configuration>
<!-- 把两个NameNode的地址组装成一个集群namenodecluster -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://namenodecluster</value>
</property>
<!-- 指定hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/modules/ha/hadoop-2.7.2/data/tmp</value>
</property>
</configuration>
4. Configure hdfs-site.xml file
<configuration>
<!-- 完全分布式集群名称 -->
<property>
<name>dfs.nameservices</name>
<value>namenodecluster</value>
</property>
<!-- 集群中NameNode节点都有哪些 -->
<property>
<name>dfs.ha.namenodes.namenodecluster</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.namenodecluster.nn1</name>
<value>master:9000</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.namenodecluster.nn2</name>
<value>slave01:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.namenodecluster.nn1</name>
<value>master:50070</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.namenodecluster.nn2</name>
<value>slave01:50070</value>
</property>
<!-- 指定NameNode元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://master:8485;slave01:8485;slave02:8485/namenodecluster</value>
</property>
<!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!-- 使用隔离机制时需要ssh无秘钥登录-->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 声明journalnode服务器存储目录-->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/modules/ha/hadoop-2.7.2/data/jn</value>
</property>
<!-- 关闭权限检查-->
<property>
<name>dfs.permissions.enable</name>
<value>false</value>
</property>
<!-- 访问代理类：client，mycluster，active配置失败自动切换实现方式-->
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
</configuration>5. Distribute the set files to other nodes
xsync /opt/modules/ha2.2.5 Start HDFS HA cluster
1. First start the journalnode service:
sbin/hadoop-daemons.sh start journalnode
2. Format and start the NameNode on nn1:
bin/hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
3. Synchronize the metadata information of nn1 on nn2 and start it. It should be noted here that the NameNode can only be formatted once!
bin/hdfs namenode -bootstrapStandby
sbin/hadoop-daemon.sh start namenode4. Browser access for testing
5. Start all DataNodes on nn1
sbin/hadoop-daemons.sh start datanode6. Change the NameNode of nn1 to Active
bin/hdfs haadmin -transitionToActive nn1Refresh the browser again:
You can also use the command to check whether nn1 is Active:
bin/hdfs haadmin -getServiceState nn1
2.2.6 Configure automatic failover
1. Modify the hdfs-site.xml configuration file and add the following content below:
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>2. Add the following content to the core-site.xml file:
<property>
<name>ha.zookeeper.quorum</name>
<value>master:2181,slave01:2181,slave02:2181</value>
</property>Don't forget to distribute the modified files to the rest of the machines after configuration.
3. Delete the data and logs files in the directory.
rm -rf ./data ./logs
4. Start the service
(1) First start the Zookeeper cluster and journalnode service.
bin/zkServer.sh start
sbin/hadoop-daemons.sh start journalnode(2) Format the NameNode. Similarly, format it on only one machine.
bin/hdfs namenode -format(3) Initialize the HA state in Zookeeper. It is worth noting that it only needs to be executed on one of the machines .
bin/hdfs zkfc -formatZK
Here, it is equivalent to creating a new Node in Zookeeper, as shown below:
(4) Start the HDFS service
sbin/start-dfs.sh(5) Execute the following command on nn2 to synchronize the metadata information of nn1
bin/hdfs namenode -bootstrapStandby(6) Start the NameNode of nn2 separately
sbin/hadoop-daemon.sh start namenode5. Test verification
Currently, the NameNode of the master node is in the Active state. Now we manually kill the NameNode of the master node and find that the NameNode on the slave01 node is automatically converted from Standby to Active.
Three, Yarn HA
3.1 Planning the cluster
| hadoop102 |
hadoop103 |
hadoop104 |
| NameNode |
NameNode |
|
| JournalNode |
JournalNode |
JournalNode |
| DataNode |
DataNode |
DataNode |
| ZK |
ZK |
ZK |
| ResourceManager |
ResourceManager |
|
| NodeManager |
NodeManager |
NodeManager |
3.2 Configure Yarn HA
1. Add the following configuration in yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--启用resourcemanager ha-->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!--声明两台resourcemanager的地址-->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster-yarn1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>slave01</value>
</property>
<!--指定zookeeper集群的地址-->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>master:2181,slave01:2181,slave02:2181</value>
</property>
<!--启用自动恢复-->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<!--指定resourcemanager的状态信息存储在zookeeper集群-->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
</configuration>2. Synchronize the modified content to the remaining nodes.
3. Start the service
(1) Start Zookeeper
bin/zkServer.sh start(2) Start hdfs related services
sbin/start-dfs.sh(3) Start the yarn service
sbin/start-yarn.sh
At this time, you need to start ResourceManager on another machine.
sbin/yarn-daemon.sh start resourcemanager4. Check status
bin/yarn rmadmin -getServiceState rm1
Through the page access, no matter if the input is master or slave01, it will jump to the node in the active state, here is the master:
This is the end of this article. If you have any problems during this process, please leave a message and let me see what problems you have encountered~