zookeeper - distributed service coordination framework

1. Overview of Zookeeper
2. Installation and deployment of Zookeeper
3. The internal implementation principle of zookeeper
4. Zookeeper client command line operation
- 1. Command line syntax
- 2. Basic command line operations
5. Construction of high-availability HA-Hadoop cluster
6. YARN-HA configuration
- Configure YARN-HA cluster
7. How to use MR program to count words in a high-availability environment

1. Overview of Zookeeper

1. Basic concepts of Zookeeper

Zookeeper is an open source distributed Apache project that provides coordination services for distributed applications.
Zookeeper is understood from the perspective of design pattern: it is a distributed service management framework designed based on the observer pattern. It is responsible for storing and managing data that everyone cares about, and then accepts the registration of observers. Once the status of these data changes, Zookeeper It will be responsible for notifying those observers registered on Zookeeper to respond accordingly, thereby achieving a similar Master/Slave management model in the cluster.
Zookeeper = file system + notification mechanism

2. Features of Zookeeper

Zookeeper: a leader and a cluster of followers.
Leader is responsible for initiating and making decisions on voting and updating the system status.
Follower is used to receive customer requests and return results to the client, and participate in voting during the election of Leader.
As long as more than half of the nodes in the group survive, the Zookeeper cluster can serve normally.
Global data consistency: Each server saves a copy of the same data. No matter which server the client connects to, the data is consistent.
Update requests are performed sequentially, and update requests from the same client are executed sequentially in the order they are sent.
Data update is atomic, a data update either succeeds or fails.
Real-time, within a certain time range, the client can read the latest data.

3. Zookeeper data structure

The structure of the ZooKeeper data model is very similar to the Unix file system. It can be regarded as a tree as a whole, and each node is called a ZNode.
The Zookeeper cluster itself maintains a set of data structures. This storage structure is a tree structure, and each node on it is called a "znode". Different from the nodes of the tree, the Znode reference method is a path reference, similar to the file path: /znode1/leaf1
Such a hierarchical structure allows each Znode node to have a unique path, which clearly isolates different information just like a namespace.
ZooKeeper nodes are maintained through a tree-like structure, and each node is marked and accessed through a path.
In addition, each node also has some of its own information, including: data, data length, creation time, modification time, etc.
From the characteristics of such a node that both contains data and is marked as a path table, it can be seen that a ZooKeeper node can be regarded as either a file or a directory, and it has the characteristics of both. Features. For ease of expression, we will use Znode to represent the ZooKeeper node in question in the future.
Each znode can store 1MB of data by default
A znode is created by the client. Its intrinsic relationship with the client that created it determines its existence. Generally, there are four types of nodes:
- PERSISTENT - Persistent node : After the client that created this node disconnects from the zookeeper service, this node will not be deleted (unless it is forced to be deleted using the API)
- PERSISTENT_SEQUENTIAL - Persistent sequential number node : When the client requests to create this node A, zookeeper will write a directory-wide unique number for this A node based on the zxid status of parent-znode (this number will only keep growing). When the client is disconnected from the zookeeper service, the node will not be deleted.
- EPHEMERAL-Temporary directory node : After the client that created this node disconnects from the zookeeper service, this node (and the involved child nodes) will be deleted.
- EPHEMERAL_SEQUENTIAL - Temporary sequential number directory node : When the client requests the creation of this node A, zookeeper will write a directory-wide unique number for this A node based on the zxid status of parent-znode (this number will only keep growing). When the client that created the node disconnects from the zookeeper service, the node is deleted.
- [Note] : No matter it is an EPHEMERAL or EPHEMERAL_SEQUENTIAL node type, the node will also be deleted after the zookeeper client terminates abnormally.

2. Installation and deployment of Zookeeper

1. Download Zookeeper

On the Zookeeper official website , select the version you need to download. The following is the version I downloaded.

2. Installation of Zookeeper

Installation and deployment in local mode (standalone mode)

Step1: Upload the downloaded compressed package to the specified directory of the virtual machine. I uploaded it to /opt/software/

Step2: Unzip the compressed package to the specified directory. I unzipped it to /opt/app/

tar -zxvf apache-zookeeper-3.8.2-bin.tar.gz -C /opt/app/

Step3: Rename the zookeeper folder

mv apache-zookeeper-3.8.2-bin/ zookeeper-3.8.2

Step4: Configure environment variables and source the configuration file to take effect

vim /etc/profile source /etc/profile

Step5: Enter /opt/app/zookeeper-3.8.2/confthe directory, rename the configuration file mv zoo_sample.cfg zoo.cfg, and then edit it. and /opt/app/zookeeper-3.8.2/create a new directorytouch zkData

Step6: Use the command zkServer.sh startto start zookeeper, use the command zkServer.sh statusto check the zookeeper status, and use the command netstat -untlpto check the port number.

Step7: Use the command zkCli.sh -server localhost:2181to enter the client

zkServer.sh stopStep8: Exit using command

Distributed (cluster mode cluster) installation and deployment

Cluster planning

在node1、node2、node3三个节点上部署Zookeeper。

First select the node1 node to decompress and install. The steps are the same as the local mode installation and deployment.
Modify the configuration file zoo.cfg

[root@node1 software]# vim /opt/app/zookeeper/conf/zoo.cfg
#修改dataDir数据目录
dataDir=/opt/module/zookeeper-3.8.2/zkData
#在文件最后增加如下配置
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

server.A=B:C:D。
A是一个数字，表示这个是第几号服务器；
B是这个服务器的ip地址；
C是这个服务器与集群中的Leader服务器交换信息的端口；
D是万一集群中的Leader服务器挂了，需要一个端口来重新进行选举，选出一个新的Leader，而这个端口就是用来执行选举时服务器相互通信的端口。
集群模式下配置一个文件myid，这个文件在dataDir目录下，这个文件里面有一个数据就是A的值，Zookeeper启动时读取此文件，拿到里面的数据与zoo.cfg里面的配置信息比较从而判断到底是哪个server。

/opt/app/zookeeper-3.8.2Create a folder under mkdir zkData, and then enter this directory to create a file touch myiddefining the number of the current host.

#在配置zoo.cfg的时候配置了server.1/2/3这个配置项中  数字123代表的就是第几号服务器
#其中这个数字必须在zookeeper的zkData的myid文件中定义  并且定义的时候必须和配置项对应的IP相互匹配
[root@node1 zookeeper]# touch /opt/app/zookeeper/zkData/myid
[root@node1 zookeeper]# vim /opt/app/zookeeper/zkData/myid
#文件中写入当前主机对应的数字 然后保存退出即可 例 node1节点的myid写入1   node2节点的myid写入：2   node3节点的myid写入：3

Copy the configured zookeeper to other machines

scp -r /opt/app/zookeeper-3.8.2/ root@node2:/opt/app/
scp -r /opt/app/zookeeper-3.8.2/ root@node3:/opt/app/
并分别修改myid文件中内容为2、3

Configure the environment variables required by zookeeper on node2 and node3 nodes.

Start zookeeper separately and check the status

3. The internal implementation principle of zookeeper

1. Election mechanism

Half mechanism (Paxos protocol): More than half of the machines in the cluster survive and the cluster is available. Therefore, zookeeper is suitable for installation on an odd number of machines.
Although Zookeeper does not specify master and slave in the configuration file. However, when zookeeper works, one node is the leader and the others are followers. The leader is temporarily generated through the internal election mechanism.

1. The election mechanism when Zookeeper is started for the first time

Zookeeper’s internal election mechanism
- Suppose there is a zookeeper cluster composed of five servers. Their IDs range from 1 to 5. At the same time, they are all newly started, that is, there is no historical data. The amount of data stored is the same. Assuming that these servers are started sequentially, their internal implementation process is as shown in the figure
  - Server 1 is started. At this time, only one server is started. There is no response to the reports it sends, so its election status is always in the LOOKING state.
  - Server 2 starts. It communicates with Server 1, which was started at the beginning, and exchanges its own election results with each other. Since both have no historical data, Server 2 with a larger ID value wins, but since more than half of the servers have not reached Agree to elect it (more than half of them are 3 in this example), so servers 1 and 2 continue to maintain the LOOKING state.
  - Server 3 starts. According to the previous theoretical analysis, server 3 becomes the leader among servers 1, 2, and 3. The difference from the above is that three servers elected it at this time, so it becomes the leader of this election.
  - Server 4 is started. According to the previous analysis, theoretically server 4 should be the largest among servers 1, 2, 3, and 4. However, since more than half of the previous servers have elected server 3, it can only receive the order of being the younger brother. .
  - Server 5 starts and acts as the younger brother like 4.

2. The election mechanism of zookeeper when it is not started for the first time

SID: Server ID. Used to uniquely identify a machine in the ZooKeeper cluster. Each machine cannot be repeated and is consistent with myid.
ZXID: Transaction ID. ZXID is a transaction ID used to identify a change in server status. At a certain moment, the ZXID value of each machine in the cluster may not be exactly the same. This is related to the ZooKeeper server's processing logic for the client's "update request".
Epoch: The code name of each Leader term. When there is no leader, the logical clock value in the same round of voting is the same. This data will increase every time a vote is cast.

zk election

2. Zookeeper writing data process

1. The process of writing the Leader node directly

write leader

2. Directly write the Follower process

write follower

3. Detailed overview of writing data process

For example, the Client writes data to Server1 of ZooKeeper and sends a write request.
If Server1 is not the Leader, then Server1 will further forward the received request to the Leader, because one of each ZooKeeper Server is the Leader. The Leader will broadcast the write request to each Server, such as Server1 and Server2. After each Server writes successfully The Leader will be notified.
When the Leader receives most of the Server data and writes it successfully, it means that the data is written successfully. If there are three nodes here, as long as the data of two nodes is written successfully, then the data is considered to be written successfully. After the writing is successful, Leader will tell Server1 that the data is written successfully.
Server1 will further notify the Client that the data is written successfully, and then the entire write operation is considered successful. The entire data writing process of ZooKeeper is like this.

4. Zookeeper client command line operation

Use the command to connect to the zookeeper clusterzkCli.sh -server node:2181,node2:2181,node3:2181

1. Command line syntax

Basic command syntax	Function description
help	Show all operation commands
ls path [watch]	Use the ls command to view the content contained in the current znode
ls -s path [watch]	View current node information
create [-e] [-s]	Create node -s with sequence -e temporary (restart or disappear after timeout)
get path [watch]	Get the value of a node
set	Set the specific value of the node
stat	View node status
delete	Delete node
rmr/deleteall	Recursively delete nodes

2. Basic command line operations

Start the command line client

zkCli.sh -server node1:2181,node2:2181,node3:2181

Show all operation commands
```
help
```
View znode node information
```
ls / 
```

View detailed information about a node in znode

[zk: node1:2181(CONNECTED) 5] ls -s / 
[zookeeper]cZxid = 0x0 
ctime = Thu Jan 01 08:00:00 CST 1970 
mZxid = 0x0 
mtime = Thu Jan 01 08:00:00 CST 1970 
pZxid = 0x0 
cversion = -1 
dataVersion = 0 
aclVersion = 0 
ephemeralOwner = 0x0 
dataLength = 0 
numChildren = 1 

 （1）czxid：创建节点的事务 zxid 每次修改 ZooKeeper 状态都会产生一个 ZooKeeper 事务 ID。事务 ID 是 ZooKeeper 中所 有修改总的次序。每次修改都有唯一的 zxid，如果 zxid1 小于 zxid2，那么 zxid1 在 zxid2 之前发生。 
 （2）ctime：znode 被创建的毫秒数（从 1970 年开始） 
 （3）mzxid：znode 最后更新的事务 zxid 
 （4）mtime：znode 最后修改的毫秒数（从 1970 年开始） 
 （5）pZxid：znode 最后更新的子节点 zxid
 （6）cversion：znode 子节点变化号，znode 子节点修改次数 
 （7）dataversion：znode 数据变化号 
 （8）aclVersion：znode 访问控制列表的变化号 
 （9）ephemeralOwner：如果是临时节点，这个是 znode 拥有者的 session id。如果不是 临时节点则是 0。 
 （10）dataLength：znode 的数据长度 
 （11）numChildren：znode 子节点数量

Create a normal node (permanent node + without serial number)
```
create /sanguo "weishuwu"
```
Get the value of a node
```
get -s /test 
```

Create a node with a serial number (permanent node + with a serial number)

create  -s  /a

create  -s  /a 

create  /a 

如果原来没有序号节点，序号从 0 开始依次递增。如果原节点下已有 2 个节点，则再排序时从 2 开始，以此类推。

Create a transient node (ephemeral node + without serial number or with serial number)

 （1）创建短暂的不带序号的节点 
create -e /b

 （2）创建短暂的带序号的节点 
create -e -s /b

 （3）在当前客户端是能查看到的 
ls /

 （4）退出当前客户端然后再重启客户端 
 [zk: node1:2181(CONNECTED) 12] quit 
 [root@node1 zookeeper-3.5.7]$ bin/zkCli.sh 
 （5）再次查看根目录下短暂节点已经删除 
ls /

Modify node data value

[zk: node1:2181(CONNECTED) 6] set /sanguo/weiguo "simayi"

Delete node
```
delete /test
```
Recursively delete nodes
```
deleteall /test
```
View node status
```
stat /sanguo 
```
13. Monitor data changes of nodes
```
get -w /sanguo
```
14. Listen for changes in the child nodes of a node
```
ls -w /sanguo
```

5. Construction of high-availability HA-Hadoop cluster

1. Overview of high availability HA

The so-called HA (high available) means high availability (uninterrupted service 7*24 hours).
The most critical strategy for achieving high availability is to eliminate single points of failure. Strictly speaking, HA should be divided into HA mechanisms for each component: HDFS HA and YARN HA.
Before Hadoop2.0, NameNode had a single point of failure (SPOF) in the HDFS cluster.
NameNode mainly affects the HDFS cluster in the following two aspects:
- If an accident occurs to the NameNode machine, such as downtime, the cluster will be unusable until the administrator restarts it.
- The NameNode machine needs to be upgraded, including software and hardware upgrades, and the cluster will not be usable at this time.
The HDFS HA function solves the above problems by configuring two nameNodes, Active/Standby, to implement hot backup of the NameNode in the cluster. If a failure occurs, such as a machine crash or the machine needs to be upgraded and maintained, the NameNode can be quickly switched to another machine in this way.

2. HDFS-HA working mechanism: eliminate single points of failure through dual namenodes

HDFS-HA work points

Metadata management methods need to be changed (SecondaryNameNode is not required)

内存中各自保存一份元数据；
Edits日志只有Active状态的namenode节点可以做写操作；
两个namenode都可以读取edits；
共享的edits放在一个共享存储中管理（qjournal和NFS两个主流实现）；

A status management function module is required

实现了一个zkfailover，常驻在每一个namenode所在的节点，每一个zkfailover负责监控自己所在namenode节点，利用zk进行状态标识，当需要进行状态切换时，由zkfailover来负责切换，切换时需要防止brain split现象的发生。

It must be ensured that SSH password-less login is possible between the two NameNodes.
Isolation (Fence) means that only one NameNode provides external services at the same time.

3. HDFS-HA cluster configuration

Environment preparation:
- Modify IP
- Modify the host name and the mapping between host name and IP address
- Turn off firewall
- ssh password-free login
- Install JDK, configure environment variables, etc.
Planning a cluster

node1	node2	node3
NameNode	NameNode	-
JournalNode	JournalNode	JournalNode
DataNode	DataNode	DataNode
ZK	ZK	ZK
ResourceManager
NodeManager	NodeManager	NodeManager

Configuring the Zookeeper cluster: Already recorded in the notes above!

Configure HDFS-HA cluster:

Configure hadoop-env.sh
```
export JAVA_HOME=/opt/app/jdk
```

Configure core-site.xml

<configuration>
<!-- 把两个NameNode的地址组装成一个集群HadoopCluster -->
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://HC</value>
      </property>
      <!-- 指定hadoop运行时产生文件的存储目录 -->
      <property>
          <name>hadoop.tmp.dir</name>
          <value>/opt/app/hadoop-3.1.4/metaData</value>
      </property>
      <!--配置连接的zookeeper的地址-->
      <property>
          <name>ha.zookeeper.quorum</name>
          <value>node1:2181,node2:2181,node3:2181</value>
      </property>
</configuration>

Configure hdfs-site.xml

<configuration>
      <!-- 完全分布式集群名称 -->
  <property>
      <name>dfs.nameservices</name>
      <value>HC</value>
  </property>
  <!-- 集群中NameNode节点都有哪些 -->
  <property>
      <name>dfs.ha.namenodes.HC</name>
      <value>nn1,nn2</value>
  </property>
  <!-- nn1的RPC通信地址 -->
  <property>
      <name>dfs.namenode.rpc-address.HC.nn1</name>
      <value>node1:9000</value>
  </property>
  <!-- nn2的RPC通信地址 -->
  <property>
      <name>dfs.namenode.rpc-address.HC.nn2</name>
      <value>node2:9000</value>
  </property>
  <!-- nn1的http通信地址 -->
  <property>
      <name>dfs.namenode.http-address.HC.nn1</name>
      <value>node1:9870</value>
  </property>
  <!-- nn2的http通信地址 -->
  <property>
      <name>dfs.namenode.http-address.HC.nn2</name>
      <value>node2:9870</value>
  </property>
  <!-- 指定NameNode元数据在JournalNode上的存放位置 -->
  <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://node1:8485;node2:8485;node3:8485/HadoopCluster</value>
  </property>
  <!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
  <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
  </property>
  <!-- 使用隔离机制时需要ssh无秘钥登录-->
  <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/root/.ssh/id_rsa</value>
  </property>
  <!-- 声明journalnode服务器存储目录-->
  <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/opt/app/hadoop-3.1.4/journalnodeData</value>
  </property>
  <!-- 关闭权限检查-->
  <property>
      <name>dfs.permissions.enable</name>
      <value>false</value>
  </property>
  <!-- 访问代理类：client，HadoopCluster，active配置失败自动切换实现方式-->
  <property>
        <name>dfs.client.failover.proxy.provider.HC</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
      <property>
          <name>dfs.replication</name>
          <value>3</value>
      </property>
      <property>
          <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
          <value>true</value>
      </property>
    <property>
      <name>dfs.ha.automatic-failover.enabled</name>
      <value>true</value>
    </property>
 </configuration>

Copy the configured hadoop environment to other nodes

scp /opt/app/hadoop-3.1.4/etc/hadoop/core-site.xml root@node2:/opt/app/hadoop-3.1.4/etc/hadoop/
scp /opt/app/hadoop-3.1.4/etc/hadoop/core-site.xml root@node3:/opt/app/hadoop-3.1.4/etc/hadoop/
scp /opt/app/hadoop-3.1.4/etc/hadoop/hdfs-site.xml root@node3:/opt/app/hadoop-3.1.4/etc/hadoop/
scp /opt/app/hadoop-3.1.4/etc/hadoop/hdfs-site.xml root@node2:/opt/app/hadoop-3.1.4/etc/hadoop/

Start HDFS-HA cluster
- Install psmisc software
The automatic failover of zkfc needs to be completed with the help of psmisc software, so this software needs to be installed on three nodes.
```
yum install -y psmisc
```
- On each JournalNode node, enter the following command to start the journalnode service:
  sbin/hadoop-daemon.sh start journalnode
- On [nn1], format it, and start:
```
  rm -rf metaData/ journalnodeData/  删除三台节点
 /opt/app/hadoop-3.1.4   hdfs namenode -format  只需要在第一台节点格式化
                         hadoop-daemon.sh start namenode    只需要执行一次即可，之后就不需要再执行
```
- Encountered an error, as shown in the figure
  - vim /opt/app/hadoop-3.1.4/etc/hadoop/hadoop-env.sh
  - scp /opt/app/hadoop-3.1.4/etc/hadoop/hadoop-env.sh root@node2:/opt/app/hadoop-3.1.4/etc/hadoop/
  - scp /opt/app/hadoop-3.1.4/etc/hadoop/hadoop-env.sh root@node3:/opt/app/hadoop-3.1.4/etc/hadoop/
  - start upstart-dfs.sh
- On [nn2], synchronize the metadata information of nn1:
  bin/hdfs namenode -bootstrapStandby it only needs to be executed once, and there is no need to execute it again;
  
  hadoop-daemon.sh start namenodeand start namenode on the second node
- Start datanode on three nodeshadoop-daemon.sh start datanode
- Restart HDFS
  - Shut down all HDFS services: sbin/stop-dfs.sh
  - Start the Zookeeper cluster: bin/zkServer.sh start
  - Initialize HA status in Zookeeper: bin/hdfs zkfc -formatZK
  - Start HDFS service: sbin/start-dfs.sh
  - Start the DFSZK Failover Controller on each NameNode node. Which machine is started first? The NameNode of which machine is the Active NameNode: sbin/hadoop-daemin.sh start zkfc
- verify
  - Kill the Active NameNode process:kill -9 the process id of the namenode
  - Disconnect the Active NameNode machine from the network: service network stop

6. YARN-HA configuration

Configure YARN-HA cluster

Environmental preparation
- Modify IP
- Modify the host name and the mapping between host name and IP address
- Turn off firewall
- ssh password-free login
- Install JDK, configure environment variables, etc.
- Configure Zookeeper cluster
Planning a cluster

node1	node2	node3
NameNode	NameNode
JournalNode	JournalNode	JournalNode
DataNode	DataNode	DataNode
ZK	ZK	ZK
ResourceManager	ResourceManager
NodeManager	NodeManager	NodeManager

Specific configuration - configure on each node

yarn-site.xml

<configuration>
  <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
  </property>
  <!--启用resourcemanager ha-->
  <property>
      <name>yarn.resourcemanager.ha.enabled</name>
      <value>true</value>
  </property>
  <!--声明两台resourcemanager的地址-->
  <property>
      <name>yarn.resourcemanager.cluster-id</name>
      <value>cluster-yarn1</value>
  </property>
  <property>
      <name>yarn.resourcemanager.ha.rm-ids</name>
      <value>rm1,rm2</value>
  </property>
  <property>
      <name>yarn.resourcemanager.hostname.rm1</name>
      <value>node1</value>
  </property>
  <property>
      <name>yarn.resourcemanager.hostname.rm2</name>
      <value>node2</value>
  </property>
  <!--指定zookeeper集群的地址--> 
  <property>
      <name>yarn.resourcemanager.zk-address</name>
      <value>node1:2181,node2:2181,node3:2181</value>
  </property>
  <!--启用自动恢复--> 
  <property>
      <name>yarn.resourcemanager.recovery.enabled</name>
      <value>true</value>
  </property>
  <!--指定resourcemanager的状态信息存储在zookeeper集群--> 
  <property>
      <name>yarn.resourcemanager.store.class</name>     
      <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  </property>
</configuration>

Synchronously update the configuration information of other nodes

scp /opt/app/hadoop-3.1.4/etc/hadoop/yarn-site.xml root@node2:/opt/app/hadoop-3.1.4/etc/hadoop/
scp /opt/app/hadoop-3.1.4/etc/hadoop/yarn-site.xml root@node3:/opt/app/hadoop-3.1.4/etc/hadoop/

Start hdfs (you don’t need to do this step if you have built an HA-Hadoop cluster)
- On each JournalNode node, enter the following command to start the journalnode service: sbin/hadoop-daemon.sh start journalnode
- On [nn1], format it, and start:
```
bin/hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
```
- On [nn2], synchronize the metadata information of nn1: bin/hdfs namenode -bootstrapStandby
- Start [nn2]: sbin/hadoop-daemon.sh start namenode
- Start all datanodes: sbin/hadoop-daemons.sh start datanode
- Switch [nn1] to Active: bin/hdfs haadmin -transitionToActive nn1
Start yarn
- Execute in node1: sbin/start-yarn.sh
- Execute in node2: sbin/yarn-daemon.sh start resourcemanager
- Check the service status: bin/yarn rmadmin -getServiceState rm1

7. How to use MR program to count words in a high-availability environment

Edit wc.txt and upload it to hdfs

Open idea, create a maven project, and introduce programming dependencies in pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.kang</groupId>
  <artifactId>ha-test</artifactId>
  <version>1.0</version>
  <packaging>jar</packaging>

  <name>ha-test</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>3.1.4</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>3.1.4</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>1.6.1</version>
    </dependency>
  </dependencies>
</project>

Writing MapReduce code

package com.kang;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WCMapper extends Mapper<LongWritable, Text,Text,LongWritable> {
    
    
    @Override
    protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, LongWritable>.Context context) throws IOException, InterruptedException {
    
    
        String line = value.toString();
        String[] words = line.split(" ");
        for (String word : words) {
    
    
            context.write(new Text(word),new LongWritable(1L));
        }
    }
}


package com.kang;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class WCReduce extends Reducer<Text, LongWritable,Text,LongWritable> {
    
    
    @Override
    protected void reduce(Text key, Iterable<LongWritable> values, Reducer<Text, LongWritable, Text, LongWritable>.Context context) throws IOException, InterruptedException {
    
    
        long sum = 0L;
        for (LongWritable value : values) {
    
    
            sum += value.get();
        }
        context.write(key,new LongWritable(sum));
    }
}



package com.kang;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import javax.xml.soap.Text;
import java.io.IOException;

public class WCDriver {
    
    
    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
    
    
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","hdfs://HC");

        Job job = Job.getInstance(conf);
        job.setJarByClass(WCDriver.class);
        FileInputFormat.setInputPaths(job,new Path("/wc.txt"));

        job.setMapperClass(WCMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);

        job.setReducerClass(WCReduce.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);
        job.setNumReduceTasks(0);

        FileOutputFormat.setOutputPath(job,new Path("/output"));

        boolean flag = job.waitForCompletion(true);
        System.exit(flag?0:1);
    }
}

Export hdfs-site.xml and core-site.xml from the virtual machine to the Java project directory

Then run WCDriver, you will receive an error message and it will show insufficient permissions. Then in order to achieve this task, we modify the permissions.

Run the program again and return code 0, which means the operation is successful.

Finally, we restore the default permissions of files and directories in the Hadoop Distributed File System (HDFS) to the default permission types