Shang Silicon Valley Big Data - Build Hadoop Cluster - Software Installation

Enterprise 2023-06-05 00:31:29 views: null

1. Preliminary preparation

1. Install JDK on hadoop102, and use root account to log in to the system

Note: Before installing JDK, make sure to delete the JDK that comes with the virtual machine in advance

# 查看系统自带的java版本
java -version
# 查找JDK相关包是否被安装
rpm -qa |grep jdk
# 使用yum -y remove 命令卸载已经安装的jdk相关包
yum -y remove copy-jdk-configs-3.3-2.el7.noarch
yum -y remove java-1.8.0-openjdk-headless-1.8.0.161-2.b14.el7.x86_64
yum -y remove java-1.7.0-openjdk-1.7.0.171-2.6.13.2.el7.x86_64
yum -y remove java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
yum -y remove java-1.7.0-openjdk-headless-1.7.0.171-2.6.13.2.el7.x86_64
# 再次查找JDK相关包是否被安装, 结果为空, 表示全部卸载
rpm -qa |grep jdk
# 再次查看系统自带的java版本
java -version

insert image description here
insert image description here

install new jdk

  1. new directory
mkdir /opt/module
mkdir /opt/software
  1. Upload the installation package to the /opt/software directory
    insert image description here
  2. Unzip the installation package
# 进入/opt/software目录
cd /opt/software
# 解压jdk到/opt/module/目录下
tar -zxvf jdk-8u212-linux-x64.tar.gz -C /opt/module/
  1. Configure JDK environment variables
# 进入/etc/profile.d/目录下
cd /etc/profile.d/

# 在/etc/profile.d/文件下新建一个my_env.sh文件, 里面用来存放自己安装软件的环境变量信息
vim /etc/profile.d/my_env.sh

# 在/etc/profile.d/my_env.sh中添加java环境变量信息
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_212
export PATH=$PATH:$JAVA_HOME/bin

# source一下/etc/profile文件，让新的环境变量PATH生效
source /etc/profile	

# 查看是否安装成功, 如果出现版本号就说明安装成功
java -version

# 如果java -version命令没有出现版本号, 就需要重启系统
reboot

Why do you want to create a new my_env.sh under /etc/profile.d?
There is one in the Linux system configuration file /etc/profile

[root@hadoop102 software]# cat /etc/profile
# /etc/profile

# System wide environment and startup programs, for login setup
# Functions and aliases go in /etc/bashrc

# It's NOT a good idea to change this file unless you know what you
# are doing. It's much better to create a custom.sh shell script in
# /etc/profile.d/ to make custom changes to your environment, as this
# will prevent the need for merging in future updates.

pathmunge () {
    
    
    case ":${
     
     PATH}:" in
        *:"$1":*)
            ;;
        *)
            if [ "$2" = "after" ] ; then
                PATH=$PATH:$1
            else
                PATH=$1:$PATH
            fi
    esac
}


if [ -x /usr/bin/id ]; then
    if [ -z "$EUID" ]; then
        # ksh workaround
        EUID=`/usr/bin/id -u`
        UID=`/usr/bin/id -ru`
    fi
    USER="`/usr/bin/id -un`"
    LOGNAME=$USER
    MAIL="/var/spool/mail/$USER"
fi

# Path manipulation
if [ "$EUID" = "0" ]; then
    pathmunge /usr/sbin
    pathmunge /usr/local/sbin
else
    pathmunge /usr/local/sbin after
    pathmunge /usr/sbin after
fi

HOSTNAME=`/usr/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL=ignoredups
fi

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL

# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
    umask 002
else
    umask 022
fi

for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
    if [ -r "$i" ]; then
        if [ "${-#*i}" != "$-" ]; then 
            . "$i"
        else
            . "$i" >/dev/null
        fi
    fi
done

unset i
unset -f pathmunge
[root@hadoop102 software]#

2. Install Hadoop on hadoop102, log in to the system with root account

install hadoop

  1. new directory
mkdir /opt/module
mkdir /opt/software
  1. Upload the installation package to the /opt/software directory
    insert image description here
  2. Unzip the installation package
# 进入/opt/software目录
cd /opt/software
# 解压jdk到/opt/module/目录下
tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/
  1. Configure Hadoop environment variables
# 进入/etc/profile.d/目录下
cd /etc/profile.d/

# 在/etc/profile.d/文件下新建一个my_env.sh文件, 里面用来存放自己安装软件的环境变量信息
vim /etc/profile.d/my_env.sh

# 在/etc/profile.d/my_env.sh中添加Hadoop环境变量信息
#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

# source一下/etc/profile文件，让新的环境变量PATH生效
source /etc/profile	

# 查看是否安装成功, 如果出现版本号就说明安装成功
hadoop version

# 如果java -version命令没有出现版本号, 就需要重启系统
reboot

hadoop directory structure
(1) bin directory: store scripts for operating Hadoop-related services (hdfs, yarn, mapred)
(2) etc directory: Hadoop configuration file directory, store Hadoop configuration files
(3) lib directory: store Hadoop local library (compression and decompression function for data)
(4) sbin directory: store scripts to start or stop Hadoop-related services
(5) share directory: store Hadoop dependent jar packages, documents, and official cases
insert image description here

3. Cluster distribution script

Copy the installation files of JDK and hadoop from hadoop102 to hadoop103 and hadoop104

3.1 scp (secure copy) safe copy

Prerequisite: Prepare three Linux machines, configure static ip address, configure host name, and configure hosts file for host name mapping
insert image description here
Prerequisite: Two directories /opt/module and /opt/software have been created in hadoop102, hadoop103 and hadoop104

chown root:root -R /opt/module

scp   -r      $user@$host:$pdir/$fname               $user@$host:$pdir/$fname
命令  递归    来源地登录用户名@主机:来源地路径/名称     目的地登录用户名@主机:目的地路径/名称
如果不填写 $user@$host 则默认是本机
  1. On hadoop102, copy the /opt/module/jdk1.8.0_212 directory in hadoop102 to hadoop103.
scp -r /opt/module/jdk1.8.0_212  root@hadoop103:/opt/module
  1. On hadoop103, copy the /opt/module/hadoop-3.1.3 directory in hadoop102 to hadoop103
scp -r root@hadoop102:/opt/module/hadoop-3.1.3 /opt/module/
  1. Operate on hadoop103, copy all directories under the /opt/module directory in hadoop102 to hadoop104
scp -r root@hadoop102:/opt/module/* root@hadoop104:/opt/module

3.2 rsync remote synchronization tool

Using rsync to copy files is faster than scp, and rsync only updates the difference files. scp copies all files

rsync   -av      $user@$host:$pdir/$fname      $user@$host:$pdir/$fname
命令   选项参数   要拷贝的用户@主机:文件路径/名称   目的地用户@主机:目的地路径/名称
  1. Delete /opt/module/hadoop-3.1.3/wcinput in hadoop103
rm  -rf  /opt/module/hadoop-3.1.3/wcinput
  1. On hadoop102, synchronize /opt/module/hadoop-3.1.3 in hadoop102 to hadoop103
rsync -av hadoop-3.1.3/ root@hadoop103:/opt/module/hadoop-3.1.3/

3.3 xsync cluster distribution script

  1. Create a file
cd /bin
vim xsync
  1. Add the following script in xsync
#!/bin/bash

#1. 判断参数个数
if [ $# -lt 1 ]
then
    echo Not Enough Arguement!
    exit; #退出
fi # if结束

#2. 遍历集群所有机器
for host in hadoop102 hadoop103 hadoop104
do
    echo ====================  $host  ====================
    #3. 遍历所有目录，挨个发送
    # $@表示传递给函数或脚本的所有参数
    for file in $@ 
    do
        #4. 判断文件是否存在
        if [ -e $file ]
            then
                #5. 获取父目录
                pdir=$(cd -P $(dirname $file); pwd)

                #6. 获取当前文件的名称
                fname=$(basename $file)
				# $host是上面for循环的临时变量, ssh命令连接远程主机
                ssh $host "mkdir -p $pdir" 
                rsync -av $pdir/$fname $host:$pdir
            else
                echo $file does not exists!
        fi
    done
done
  1. Modify the script xsync to have execution permission
chmod +x xsync
或者
chmod 777 xsync
  1. Synchronize the /etc/profile.d/my_env.sh file of hadoop102 to hadoop103 and hadoop104.
    There will be an extra my_env.sh file in the /etc/profile.d/ directory of hadoop103 and hadoop104
# xsync命令 路径要写全
/bin/xsync /etc/profile.d/my_env.sh

insert image description here
insert image description here

  1. Make the environment variables of hadoop103 and hadoop104 take effect
[root@hadoop103 ~]# source /etc/profile
[root@hadoop104 ~]# source /etc/profile
  1. Test whether the environment variable is valid
[root@hadoop103 ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
[root@hadoop103 ~]# hadoop version
Hadoop 3.1.3
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579
Compiled by ztang on 2019-09-12T02:47Z
Compiled with protoc 2.5.0
From source with checksum ec785077c385118ac91aadde5ec9799
This command was run using /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-common-3.1.3.jar
[root@hadoop103 ~]#

[root@hadoop104 ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

[root@hadoop104 ~]# hadoop version
Hadoop 3.1.3
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579
Compiled by ztang on 2019-09-12T02:47Z
Compiled with protoc 2.5.0
From source with checksum ec785077c385118ac91aadde5ec9799
This command was run using /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-common-3.1.3.jar
[root@hadoop104 ~]#

4. SSH passwordless login configuration

normal ssh syntax

ssh hadoop103
# 输入密码后连接到 hadoop103
  1. Password-free login principle
    insert image description here
  2. To generate the public key and private key,
    all three machines need to perform the following operations, so that any machine can log in to other machines without secrets. Here I use the root user
# 进入/root目录下
cd /root
# 生成公钥和私钥, 执行命令后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）
ssh-keygen -t rsa
# 进入/root/.ssh查看
cd /root/.ssh

# 将公钥拷贝到要免密登录的目标机器上, 输入 yes , 输入对应机器的密码
ssh-copy-id hadoop102
ssh-copy-id hadoop103
ssh-copy-id hadoop104

# 测试, 无需输入密码
ssh hadoop102
ssh hadoop103
ssh hadoop104

Function description of each file in the /root/.ssh directory

file name illustrate
known_hosts Record the public key (public key) of the computer that ssh has accessed
id_rsa generated private key
id_rsa.pub generated public key
authorized_keys Store the authorized non-secret login server public key

2. Cluster configuration

1. Cluster deployment planning instructions

 NameNode and SecondaryNameNode should not be installed on the same server
.  ResourceManager also consumes a lot of memory. Do not configure it on the same machine as NameNode and SecondaryNameNode.

hadoop102 hadoop103 hadoop104
HDFS NameNode、DataNode DataNode SecondaryNameNode 、DataNode
YARN NodeManager ResourceManager、NodeManager NodeManager

2. Description of the cluster configuration file

Hadoop configuration files are divided into two categories: default configuration files and custom configuration files. Only when users want to modify a default configuration value, they need to modify the custom configuration file and change the corresponding attribute value.

default file to fetch The location where the file is stored in the Hadoop jar package
[core-default.xml] hadoop-common-3.1.3.jar/core-default.xml
[hdfs-default.xml] hadoop-hdfs-3.1.3.jar/hdfs-default.xml
[yarn-default.xml] hadoop-yarn-common-3.1.3.jar/yarn-default.xml
[mapred-default.xml] hadoop-mapreduce-client-core-3.1.3.jar/mapred-default.xml

Custom configuration files:
core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml four configuration files are stored in the $HADOOP_HOME/etc/hadoop path, users can re- Modify the configuration

3. Configure the cluster configuration file

3.1 Core configuration file: core-site.xml

cd $HADOOP_HOME/etc/hadoop
vim core-site.xml

The content in the default configuration tag is empty, we add the following content

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!-- 指定NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop102:8020</value>
    </property>

    <!-- 指定hadoop数据的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop-3.1.3/data</value>
    </property>

    <!-- 配置HDFS网页登录使用的静态用户为atguigu -->
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>atguigu</value>
    </property>
</configuration>

3.2 HDFS configuration file: hdfs-site.xml

cd $HADOOP_HOME/etc/hadoop
vim hdfs-site.xml

The content in the default configuration tag is empty, we add the following content

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
	<!-- nn web端访问地址-->
	<property>
        <name>dfs.namenode.http-address</name>
        <value>hadoop102:9870</value>
    </property>
	<!-- 2nn web端访问地址-->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop104:9868</value>
    </property>
</configuration>

3.3 YARN configuration file: yarn-site.xml

cd $HADOOP_HOME/etc/hadoop
vim yarn-site.xml

The content in the default configuration tag is empty, we add the following content

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!-- 指定MR走shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 指定ResourceManager的地址-->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop103</value>
    </property>

    <!-- 环境变量的继承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

3.4 MapReduce configuration file: mapred-site.xml

cd $HADOOP_HOME/etc/hadoop
vim mapred-site.xml

The content in the default configuration tag is empty, we add the following content

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<!-- 指定MapReduce程序运行在Yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

3.5 Distribute Hadoop configuration files configured on hadoop102 on the cluster

/bin/xsync /opt/module/hadoop-3.1.3/etc/hadoop/

Go to 103 and 104 to check the file distribution

[root@hadoop103 hadoop]# cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml
[root@hadoop104 hadoop]# cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml

4. Start the cluster

4.1 Configure workers

vim /opt/module/hadoop-3.1.3/etc/hadoop/workers

Add the following content to the file, there are several nodes on the cluster, here are a few configurations

hadoop102
hadoop103
hadoop104

Synchronize configuration files to other nodes

/bin/xsync /opt/module/hadoop-3.1.3/etc/hadoop/

4.2 Start HDFS on hadoop102

cd /opt/module/hadoop-3.1.3
# 如果集群是第一次启动，需要在hadoop102节点格式化NameNode
hdfs namenode -format
# 启动HDFS
sbin/start-dfs.sh
# 启动报错, 重新启动之前需要关闭HDFS
sbin/stop-dfs.sh

Use the jsp command to test the HDFS startup result

# hadoop102中启动了 NameNode 与 DataNode
[root@hadoop102 hadoop-3.1.3]# jps
9202 DataNode
9364 Jps
9048 NameNode

# hadoop103中只启动 DataNode
[root@hadoop103 hadoop-3.1.3]# jps
8649 DataNode
8733 Jps

# hadoop102中启动了 SecondaryNameNode 与 DataNode
[root@hadoop104 hadoop-3.1.3]# jps
9427 Jps
9379 SecondaryNameNode
9274 DataNode

The following error was reported when it was started for the first time. Solution: Use the root user to start hadoop-3.1.3 and report an error

[root@hadoop102 hadoop-3.1.3]# sbin/start-dfs.sh
Starting namenodes on [hadoop102]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hadoop104]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

The following situation occurs when starting HDFS. Solution: The configuration parameters of the new version of Hadoop have been updated, causing the startup to fail.

[root@hadoop102 hadoop-3.1.3]# sbin/start-dfs.sh
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [hadoop102]
上一次登录：三 9月 28 15:39:17 CST 2022从 hadoop104pts/1 上
Starting datanodes
上一次登录：三 9月 28 18:37:26 CST 2022pts/0 上

4.3 Start YARN on hadoop103

cd /opt/module/hadoop-3.1.3
# 启动 YARN
sbin/start-yarn.sh
# 关闭 YARN
sbin/stop-yarn.sh

Use the jps command to test the YARN startup result

# hadoop102的 HDFS 中启动了 NameNode 与 DataNode
# hadoop102的 YARN 中启动了 NodeManager
[root@hadoop102 hadoop-3.1.3]# jps
13161 NodeManager
13355 Jps
11789 NameNode
11967 DataNode


# hadoop103的 HDFS 中只启动 DataNode
# hadoop102的 YARN 中启动了 ResourceManager 与 NodeManager
[root@hadoop103 hadoop-3.1.3]# jps
11394 Jps
9254 DataNode
10814 ResourceManager
11150 NodeManager


# hadoop102的 HDFS 中启动了 SecondaryNameNode 与 DataNode
# hadoop102的 YARN 中启动了 NodeManager
[root@hadoop104 hadoop-3.1.3]# jps
9379 SecondaryNameNode
10617 Jps
9274 DataNode
10411 NodeManager

The following situation occurs when starting HDFS. Solution: Start start-yarn.sh and report an error ERROR: Attempting to operate on yarn resourcemanager as root ERROR: but there is no

[root@hadoop103 hadoop-3.1.3]# sbin/start-yarn.sh
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
上一次登录：三 9月 28 15:39:32 CST 2022从 hadoop104pts/2 上
^C

4.4 View cluster information on the web

Close the firewall of hadoop102, hadoop03, hadoop104

systemctl stop firewalld

Enter in the browser: http://hadoop102:9870
to view the data information stored on HDFS

insert image description here

Enter in the browser: http://hadoop103:8088
to view the job information running on YARN

insert image description here

5. Cluster basic test

5.1 Upload files to the cluster

Command: hadoop fs -put local file HDFS directory

# 在 HDFS 的根目录下创建一个 input文件夹
hadoop fs -mkdir /input
# 將 /hadoop-3.1.3/README.txt上传到HDFS中新建的 input 目录下
hadoop fs -put $HADOOP_HOME/README.txt /input
# 在 HDFS 隔壁目录下上传 jdk-8u212-linux-x64.tar.gz
hadoop fs -put  /opt/software/jdk-8u212-linux-x64.tar.gz  /

Refresh HDFS web page http://hadoop102:9870/explorer.html#/
Create new input directory
insert image description here
Uploaded README.txt file
insert image description here
Uploaded jdk-8u212-linux-x64.tar.gz
insert image description here

5.2 HDFS file storage location

Configured in core-site.xml earlier, the file is stored under /opt/module/hadoop-3.1.3/data/

# 在 hadoop102 hadoop103 hadoop104 做个三次备份
cd /opt/module/hadoop-3.1.3/data/dfs/data/current/BP-383546732-192.168.10.102-1664361440225/current/finalized/subdir0/subdir0
# 查看文件内容, 发现是刚刚上传的 README.txt 中的内容
cat blk_1073741825

insert image description here
insert image description here

5.3 Download files on HDFS to local

Command: hadoop fs -get local directory of files on HDFS

hadoop fs -get /jdk-8u212-linux-x64.tar.gz ./

5.4 Testing YARN

WordCount is Hadoop's default "Hello World program" - used to count words

cd /opt/module/hadoop-3.1.3/
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

Refresh the web page of HDFS http://hadoop102:9870/explorer.html#/
There is an output directory under the / directory of HDFS
Refresh the web page of YARN: http://hadoop103:8088/cluster
insert image description here

6. Configure History Server

In the above YARN web page, clicking History behind the record will fail to jump. The reason for the failure is that the history server has not been configured.
In order to view the historical running status of the program, you need to configure the history server
to configure mapred-site.xml

cd /opt/module/hadoop-3.1.3/etc/hadoop/
vim mapred-site.xml

Add the following to the configuration tag

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>
<!-- 指定MapReduce程序运行在Yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

<!-- 历史服务器端地址 -->
   <property>
       <name>mapreduce.jobhistory.address</name>
       <value>hadoop102:10020</value>
   </property>

<!-- 历史服务器web端地址 -->
   <property>
       <name>mapreduce.jobhistory.webapp.address</name>
       <value>hadoop102:19888</value>
   </property>
</configuration>

# 分发配置
/bin/xsync $HADOOP_HOME/etc/hadoop/mapred-site.xml

# 在hadoop102启动历史服务器
mapred --daemon start historyserver

Check whether the startup is successful

[root@hadoop102 hadoop]# jps
12386 JobHistoryServer # 历史服务器
12419 Jps
2952 DataNode
2799 NameNode

View historical service information on the Web side http://hadoop102:19888/jobhistory
insert image description here

7. Configure log aggregation

Log aggregation concept: After the application is running, the program running log information is uploaded to the HDFS system.
Configure yarn-site.xml

cd /opt/module/hadoop-3.1.3/etc/hadoop/
vim yarn-site.xml

Add the following to the configuration tag

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

    <!-- 指定MR走shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 指定ResourceManager的地址-->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop103</value>
    </property>

    <!-- 环境变量的继承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

   <!-- 开启日志聚集功能 -->
   <property>
       <name>yarn.log-aggregation-enable</name>
       <value>true</value>
   </property>
   <!-- 设置日志聚集服务器地址 -->
   <property>  
       <name>yarn.log.server.url</name>  
       <value>http://hadoop102:19888/jobhistory/logs</value>
   </property>
   <!-- 设置日志保留时间为7天 -->
   <property>
       <name>yarn.log-aggregation.retain-seconds</name>
       <value>604800</value>
   </property>
</configuration>

# 分发配置
/bin/xsync $HADOOP_HOME/etc/hadoop/yarn-site.xml

To enable the log aggregation function, NodeManager, ResourceManager, and HistoryServer need to be restarted.

# 进入hadoop目录
cd $HADOOP_HOME
# 关闭NodeManager 、ResourceManager
sbin/stop-yarn.sh
# 关闭HistoryServer
mapred --daemon stop historyserver
# 启动NodeManager 、ResourceManage
start-yarn.sh
# 启动HistoryServer
mapred --daemon start historyserver

Delete the files generated by the last execution in HDFS

hadoop fs -rm -r /output

Execute the WordCount program

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

There is one more record in the history
insert image description here
insert image description here

8. Summary of cluster start/stop methods

cd /opt/module/hadoop-3.1.3/sbin
# 启动HDFS
start-dfs.sh
# 停止HDFS
stop-dfs.sh
# 启动YARN
start-yarn.sh
# 停止YARN
stop-yarn.sh
# 关闭HistoryServer
mapred --daemon stop historyserver

cd /opt/module/hadoop-3.1.3/bin
# 启动NodeManager 、ResourceManage
start-yarn.sh
# 启动HistoryServer
mapred --daemon start historyserver

8.1 Cluster start and stop scripts at the same time

cd /opt/module/hadoop-3.1.3/sbin
vim myhadoop.sh

#!/bin/bash

if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit ;
fi

case $1 in
"start")
        echo " =================== 启动 hadoop集群 ==================="

        echo " --------------- 启动 hdfs ---------------"
        ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
        echo " --------------- 启动 yarn ---------------"
        ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
        echo " --------------- 启动 historyserver ---------------"
        ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver"
;;
"stop")
        echo " =================== 关闭 hadoop集群 ==================="

        echo " --------------- 关闭 historyserver ---------------"
        ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
        echo " --------------- 关闭 yarn ---------------"
        ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
        echo " --------------- 关闭 hdfs ---------------"
        ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
    echo "Input Args Error..."
;;
esac

Give the script execute permission

chmod +x myhadoop.sh

8.2 View the Java process scripts of the three servers

cd /opt/module/hadoop-3.1.3/sbin
vim jpsall.sh

#!/bin/bash

for host in hadoop102 hadoop103 hadoop104
do
        echo =============== $host ===============
        ssh $host jps 
done

Give the script execute permission

chmod +x jpsall.sh

Synchronize the two scripts just edited for the entire cluster

/bin/xsync /opt/module/hadoop-3.1.3/sbin

8.4 Test script

# 启动集群
myhadoop.sh start
# 停止集群
myhadoop.sh stop
# 查看三台服务器Java进程
jpsall.sh

9. Port changes

port name Hadoop2.x Hadoop3.x
NameNode internal communication port 8020 / 9000 8020 / 9000/9820
NameNode HTTP UI 50070 9870
MapReduce view execution task port 8088 8088
History server communication port 19888 19888

10. Cluster time synchronization

Find a machine as a time server, and all machines will synchronize with this cluster time regularly. The production environment requires periodic synchronization according to the accuracy of tasks.

# 查看 ntpd 状态
systemctl status ntpd
# 启动 ntpd 
systemctl start ntpd
# 设置 ntpd 服务开机自启动
systemctl is-enabled ntpd

Modify the ntp.conf configuration file of hadoop102

vim /etc/ntp.conf


将 # restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
改成 : restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap

将 
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
改成 
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst

添加下面两行
server 127.127.1.0
fudge 127.127.1.0 stratum 10

Modify the /etc/sysconfig/ntpd file of hadoop102

vim /etc/sysconfig/ntpd

SYNC_HWCLOCK=yes

Restart the ntpd service

systemctl start ntpd

Set the ntpd service to start at boot

systemctl enable ntpd

Modified in hadoop103 and hadoop104

# 停止 ntpd
systemctl stop ntpd
# 设置开机自启动
systemctl disable ntpd
# 添加定时任务
crontab -e
# 定时任务脚本
*/1 * * * * /usr/sbin/ntpdate hadoop102

# 更改时间
date -s "2021-9-11 11:11:11"
# 查看时间, 一分钟后再次查看时间
date

11. Problems encountered

11.1 The Hadoop on the virtual machine starts normally, the xshell connection is normal but cannot be accessed in the browser: http://hadoop102:9870

First check if the firewall is turned off

# 启动防火墙： 
systemctl start firewalld
# 查看防火墙状态： 
systemctl status firewalld
# 停止防火墙： 
systemctl disable firewalld
# 禁用防火墙： 
systemctl stop firewalld

See if hadoop is started

jps

Check if the windows host file is configured with hadoop102 mapping.
Open the hosts file in C:\Windows\System32\drivers\etc.
insert image description here
When the configuration is complete, refresh the windows DNS resolution
. Open cmd, enter

ipconfig/flushdns

Guess you like

Origin blog.csdn.net/qq_44154912/article/details/127065971
Recommended
Ranking
ElasticSearch-- data modeling best practices
Permission Maintenance - Shadow User Backdoor
Refactor the code using MVP mode
Quantitative investment-fundamental model-PVC multi-factor model
Spark Big Data Processing Lecture Notes 3.2 Mastering RDD Operators
Blazor page components (2)
Erlernen von Kenntnissen zur Android-Entwicklung – Kodierung, Verschlüsselung, Hash, Serialisierung und Zeichensätze
About Qi high in JAVA study notes SORM summary detailed personal explanation
Will you calculate the accuracy of the rope displacement sensor in the measurement?
OPENJTAG debugging learning (3): debugging using the gdb command line
Daily
More
2024-05-01(4)
2024-04-30(36)
2024-04-29(5)
2024-04-28(12)
2024-04-27(29)
2024-04-26(22)
2024-04-25(32)
2024-04-24(30)
2024-04-23(30)
2024-04-22(5)

Code World

© 2018 codetd.com