Cluster Four Parts (2): Perfect Hadoop Cluster Construction

Others 2022-04-23 03:56:24 views: 0

    Following the successful construction of the Zookeeper cluster last time, now let's talk about the construction of the Hadoop cluster. Well, without further ado, let's start today's adventure.

    1. Environment: Virtual machine CentOs7 system, complete environment, please confirm that JDK and Hadoop installation package have been installed (I use 2.9.0), the node still uses the two cloned last time, the following is the environment of one of them build.

    2. Hadoop configuration (decompress it by yourself)

    Explain before configuration (it is very important, take note of it): First of all, let’s take a look at namenode, datanode, and ResourceManager. Here we are going to use slave01 as namenode, slave02 and slave03 as datanode. The following operations are based on this premise. Please Everyone do a good job reviewing before the war, in case you don't know the meaning of the following operations . Let's start the show:

    After decompression, the file directory structure is basically as follows:

[hadoop@slave01 hadoop]$ pwd
/usr/local/hadoop
[hadoop@slave01 hadoop]$ ls
bin  include  libexec      logs        README.txt  share
etc  lib      LICENSE.txt  NOTICE.txt  sbin        tmp
[hadoop@slave01 hadoop]$ cd etc/hadoop/
[hadoop@slave01 hadoop]$ ls
capacity-scheduler.xml      httpfs-env.sh            mapred-env.sh
configuration.xsl           httpfs-log4j.properties  mapred-queues.xml.template
container-executor.cfg      httpfs-signature.secret  mapred-site.xml.template
core-site.xml               httpfs-site.xml          slaves
hadoop-env.cmd              kms-acls.xml             ssl-client.xml.example
hadoop-env.sh               kms-env.sh               ssl-server.xml.example
hadoop-metrics2.properties  kms-log4j.properties     yarn-env.cmd
hadoop-metrics.properties   kms-site.xml             yarn-env.sh
hadoop-policy.xml           log4j.properties         yarn-site.xml
hdfs-site.xml               mapred-env.cmd

    Turn off the firewall, everyone understands:

systemctl stop firewalld    #只在本次运用时生效，下次开启机器时需重复此操作
或
systemctl disable firewalld   #此命令在下次重启时生效，将永久关闭防火墙

    (1) Add Hadoop to the environment variable

vim /etc/profile

    Add HADOOP_HOME and modify it as follows:

JAVA_HOME=/usr/java/jdk1.8.0_161
JRE_HOME=/usr/java/jdk1.8.0_161/jre
SCALA_HOME=/usr/local/scala
HADOOP_HOME=/usr/local/hadoop
ZOOKEEPER_HOME=/usr/local/zookeeper
KAFKA_HOME=/usr/local/kafka
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME SCALA_HOME HADOOP_HOME ZOOKEEPER_HOME KAFKA_HOME PATH CLASSPATH

    Check whether the configuration in the Hadoop decompression directory: /etc/hadoop/hadoop-env.sh is normal:

# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.8.0_161

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol.  Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

    Change JAVA_HOME to your own local jdk installation address .

    (2) Modify the core-site.xml file

    Locate the file from the above directory structure and open it to display the following initial configuration:

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

    Make the following modifications:

<configuration>
    <!-- 指定Hadoop临时目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <!-- 指定HDFS的namenode的通信地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://slave01:9000</value>
    </property>
</configuration>

    (3) Modify the hdfs-site.xml file

     Locate the file from the above directory structure and open it to display the following initial configuration:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

    Make the following modifications:

<configuration>
    <!-- 设置namenode的http通信地址 -->
    <property>
        <name>dfs.namenode.http-address</name>
        <value>slave01:50070</value>
    </property>
    <!-- 设置hdfs副本数量 --> 
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <!-- 设置namenode存放路径 -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <!--设置datanode存放路径  -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

    Other configuration parameters can be added here. I will not introduce them here. You can view them on Baidu. Pay attention to the creation of the folder of the set storage path. If it does not exist, create it .

    (4) Modify the mapred-site.xml file

    First make a copy of the file's temple and rename it:

cp mapred-site.xml.template mapred-site.xml

    The initial content is empty, and it is modified as follows:

<configuration>
      <!-- 指定mapreduce框架为yarn方式 -->
      <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
      </property>
</configuration>

    (5) Modify the yarn-site.xml file

    The initial content is empty, and it is modified as follows:

<configuration>

<!-- Site specific YARN configuration properties -->
        <!-- 设置 resourcemanager 在哪个节点-->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>slave01</value>
        </property>
         <!-- reducer取数据的方式是mapreduce_shuffle -->
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

    (6) Modify the slaves file ( just add the server name as datanode )

slave02
slave03

    (7) Copy the configuration file to other servers

    If you do not want to configure the files one by one, you can copy the configured files to other required servers. Note that after the copy is successful, execute the command: source /etc/profile to make it effective :

//slave01上的/etc/profile文件拷贝到slave02
scp -r /etc/profile slave02:/etc/profile
//slave01上的/usr/local/hadoop文件夹整个目录拷贝到slave02
scp -r /usr/local/hadoop slave02:/usr/local/

    (8) Format HDFS

    Execute the command on the server of the namenode node, that is, slave01:

hdfs namenode -format

    If the format is successful, it will be displayed as follows:

8/03/16 15:40:23 INFO namenode.FSDirectory: XAttrs enabled? true
18/03/16 15:40:23 INFO namenode.NameNode: Caching file names occurring more than 10 times
18/03/16 15:40:23 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAccessTimeOnlyChange: false
18/03/16 15:40:23 INFO util.GSet: Computing capacity for map cachedBlocks
18/03/16 15:40:23 INFO util.GSet: VM type       = 64-bit
18/03/16 15:40:23 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
18/03/16 15:40:23 INFO util.GSet: capacity      = 2^18 = 262144 entries
18/03/16 15:40:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/03/16 15:40:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/03/16 15:40:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/03/16 15:40:23 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/03/16 15:40:23 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/03/16 15:40:23 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/03/16 15:40:23 INFO util.GSet: VM type       = 64-bit
18/03/16 15:40:23 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
18/03/16 15:40:23 INFO util.GSet: capacity      = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /usr/local/hadoop/tmp/dfs/name ? (Y or N) y
18/03/16 15:40:26 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1094714660-127.0.0.1-1521186026480
18/03/16 15:40:26 INFO common.Storage: Will remove files: [/usr/local/hadoop/tmp/dfs/name/current/VERSION, /usr/local/hadoop/tmp/dfs/name/current/seen_txid, /usr/local/hadoop/tmp/dfs/name/current/fsimage_0000000000000000000.md5, /usr/local/hadoop/tmp/dfs/name/current/fsimage_0000000000000000000, /usr/local/hadoop/tmp/dfs/name/current/edits_0000000000000000001-0000000000000000004, /usr/local/hadoop/tmp/dfs/name/current/fsimage_0000000000000000004.md5, /usr/local/hadoop/tmp/dfs/name/current/fsimage_0000000000000000004, /usr/local/hadoop/tmp/dfs/name/current/edits_0000000000000000005-0000000000000000005, /usr/local/hadoop/tmp/dfs/name/current/edits_inprogress_0000000000000000006]
18/03/16 15:40:26 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
18/03/16 15:40:26 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/03/16 15:40:26 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
18/03/16 15:40:26 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/03/16 15:40:26 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave01/127.0.0.1
************************************************************/
[hadoop@slave01 hadoop]$

    If there is an operation to be done in the middle, just press the prompt. Note that the following error is reported. Many students may have encountered it when configuring it. It takes a lot of time:

18/03/16 15:54:14 WARN namenode.NameNode: Encountered exception during format: 
java.io.IOException: Cannot remove current directory: /usr/local/hadoop/tmp/dfs/name/current
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:358)
	at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:571)
	at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:592)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:166)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1614)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1741)
18/03/16 15:54:14 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot remove current directory: /usr/local/hadoop/tmp/dfs/name/current
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:358)
	at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:571)
	at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:592)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:166)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1614)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1741)
18/03/16 15:54:14 INFO util.ExitUtil: Exiting with status 1: java.io.IOException: Cannot remove current directory: /usr/local/hadoop/tmp/dfs/name/current
18/03/16 15:54:14 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave02/127.0.0.1
************************************************************/
[hadoop@slave02 hadoop]$

    The reason for the error is just because the permissions of the /usr/local/hadoop/tmp folder are not set , and the sadness is so great that I am speechless. The solution, of course, is to add permissions to this directory, chmod to understand, and perform this operation on all three servers:

sudo chmod -R a+w /usr/local/hadoop

    At this point, the basic configuration has been completed. Of course, there are still many configurations that are not used, here is just a preliminary and simple cluster.

    (3) Start the hadoop cluster

    (1) Execute the following command on slave01

start-dfs.sh
start-yarn.sh
jps

    A successful startup is displayed as follows:

[hadoop@slave01 sbin]$ ./start-dfs.sh 
Starting namenodes on [slave01]
slave01: namenode running as process 26894. Stop it first.
The authenticity of host 'slave03 (192.168.89.131)' can't be established.
ECDSA key fingerprint is SHA256:AJ/rhsl+I6zFOYitxSG1CuDMEos0Oue/u8co7cF5L0M.
ECDSA key fingerprint is MD5:75:eb:3c:52:df:9b:35:cb:b3:05:c4:1a:20:13:73:01.
Are you sure you want to continue connecting (yes/no)? slave02: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave02.out
yes
slave03: Warning: Permanently added 'slave03,192.168.89.131' (ECDSA) to the list of known hosts.
slave03: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave03.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:AJ/rhsl+I6zFOYitxSG1CuDMEos0Oue/u8co7cF5L0M.
ECDSA key fingerprint is MD5:75:eb:3c:52:df:9b:35:cb:b3:05:c4:1a:20:13:73:01.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-slave01.out
[hadoop@slave01 sbin]$ jps
27734 Jps
27596 SecondaryNameNode
26894 NameNode
[hadoop@slave01 sbin]$ ./start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-slave01.out
slave03: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave03.out
slave02: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave02.out
[hadoop@slave01 sbin]$ jps
28080 Jps
27814 ResourceManager
27596 SecondaryNameNode
26894 NameNode
[hadoop@slave01 sbin]$

    If you need to operate in the middle, follow the prompts. From the results displayed by the jps command, you can see that the NameNode and ResourceManager have been started normally, which is very gratifying.

    (2) Execute the jps command on slave02 and slave03

#slave02
[hadoop@slave02 hadoop]$ jps
12296 DataNode
13226 Jps
12446 NodeManager
[hadoop@slave02 hadoop]$ 

#slave03
[hadoop@slave03 hadoop]$ jps
12122 NodeManager
11978 DataNode
12796 Jps
[hadoop@slave03 hadoop]$

    You can see that both DataNode and NodeManager are started normally. If it does not start successfully, you need to turn off the process in slave01 and find the reason . I also have this problem. Later, I deleted the data folder in the /hadoop/tmp/dfs directory on the three servers, and the restart was successful. . Of course it depends.

    (3) View hadoop operation management interface

    Visit http://slave01:8088 (default port 8088, if it is occupied, please modify it):

    Visit http://slave01:50070:

    Well, now the construction of the Hadoop cluster has been completed, O(∩_∩)O.

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=325443128&siteId=291194637
Recommended
Ranking
Linux关机和重启详解（shutdown、halt、poweroff、reboot、init）
Netty work notes 0007---NIO's three core component relationships
Knife4j tutorial
2021.10.29，内容:什么时候用接口和抽象类
How to solve the problem that changing the memory frequency causes the computer to become unusable?
SpringMVC Tutorial - Controller
linux learning skills -Linux 25 transport Vega paid special privileges and facl extension
Financial quarterly report evaluation report data automatic generation 1.0
Agile Development Series - The Values of Agile Development
scrapy achieve browsercookie Middleware
Daily
More
2024-05-19(0)
2024-05-18(31)
2024-05-17(6)
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)

Code World

© 2018 codetd.com