Hadoop pseudo-distributed installation and configuration

Required:

jdk
hadoop
finalshell

1. Upload files

Upload both downloaded jdk and hadoop to linux
insert image description here

1. Install jdk

First decompress the uploaded jdk
and enter the jdk directory
cd /usr/local
to see the jdk just uploaded. My version is jdk-8u261-linux-x64.tar
and start decompressing

tar package with:

tar -xvf jdk-8u261-linux-x64.tar

tar.gz package with:

tar -xzvf jdk-8u261-linux-x64.tar.gz

After the decompression is complete, the current path will have a decompressed jdk,
you can delete the compressed package conveniently

rm -f jdk-8u261-linux-x64.tar

3. Configure environment variables

vi ~/.bash_profile

then add the following
Pay attention to the jdk path and specific version

export JAVA_HOME=/usr/local/jdk1.8.0_261
export JRE_HOME=$JAVA_HOME/jre 
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

insert image description here

Make it effective and check whether it is successful

source ~/.bash_profile

Then test whether the environment configuration is successful or not

java -version

insert image description here

4. Turn off the firewall

systemctl stop firewalld.service
systemctl disable firewalld.service

After closing, confirm whether the firewall status is closed

systemctl status firewalld

**Active: active (running)** appears and is highlighted to indicate that it is in the startup state.
Appear **Active: inactive (dead)** gray means stop

5. Configure the hostname

hostnamectl --static set-hostname niit01

Description: The –static parameter represents the permanent effect
insert image description here
niit01 expresses the host name you want to set

6. Install hadoop

Create a new tools folder and put it
in Hadoop to upload the downloaded one (upload is the same as the upload method above)

6.1 Decompression

tar -xvf hadoop-2.7.3.tar

insert image description here

6.2 Configure environment variables

vi  ～/.bash_profile

insert image description here
Let the environment variable take effect

source ～/.bash_profile

and verify that it works

hdfs

insert image description here
It will take effect when relevant information appears

6.3 Configure the mapping relationship between host name and IP address

Modify the host file

vi /etc/host

Add directly at the end of the file, for example:
ip is your ip niit01 is the host name configured in step 5

192.168.1.234  niit01

Save and exit
insert image description here

6.4 Create a new tmp directory (note the actual path of your hadoop):

mkdir  /usr/tools/hadoop-2.7.3/tmp

6.5 Five files need to be configured:

hadoop-env.sh
hdfs-site.xml
core-site.xml
mapper-site.xml
yarn-site.xml

6.5.1 Configuration of hadoop-env.sh

vi /usr/tools/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

Find JAVA_HOME in the hadoop-env.sh file, and make the following modifications
export JAVA_HOME=/usr/local/jdk1.8.0_171
(note the version, and the path where you installed jdk, please delete this note when you see it)

6.5.2 hdfs-site.xml file

vi /usr/tools/hadoop-2.7.3/etc/hadoop/hdfs-site.xml

Add the following information between the hdfs-site.xml file:

<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

6.5.3 core-site.xml file

vi /usr/tools/hadoop-2.7.3/etc/hadoop/core-site.xml

Add the following information between the core-site.xml files:

<property>
<name>fs.defaultFS</name>
<value>hdfs://niit01:9000</value> //(注意主机名,9000不变,看到请删除此注释)
</property>              
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/tools/hadoop-2.7.3/tmp</value>
</property>

6.5.4 mapper-site.xml document

This file does not exist in advance, you need to create a copy of this file first

cp mapred-site.xml.template mapred-site.xml

vi /usr/tools/hadoop-2.7.3/etc/hadoop/mapred-site.xml

Add the following information between "configuration" and "/configuration" of the mapper-site.xml file:

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

6.5.5 yarn-site.xml file

vi /usr/tools/hadoop-2.7.3/etc/hadoop/yarn-site.xml

Add the following information between "configuration" and "/configuration" of the yarn-site.xml file:

<property>
<name>yarn.resourcemanager.hostname</name>
<value>niit01</value>       //(注意主机名，看到请删除此注释)
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

6.6 Format HDFS (NameNode)

hdfs namenode -format

6.7 Configure password-free login

ssh-keygen -t rsa #（一路回车,有y/n选择y，看到请删除此注释）
cd ~/.ssh/
ssh-copy-id -i id_rsa.pub root@niit01 #（注意主机名，看到请删除此注释）

insert image description here

Start the service successfully as shown in the figure

start-all.sh

insert image description here
When the service is started, the following problems occur.
This local address corresponds to 0.0.0.0 and 127.0.0.1. This IP is not configured for password-free login. can be executed

 ssh-copy-id -i ~/.ssh/id_rsa.pub 0.0.0.0

insert image description here

Successfully
execute the jps command to see if there are the following 5 processes:
1.NameNode
2.DataNode
3.SecondaryNameNode
4.ReourceManager
5.NodeManager
insert image description here

6.8 Verification

1) Verify
HDFS on the web interface ------ http://niit01:50070 (note the host name, use 192.168.xxx.xxx:50070 if you can't open it) You can use IP or host name
Yarn---- ----http://niit01:8088
insert image description here

6.9 Stop service

stop-all.sh