Big data combat Linux Ubuntu 20.04.1 server Hadoop2.8.5 cluster deployment

1. Preliminary work

Click here for details to learn more

2. Time synchronization

Idea: The time for the master node to connect to the external network, and the time for the slave node to only connect to the master node, achieve the same purpose as the time of 3 machines

2.1 Master node time synchronization

2.1.1 Install the software package

apt-get install chrony -y
Insert picture description here

2.1.2 Edit /chrony.conf

vi /etc/chrony/chrony.conf
add
local stratum 10
allow 172.25.0.0/24
Insert picture description here

2.1.3 Activate start NTP service

systemctl enable chrony
systemctl restart chrony
Insert picture description here

2.2 Slave node time synchronization

2.2.1 Install the software package

apt-get install chrony -y
Insert picture description here

2.2.2 Edit /chrony.conf

vi /etc/chrony/chrony.conf
comment these 4 lines and add server master iburst
Insert picture description here

2.2.3 Activate start NTP service

Insert picture description here

2.3 Whether the test is successful

2.3.1 Master node test

chronyc sources
Insert picture description here

2.3.2 Slave node test

Insert picture description here
At this point NTP is complete.

3. Password-free SSH settings

The purpose is to send the edited document to the slave node

3.1 Create a group for all nodes

sudo groupadd -g 285 angel
(You can operate under the ordinary user of bass, you can operate under the root user. If you need to increase the privileges of sudo in bass, the root user does not need it.)
Insert picture description here
285 is the group number, and angel is the group name.

3.2 Create users for all nodes

$ sudo useradd -u 285 -g 285 -m -s /bin/bash angel
Insert picture description here

The user number is 285, the user group number is 285, and the user name is angel.

3.3 Set the password of the angel user

sudo gpasswd -a angel sudo
adds the angel user to the sudo group.
Insert picture description here
sudo passwd angel
password is 123
Insert picture description here

3.4 switch angel user

su-angel
password: 123
Insert picture description here

3.5 Generate a certificate (operate on the master node)

ssh-keygen -t rsa
encryption algorithm selection rsa,
Insert picture description here

3.6 Copy the public key to all points

3.6.1 Master node

ssh-copy-id -i .ssh/id_rsa.pub master

yes
password: 123
Insert picture description here

3.7 Test

ssh master
ctrl+D exit
logout
Insert picture description here

3.7.1 Connect the slave node at the master node

ssh slave1
ssh slave2

4.Java installation

4.1 Create app directory for all nodes

Create
sudo mkdir /app
Insert picture description here
sudo chown -R angel:angel /app under the angel user

4.2 Edit jdk environment variables for all nodes

vi /home/angel/.profile
add 2 lines at the end
Insert picture description here

4.3 All node jdk environment variables take effect

source /home/angel/.profile

4.4 Upload jdk compression to angel user

4.4.1 Use winscp tool to log in as root user

Insert picture description here

4.5 Unzip the jdk compressed package and place it in the /app directory

cd / app
tar xzvf /home/angel/jdk-8u261-linux-x64.tar.gz -C / app

Insert picture description here

5. Deliver the content on /app to the slave node

scp -r /app/* [email protected]:/app
scp -r /app/* [email protected]:/app

Insert picture description here

6 test

java -version
javac -version
Insert picture description here

7 Hadoop installation

7.1 Upload hadoop package to angel user

tar xzvf /home/angel/hadoop-2.8.5.tar.gz -C / app
Insert picture description here

7.2 Edit Hadoop environment variables for all nodes

vi /home/angel/.profile
add:

export HADOOP_HOME=/app/hadoop-2.8.5 
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin 
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

Insert picture description here

7.3 Hadoop environment variables take effect on all nodes

source /home/angel/.porfile

7.4 Modify Hadoop configuration file

7.4.1 hadoop-env.sh

vi /app/hadoop-2.8.5/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/app/jdk1.8.0_261

Insert picture description here

7.4.2 core-site.xml

vi /app/hadoop-2.8.5/etc/hadoop/core-site.xml

<property> 
<name>fs.defaultFS</name> 
<value>hdfs://master:9000</value> 
</property> 
<property> 
<name>hadoop.tmp.dir</name> 
<value>/tmp/hadoop-2.8.5</value> 
</property> 
<property> 
<name>hadoop.proxyuser.angel.hosts</name> 
<value>*</value> 
</property> 
<property> 
<name>hadoop.proxyuser.angel.groups</name> 
<value>*</value> 
</property> 

Insert picture description here

7.4.3 hdfs-site.xml

vi /app/hadoop-2.8.5/etc/hadoop/hdfs-site.xml

<property> 
 <name>dfs.replication</name> 
<value>2</value> 
</property> 
<property> 
<name>dfs.namenode.name.dir</name> 
<value>/app/hadoop-2.8.5/dfs/name</value> 
</property> 
<property> 
<name>dfs.datanode.data.dir</name> 
<value>/app/hadoop-2.8.5/dfs/data</value> 
</property> 

Insert picture description here

7.4.4 mapred-site.xml

First copy mapred-site.xml.template and name it mapred-site.xml
cp /app/hadoop-2.8.5/etc/hadoop/mapred-site.xml.template /app/hadoop-2.8.5/etc/hadoop /mapred-site.xml

<property> 
<name>mapreduce.framework.name</name> 
<value>yarn</value> 
</property> 
<property> 
<name>mapreduce.jobhistory.address</name> 
<value>master:10020</value> 
</property> 
<property> 
<name>mapreduce.jobhistory.webapp.address</name> 
<value>master:19888</value> 
</property> 

Insert picture description here

7.4.5 yarn-site.xml

vi /app/hadoop-2.8.5/etc/hadoop/yarn-site.xml

<property> 
<name>yarn.resourcemanager.hostname</name> 
<value>master</value> 
</property> 
<property> 
<name>yarn.nodemanager.aux-services</name> 
<value>mapreduce_shuffle</value> 
</property> 

Insert picture description here

7.4.6 slaves

vi /app/hadoop-2.8.5/etc/hadoop/slaves
Insert picture description here

7.5 Copy to slave node

scp -r /app/hadoop-2.8.5/ [email protected]:/app
scp -r /app/hadoop-2.8.5/ [email protected]:/app

Wait patiently for the delivery to complete
Insert picture description here

8.Hadoop startup

8.1 Format namenode

hdfs purpose -format
Insert picture description here

8.2 Start Hadoop

start-dfs.sh
Insert picture description here

8.3 Start yarn

start-yarn.sh
Insert picture description here

8.4 Start JobHistoryServer

mr-jobhistory-daemon.sh start historyserver
Insert picture description here

8.5 View process

8.5.1 master node

jps
Insert picture description here

8.5.2 slave1, slave2 nodes

jps
Insert picture description here
Insert picture description here

Insert picture description here
Insert picture description here
Insert picture description here
There are 2 live nodes, port numbers 50070 and 8088 have
been successfully configured in this hadoop environment!

Guess you like

Origin blog.csdn.net/qq_45059457/article/details/108951024