1. Preliminary work
Click here for details to learn more
2. Time synchronization
Idea: The time for the master node to connect to the external network, and the time for the slave node to only connect to the master node, achieve the same purpose as the time of 3 machines
2.1 Master node time synchronization
2.1.1 Install the software package
apt-get install chrony -y
2.1.2 Edit /chrony.conf
vi /etc/chrony/chrony.conf
add
local stratum 10
allow 172.25.0.0/24
2.1.3 Activate start NTP service
systemctl enable chrony
systemctl restart chrony
2.2 Slave node time synchronization
2.2.1 Install the software package
apt-get install chrony -y
2.2.2 Edit /chrony.conf
vi /etc/chrony/chrony.conf
comment these 4 lines and add server master iburst
2.2.3 Activate start NTP service
2.3 Whether the test is successful
2.3.1 Master node test
chronyc sources
2.3.2 Slave node test
At this point NTP is complete.
3. Password-free SSH settings
The purpose is to send the edited document to the slave node
3.1 Create a group for all nodes
sudo groupadd -g 285 angel
(You can operate under the ordinary user of bass, you can operate under the root user. If you need to increase the privileges of sudo in bass, the root user does not need it.)
285 is the group number, and angel is the group name.
3.2 Create users for all nodes
$ sudo useradd -u 285 -g 285 -m -s /bin/bash angel
The user number is 285, the user group number is 285, and the user name is angel.
3.3 Set the password of the angel user
sudo gpasswd -a angel sudo
adds the angel user to the sudo group.
sudo passwd angel
password is 123
3.4 switch angel user
su-angel
password: 123
3.5 Generate a certificate (operate on the master node)
ssh-keygen -t rsa
encryption algorithm selection rsa,
3.6 Copy the public key to all points
3.6.1 Master node
ssh-copy-id -i .ssh/id_rsa.pub master
yes
password: 123
3.7 Test
ssh master
ctrl+D exit
logout
3.7.1 Connect the slave node at the master node
ssh slave1
ssh slave2
4.Java installation
4.1 Create app directory for all nodes
Create
sudo mkdir /app
sudo chown -R angel:angel /app under the angel user
4.2 Edit jdk environment variables for all nodes
vi /home/angel/.profile
add 2 lines at the end
4.3 All node jdk environment variables take effect
source /home/angel/.profile
4.4 Upload jdk compression to angel user
4.4.1 Use winscp tool to log in as root user
4.5 Unzip the jdk compressed package and place it in the /app directory
cd / app
tar xzvf /home/angel/jdk-8u261-linux-x64.tar.gz -C / app
5. Deliver the content on /app to the slave node
scp -r /app/* angel@slave1:/app
scp -r /app/* angel@slave2:/app
6 test
java -version
javac -version
7 Hadoop installation
7.1 Upload hadoop package to angel user
tar xzvf /home/angel/hadoop-2.8.5.tar.gz -C / app
7.2 Edit Hadoop environment variables for all nodes
vi /home/angel/.profile
add:
export HADOOP_HOME=/app/hadoop-2.8.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
7.3 Hadoop environment variables take effect on all nodes
source /home/angel/.porfile
7.4 Modify Hadoop configuration file
7.4.1 hadoop-env.sh
vi /app/hadoop-2.8.5/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/app/jdk1.8.0_261
7.4.2 core-site.xml
vi /app/hadoop-2.8.5/etc/hadoop/core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-2.8.5</value>
</property>
<property>
<name>hadoop.proxyuser.angel.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.angel.groups</name>
<value>*</value>
</property>
7.4.3 hdfs-site.xml
vi /app/hadoop-2.8.5/etc/hadoop/hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/app/hadoop-2.8.5/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/app/hadoop-2.8.5/dfs/data</value>
</property>
7.4.4 mapred-site.xml
First copy mapred-site.xml.template and name it mapred-site.xml
cp /app/hadoop-2.8.5/etc/hadoop/mapred-site.xml.template /app/hadoop-2.8.5/etc/hadoop /mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
7.4.5 yarn-site.xml
vi /app/hadoop-2.8.5/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
7.4.6 slaves
vi /app/hadoop-2.8.5/etc/hadoop/slaves
7.5 Copy to slave node
scp -r /app/hadoop-2.8.5/ angel@slave1:/app
scp -r /app/hadoop-2.8.5/ angel@slave2:/app
Wait patiently for the delivery to complete
8.Hadoop startup
8.1 Format namenode
hdfs purpose -format
8.2 Start Hadoop
start-dfs.sh
8.3 Start yarn
start-yarn.sh
8.4 Start JobHistoryServer
mr-jobhistory-daemon.sh start historyserver
8.5 View process
8.5.1 master node
jps
8.5.2 slave1, slave2 nodes
jps
There are 2 live nodes, port numbers 50070 and 8088 have
been successfully configured in this hadoop environment!