Table of contents
Unzip and modify the folder name
Copy the secret key to this machine
Java environment configuration and Hadoop environment configuration
Modify the hadoop configuration file
Hadoop initialization configuration
operating position
Statement, for the convenience of operation, all content is under the [/opt] folder.
cd /optUpload compressed package
Two packages are required, java and hadoop, versions 1.8 and 3.1.3.
Download link:
https://download.csdn.net/download/feng8403000/88074219
upload to /opt
Unzip and modify the folder name
unzip command
tar -zxvf jdk-8u212-linux-x64.tar.gz
tar -zxvf hadoop-3.1.3.tar.gz
modify folder command
mv 文件夹名 jdk
mv 文件夹名 hadoop
You can see that the name of the folder has been changed to facilitate the configuration of system variables.
password-free configuration
ssh-keygen -t rsa
Copy the secret key to this machine
ssh-copy-id -i root@localhostNeed to enter [yes] and [root password]
ssh authentication:
ssh 'root@localhost'
As can be seen from the path, opt becomes ~.
Java environment configuration and Hadoop environment configuration
Create a script file such as: [hadoop3.sh] file, add the following path configuration
export JAVA_HOME=/opt/jdk
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
execute script
source hadoop3.shconfiguration confirmation
hadoop version
Modify the hadoop configuration file
Here we add and modify one by one
1. Modify hadoop-env.sh
Just add the following code to the top line of the file.
export JAVA_HOME=/opt/jdk
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
2. Modify yarn-env.sh
export JAVA_HOME=/opt/jdk
3. Modify core-site.xml
Optimistic about where to add, in the configuration tag.
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-record/temp</value>
</property>
4. Modify hdfs-site.xml
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop-record/nameNode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop-record/dataNode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
5. Modify mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
6. Modify yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
Here we have modified a total of 6 files, all must be changed, don't make a mistake.
Hadoop initialization configuration
hdfs namenode -format
Need to wait a while here.
Initialization is complete.
Start the Hadoop service
start-all.sh
jps
access service
The access method is【ip:9870】, the port number here is different from 50070 in 2.7.3, don't make a mistake.
For example: 【http://192.168.200.132:9870/】
If there is no access, it is because the firewall is not closed [systemctl stop firewalld]
systemctl stop firewalldAccess results after closing:
It shows that the configuration of our single machine has been completed, and the subsequent configuration of one master and two slaves is also based on this method. It is nothing more than changing the configuration of the relationship between the master and the slave.