Fully distributed hadoop installation (centos7)

Note: Non-Vmware installations are
all run under root privileges

1. Modify the host name

1. View the host name: hostname
2. Modify the host name:hostnamectl set-hostname master

Or modify:/etc/hostname

3. Write the host name to /etc/hosts

IP地址 主机名

2. Turn off the firewall

1. Check the firewall status: firewall-cmd --state| system status firewalld
2. Turn off the firewall:system stop firewalld

3.ssh

The master node generates the key: ssh-keygen -t rsa
Enter 3 times
and then copy the key to other nodes

ssh-copy-id master
ssh-copy-id slave

4. Install jdk

1. Download the jdk compressed package
2. Use the shell software to drag to the host
3. Decompress the compressed package: tar -zxvf 版本号 -C 指定文件夹
4. Modify the /etc/profile file and add:

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_221
export PATH=$PATH:$JAVA_HOME/bin

注意是自己的版本，自己的安装路径

5. Let the modified file take effect: source /etc/profile
6. Test whether the installation is successful:java -version

1-6 all nodes all nodes must

5. Install Hadoop

1. Download the hadoop compressed package, upload it to the host, and decompress the compressed package
2. Also modify the /etc/profile file and add:

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-2.9.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

3. Let the modified file take effect: source /etc/profile
4. Create a new hdfs related directory:

mkdir /home/hdfs
mkdir /home/hdfs/tmp
mkdir /home/hdfs/name
mkdir /home/hdfs/data

Steps 1-4 must be executed on all nodes

5. Next modify the hadoop configuration:
file name:hadoop_env.sh / yarn-env.sh
path:hadoop-2.9.2/etc/hadoop/(下同)

export JAVA_HOME=/usr/java/latest 

file name:core-site.xml

<configuration>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/hdfs/tmp</value>
	</property>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://master:9000</value>
	</property>
</configuration>

file name:hdfs-site.xml:

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<property>
		<name>dfs.permissions</name>
		<value>false</value>
	</property>
	<property>
		<name>dfs.name.dir</name>
		<value>/home/hdfs/name</value>
	</property>
	<property>
		<name>dfs.data.dir</name>
		<value>/home/hdfs/data</value>
	</property>
</configuration>

file name:mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
</configuration>

file name:yarn-site.xml

<configuration>
	<property>
		<name>yarn.resourcemanager.hostname</name>
		<value>master</value>
	</property>
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
	</property>
</configuration>

file name:slaves

将localhost去掉
添加集群主机：
master
slave1
slave2

将修改了的文件全部发送到从节点：
scp 路径+文件名 @从节点主机名：路径
eg: scp /opt/module/hadoop-2.9.2/etc/hadoop/yarn-site.xml @slave1:/opt/mudule-2.9.2/etc/hadoop/

6. Start Hadoop

1. Format data first: hadoop namenode -format
2. start-all.sh
3. Enter the command to jpdview: