基于百度智能云服务器搭建伪分布式Hadoop集群

引言

Hadoop集群搭建一般分为单机型、伪分布式、完全分布式，本文目的在于搭建伪分布式的Hadoop集群

运行环境

1、CentOS / 7.9 x86_64 (64bit)
2、jdk-8u281-linux-x64
3、hadoop-2.7.6

建立文件夹

mkdir /download
mkdir /software

传输所需文件

解压文件

cd /software
tar -zxvf /download/jdk-8u281-linux-x64.tar.gz 
mv jdk1.8.0_281/ jdk
mv hadoop-2.7.6/ hadoop

配置免密登陆

ssh-keygen -t rsa 

按连续3下回车

cd ~/.ssh
touch ~/.ssh/authorized_keys  
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  

检验结果

ssh localhost  

如遇
输入yes按回车

配置java

vi /etc/profile

在末尾按insert输入以下内容

#java
export JAVA_HOME=/software/jdk
export PATH=$JAVA_HOME/bin:$PATH

按ESC输入：wq按回车

source /etc/profile

检验结果

java -version

安装Hadoop

vi /etc/profile

在末尾按insert输入以下内容

#hadoop  
export HADOOP_HOME=/apps/hadoop  
export PATH=$HADOOP_HOME/bin:$PATH  

按ESC输入：wq按回车

source /etc/profile

检验结果

hadoop version  

配置Hadoop

cd /software/hadoop/etc/hadoop/

配置hadoop-env.sh

vi hadoop-env.sh

将该行改为如图所示

export JAVA_HOME=/software/jdk

按ESC输入：wq按回车

配置core-site.xml

vi core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>  
    	<name>hadoop.tmp.dir</name>  
    	<value>/software/data/tmp/hadoop/tmp</value>  
	</property>  
	<property>  
    	<name>fs.defaultFS</name>  
    	<value>hdfs://localhost:9000</value>  
	</property>  
</configuration>

按ESC输入：wq按回车

mkdir -p /software/data/tmp/hadoop/tmp

配置hdfs-site.xml

 vi hdfs-site.xml

Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>/software/data/tmp/hadoop/hdfs/name</value>
</property>  
 <property>  
     <name>dfs.datanode.data.dir</name>  
     <value>/software/data/tmp/hadoop/hdfs/data</value>
 </property>
 <property>
     <name>dfs.replication</name>
     <value>1</value>
 </property>
 <property>
     <name>dfs.permissions.enabled</name>
     <value>false</value>
 </property>

按ESC输入：wq按回车

mkdir -p /software/data/tmp/hadoop/hdfs  

配置slaves

vi slaves

将内容修改为如图所示

localhost 

按ESC输入：wq按回车

配置mapred-site.xml

mv mapred-site.xml.template mapred-site.xml
vi mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>  
</configuration>

按ESC输入：wq按回车

配置yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property> 
</configuration>

按ESC输入：wq按回车

格式化HDFS文件系统

hadoop namenode -format  

启动集群

cd /software/hadoop/sbin/
./start-all.sh

验证结果

jps

如图所示即为成功

Hadoop初体验

HDFS建立文件夹测试

hadoop fs -mkdir /myhadoop01

检验结果

shell命令检查

hadoop fs -ls -R /

如图即为成功

web页面检查

HDFS页面

浏览器登陆网址http://IP:50070/ (如果是物理机或本地虚拟机即为localhost：50070)


如图即为成功

yarn页面

浏览器登陆网址http://IP:8088/ (如果是物理机或本地虚拟机即为localhost：8088)

MapReduce运行π值运算测试

cd share/hadoop/mapreduce/
hadoop jar hadoop-mapreduce-examples-2.7.6.jar pi 3 3

类似如图结果即为成功