Hadoop集群搭建（二）-- 集群搭建（部分）

其他 2018-08-06 14:16:15 阅读次数: 0

一、基础信息

1.集群节点地址分配
主机名 ip
master 192.168.100.200
slave1 192.168.100.201
slave2 192.168.100.202
2.软件使用版本
软件 版本
jdk jdk1.8.0_172
hadoop hadoop-2.7.6
3.软件安装路径
文件名 路径
jdk1.8.0_172 /opt/app/Java/jdk1.8.0_172
hadoop-2.7.6 /opt/app/Hadoop/hadoop-2.7.6

二、基本软件安装

最小化安装需要手动安装基本的操作软件

==可以直接在shell窗口进行全部会话操作==

1.关闭防火墙
#关闭防火墙
[root@master ~]# systemctl stop firewalld
#禁止开机启动防火墙
[root@master ~]# systemctl disable firewalld

扩展

#查看防火墙状态
[root@master ~]# systemctl status firewalld
#开启防火墙
[root@master ~]# systemctl start firewalld
#设置开机启动防火墙
[root@master ~]# systemctl enable firewalld
#重启防火墙
[root@master ~]# systemctl restart firewalld
2.openssh安装

远程上传文件

[root@master ~]# yum -y install openssh-clients
3.ntpdate安装

同步时间

#安装ntpdate
[root@master ~]# yum -y install ntp ntpdate
#同步网络时间
[root@master ~]# ntpdate cn.pool.ntp.org
#系统时间写入硬件时间
[root@master ~]# hwclock --systohc
4.lrzsz安装

文件上传（rz）下载（sz），在Xshell中通过rz调出上传文件的窗口，sz文件名下载某一个文件

[root@master ~]# yum -y install lrzsz
#上传
[root@master ~]# rz
5.vim安装

系统自带vi编辑器，要使用vim编辑器需要下载

[root@master ~]# yum -y install vim

三、配置SSH免密登录

1.修改hosts文件
[root@master ~]# vim /etc/hosts
#删除或注释
#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1       localhost localhost.localdomain localhost6 localhost6.localdomain6

#添加地址主机映射
192.168.100.200 master
192.168.100.201 slave1
192.168.100.202 slave2
2.给每个节点生成.ssh
#生成.ssh文件夹，然后选择no或者直接Ctrl+C退出
[root@master ~]# ssh master
#查看是否生成.ssh文件夹
[root@master ~]# ll -a
总用量 36
dr-xr-x---.  4 root root  188 7月  28 14:38 .
dr-xr-xr-x. 17 root root  224 7月  27 18:59 ..
-rw-------.  1 root root 1339 7月  27 19:01 anaconda-ks.cfg
-rw-------.  1 root root 2045 7月  27 23:24 .bash_history
-rw-r--r--.  1 root root   18 12月 29 2013 .bash_logout
-rw-r--r--.  1 root root  176 12月 29 2013 .bash_profile
-rw-r--r--.  1 root root  176 12月 29 2013 .bashrc
-rw-r--r--.  1 root root  100 12月 29 2013 .cshrc
drwxr-xr-x.  2 root root   40 7月  27 20:47 .oracle_jre_usage
drwx------.  2 root root   80 7月  27 20:23 .ssh
-rw-r--r--.  1 root root  129 12月 29 2013 .tcshrc
-rw-------.  1 root root 4857 7月  28 14:38 .viminfo
3.给每个节点生成公钥私钥
#进入.ssh目录进行操作
[root@master ~]# cd ./.ssh
#生成公钥和私钥
[root@master .ssh]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
#回车
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:sl6sgAPPdHe2Ky4ocjkzUd0Sw3tKWcUoPc3CacCSMkI root@master
The key's randomart image is:
+---[RSA 2048]----+
|.E   +.+ B.      |
|. o o * X +      |
| . o o X o       |
|    . * o        |
|. ....o+S        |
| =.o ..* .       |
|  =+. . +        |
|o B..o.o .       |
|.o + o+..        |
+----[SHA256]-----+
#查看生成的公钥和私钥
[root@master .ssh]# ls
id_rsa  id_rsa.pub
4.选择某一个节点操作
#将.ssh文件下的公钥id_rsa.pub内容复制到authorized_keys文件中
[root@master .ssh]# cp id_rsa.pub authorized_keys 
#查看
[root@master .ssh]# ls
authorized_keys  id_rsa  id_rsa.pub
5.操作其它节点
#将其它节点的公钥id_rsa.pub也复制到此authorized_keys文件中

#slave1
[root@slave1 .ssh]# cat ~/.ssh/id_rsa.pub | ssh root@master 'cat >> ~/.ssh/authorized_keys'

The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
root@master's password: 

#slave2
[root@slave2 .ssh]# cat ~/.ssh/id_rsa.pub | ssh root@master 'cat >> ~/.ssh/authorized_keys'

The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
root@master's password:
6.authorized_keys文件信息
#查看authorized_keys文件
[root@master .ssh]# more authorized_keys 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNg57gsLDFUfBJEHF/KRTMCRObUPEaeqvZu9fR6c2Z
hBuaJcGiz9A5Z1eQpyEEw3udFRCjrKk7TehnJ1PajnjaT26OXJr2FebtUF7l1mPGVHExgYPj6xwFivG8
Zta8JECzrxY/CywrTasFq3N3gTD7x/9tW3Xrz0jsd6EeKUJdeDGiY8d7jw0IWsGTjmjjq+BGve4newAa
s9QnTAtp2APga1MSDSOLVYGLGkUOYa9ohAePQi6puC7RnOT5d/bVrqcoH6rBT/t9dGQOggeqyY3WYPTT
CfpN6yCbvMMi56mOtg/XEcEfdbJYsvSHHLGbao4gEGWeDaf3FT6J/OrW2UF1 root@master

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWB8cAg7WQrvZIYMrISnDSKemiQb9U7mcakz1AF1+A
5FENrZ1yRBJ0cHlfhKGLzGqzXqi8l6pwtk3V7B3YGSGNNIxhPYLvstVspWrnUStonBy1OLOjDvVL3dLk
+UtERR4HaqsTJk7Px1B4Bu6qpXW19NJjVryObPzeuE7e/Kfl2IfnFn47HKgNNVrPVwmjXPzndUSbVYcB
nwv3C+iB6otPQYG6koW8TaI/hctXxPGJ91bhaW4MgM7iSigOiXoZA3TX07QD1RZnAi2WK2P7Ds+qToP2
NgmezuQFdmR82RcsNmYQFxphwrFYNba4BP9cDwYYbg4P6KDXiBik/hB6O5+d root@slave2

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYIcB2/DaCl+971eENOBOVZIw74FxemGFPFgVunw
ZZL1RQF9bybzRbUpU3fZvY6DcaRlq8KtD9ClN0ol1bUBJVuWsT8rS95kw5GQyhWwui5b+JFs80qFC8yM
AYzlYUx5Ok2ofYkz88IeUblLmdigOQNCpQg/hG7Y0GGEC0bNaVmScvqWiE3+O3YGqtD4aXu5XY9IblRt
RMNXupGCwwYI57VgXP6gFFWWePcaEcyZWGT/XNfrumz8cABxq3J/prfSB5i6Gipo+13Q+hMGNzzzGhnp
tXhDoAQKiCgsPNNVkLPxiJliQvtIEaplB31r5oTW2WyWMSphY/kc/50y3Oq3 root@slave1
7.将此authorized_keys文件共享给其它节点
[root@master .ssh]# scp ~/.ssh/authorized_keys root@slave1:~/.ssh/
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
root@slave1's password: 
authorized_keys                               100% 1179   434.4KB/s   00:00    

[root@master .ssh]# scp ~/.ssh/authorized_keys root@slave2:~/.ssh/
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
root@slave2's password: 
authorized_keys                               100% 1179   381.6KB/s   00:00
8.在每个节点之间进行ssh免密登录操作，包括自己与自己
[root@master .ssh]# ssh master
The authenticity of host 'master (192.168.100.200)' can't be established.
ECDSA key fingerprint is SHA256:O2tZj1XfJ6upjXbjTa2YATYJgXVjYcMwVa3APH5+6qg.
ECDSA key fingerprint is MD5:66:3c:28:31:6d:a9:d7:60:1f:be:7d:2d:63:89:36:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.100.200' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 14:38:11 2018 from master

[root@master ~]# ssh slave1
Last login: Sat Jul 28 08:53:58 2018 from 192.168.100.1

[root@slave1 ~]# ssh slave1
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:40:24 2018 from master

[root@slave1 ~]# ssh slave2
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 08:53:59 2018 from 192.168.100.1

[root@slave2 ~]# ssh slave2
The authenticity of host 'slave2 (192.168.100.202)' can't be established.
ECDSA key fingerprint is SHA256:futhvXj6JkULXb9xcO54CZuA60xcWvKaY3NZ5SAu4Vk.
ECDSA key fingerprint is MD5:1f:40:66:17:5a:4e:b0:5d:ba:8c:78:53:be:75:1f:03.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,192.168.100.202' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:41:00 2018 from slave1

[root@slave2 ~]# ssh slave1
The authenticity of host 'slave1 (192.168.100.201)' can't be established.
ECDSA key fingerprint is SHA256:8YTgPFbLIp322KNxWivNLz5LxmytmsF/pBt/FADTHvk.
ECDSA key fingerprint is MD5:72:5c:57:e0:75:75:83:4f:8d:90:ea:52:64:f1:fa:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.100.201' (ECDSA) to the list of known hosts.
Last login: Sat Jul 28 15:40:38 2018 from slave1

[root@slave1 ~]# ssh master
Last login: Sat Jul 28 15:40:01 2018 from master

[root@master ~]# ssh slave2
Last login: Sat Jul 28 15:41:11 2018 from slave2

[root@slave2 ~]# ssh master
Last login: Sat Jul 28 15:41:57 2018 from slave1

四、安装配置java

选某一个节点操作，然后复制到其它节点

#创建指定目录
[root@master ~]# mkdir ‐p /opt/app/Java
1.安装jdk
#进入到Java目录
[root@master ~]# cd /opt/app/Java
#rz上传jdk压缩包
[root@master Java]# rz
#查看
[root@master Java]# ls
jdk-8u172-linux-x64.tar.gz
#解压到当前目录
[root@master Java]# tar ‐zxvf jdk-8u172-linux-x64.tar.gz
#查看
[root@master Java]# ls
jdk1.8.0_172  jdk-8u172-linux-x64.tar.gz
2.配置jdk环境变量
#进入系统配置文件
[root@master Java]# vim /etc/profile
#java环境变量
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172
export JRE_HOME=/opt/app/Java/jdk1.8.0_172/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
#使设置立即生效
[root@master Java]# . /etc/profile
# 或者 source /etc/profile
#检测配置是否成功
[root@master Java]# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

[root@master Java]# javac -version
javac 1.8.0_172

五、安装配置hadoop

#创建指定目录
[root@master ~]# mkdir ‐p /opt/app/Hadoop
1.安装hadoop
#进入到Hadoop目录
[root@master ~]# cd /opt/app/Hadoop
#rz上传jdk压缩包
[root@master Hadoop]# rz
#查看
[root@master Hadoop]# ls
hadoop-2.7.6.tar.gz
#解压到当前目录
[root@master Hadoop]# tar ‐zxvf hadoop-2.7.6.tar.gz
#查看
[root@master Hadoop]# ls
hadoop-2.7.6  hadoop-2.7.6.tar.gz
2.创建备用目录
#进入hadoop‐2.7.6目录
[root@master Hadoop]# cd hadoop‐2.7.6
#创建临时文件目录
[root@master hadoop‐2.7.6]# mkdir tmp
#创建日志信息目录
[root@master hadoop‐2.7.6]# mkdir logs
#创建hdfs名称空间
[root@master hadoop‐2.7.6]# mkdir ‐p hdfs/name
#创建hdfs数据空间
[root@master hadoop‐2.7.6]# mkdir ‐p hdfs/data

#查看
[root@master hadoop-2.7.6]# ls
bin  hdfs     lib      LICENSE.txt  NOTICE.txt  sbin   tmp
etc  include  libexec  logs         README.txt  share
3.修改配置文件
编辑slaves文件
#该文件记录的是集群里的数据节点
[root@master hadoop-2.7.6]# vi etc/hadoop/slaves
#删除原来内容，添加以下内容
slave1
slave2
编辑hadoop-env.sh文件
[root@master hadoop-2.7.6]# vim etc/hadoop/hadoop-env.sh 
#修改JAVA_HOME为绝对路径
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172
编辑yarn-env.sh文件
[root@master hadoop-2.7.6]# vim etc/hadoop/yarn-env.sh 
#修改JAVA_HOME为绝对路径
export JAVA_HOME=/opt/app/Java/jdk1.8.0_172
编辑core-site.xml文件
[root@master hadoop-2.7.6]# vim etc/hadoop/core-site.xml 
<configuration>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://master:9000</value>
    </property>
    <property>
            <name>hadoop.tmp.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/tmp</value>
    </property>
    <property>
            <name>io.file.buffer.size</name>
            <value>131702</value>
    </property>
</configuration>
编辑hdfs-site.xml文件
[root@master hadoop-2.7.6]# vim etc/hadoop/hdfs-site.xml
<configuration>
    <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/hdfs/name</value>
    </property>
    <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:/opt/app/Hadoop/hadoop-2.7.6/hdfs/data</value>
    </property>
    <property>
            <name>dfs.replication</name>
            <value>2</value>
    </property>
    <property>
            <name>dfs.namenode.secondary.https-address</name>
            <value>master:50070</value>
    </property>
    <property>
            <name>dfs.webhdfs.enabled</name>
            <value>true</value>
    </property>
</configuration>
编辑yarn-site.xml文件
[root@master hadoop-2.7.6]# vim etc/hadoop/yarn-site.xml 
<configuration>
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
            <name>yarn.resourcemanager.address</name>
            <value>master:8032</value>
    </property>
    <property>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>master:8030</value>
    </property>
    <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>master:8031</value>
    </property>
    <property>
            <name>yarn.resourcemanager.admin.address</name>
            <value>master:8033</value>
    </property>
    <property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>master:8088</value>
    </property>
    <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>1024</value>
    </property>
</configuration>
编辑mapred-site.xml文件
[root@master hadoop-2.7.6]# vim etc/hadoop/mapred-site.xml
<configuration>
    <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>
    <property>
            <name>mapreduce.jobhistory.address</name>
            <value>master:10020</value>
    </property>
    <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>master:19888</value>
    </property>
</configuration>
4.配置hadoop环境变量
[root@master hadoop-2.7.6]# vim /etc/profile
#hadoop环境变量
export HADOOP_HOME=/opt/app/Hadoop/hadoop-2.7.6
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
#使设置立即生效
[root@master hadoop-2.7.6]# . /etc/profile

六、文件远程拷贝

[root@master hadoop‐2.7.6]# scp ‐r /opt/app root@slave1:/opt/
[root@master hadoop‐2.7.6]# scp ‐r /opt/app root@slave2:/opt/

[root@master hadoop‐2.7.6]# scp ‐r /etc/profile root@slave1:/etc/
[root@master hadoop‐2.7.6]# scp ‐r /etc/profile root@slave2:/etc/

七、格式化Hadoop

[root@master hadoop‐2.7.6]# bin/hdfs namenode ‐format

八、启动Hadoop集群

不建议直接使用start‐all.sh命令

#先启动HDFS
[root@master hadoop‐2.7.6]# sbin/start‐dfs.sh

#然后启动YARN
[root@master hadoop‐2.7.6]# sbin/start‐yarn.sh

九、检测集群是否搭建成功

[root@master hadoop-2.7.6]# jps
16277 ResourceManager
16346 Jps
16124 SecondaryNameNode
15933 NameNode
[root@slave1 hadoop-2.7.6]# jps
11105 Jps
10904 DataNode
11004 NodeManager
[root@slave2 hadoop-2.7.6]# jps
11107 Jps
10906 DataNode
11006 NodeManager

Web登录访问：下面的图是再次搭建的时候截的，仅仅登录地址不一样而已
192.168.100.200:50070
这里写图片描述
192.168.100.200:8088
这里写图片描述

猜你喜欢

转载自blog.csdn.net/qq_41610418/article/details/81407721
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
Java自定义时间格式
同步整形电路
在开发中最最最常用的字符串的属性大集合
Linux 查看端口占用并杀掉
Java基础四：ArrayList
多线程之死锁就是这么简单
mysql 基础命令集
awk 命令详解
Centos6.3编译安装nginx+php步骤
OCR （Optical Character Recognition，光学字符识别）
每日归档
更多
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022