Big data cluster construction (jdk, hadoop, hive, mysql, spark, flume, zookeeper)

Cluster environment

table of Contents	version
system	centos7
jdk	1.8
hadoop	2.6.1
catheter	$1

Configure static IP address
Command: vi /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.248.10
NETMASK=255.255.255.0
GATEWAY=192.168.248.2
DNS1=1.2.4.8
DNS2=210.2.4.8

Then restart the network
command: systemctl restart network
ping www.baidu.com

Then you can use xshell to operate Linux

First install a vim command
yum install vim -y

Edit the hosts file
Command: vim /etc/hosts

192.168.248.10 master
192.168.248.11 slave1
192.168.248.12 slave2

Configure the network name
Command: vim /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=master

hostname master

Change the name
vim /etc/hostname

master

Turn off network management
systemctl stop NetworkManager
systemctl disable NetworkManager

Turn off the Linux kernel firewall
vim /etc/selinux/config

 SELINUX=enforcing 改为 SELINUX=disabled

Empty the kernel firewall
setenforce 0

Turn off the firewall
systemctl stop firewalld

Prohibit booting
systemctl disable firewalld

View the status of the
firewall firewall-cmd --state

I put the software to be installed under the path /usr/local/src

Edit the global configuration file
vim ~/.bashrc

export JAVA_HOME=/usr/local/src/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib
export HADOOP_HOME=/usr/local/src/hadoop-2.6.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Let the configuration file take effect immediately
source ~/.bashrc

In order to verify whether the above command is correct, you can enter a few commands in any path to test whether the global configuration file is effective

Then shut down and delete these unused ones.
Insert picture description here
Then find the master installation path, copy its folder
and open this virtual machine in VMware.
The host name of the copied virtual machine will conflict. Let's modify the host name first, as shown in the figure below. Show

Insert picture description here Open the virtual machine and select the copy option. After

opening, you need to modify the following configurations, and slave1 is configured like this

vim /etc/sysconfig/network-scripts/ifcfg-ens33   修改IP地址
vim /etc/hostname
hostname slave1
vim /etc/sysconfig/network

systemctl restart network
ping www.baidu.com

Configure password-free login, and then you can start and operate the cluster

在三台机器上都执行命令，
ssh-keygen -t rsa   执行完后连敲三个回车就可以了

执行完成后，会在这个目录下/root/.ssh，生成两个文件
-rw------- 1 root root 1675 3月  21 01:20 id_rsa
-rw-r--r-- 1 root root  393 3月  21 01:20 id_rsa.pub

三台机器都执行一下下面的命令
cat /root/.ssh/id_rsa.pub >/root/.ssh/authorized_keys

在master上执行
ssh slave1 cat /root/.ssh/authorized_keys >> /root/.ssh/authorized_keys
ssh slave2 cat /root/.ssh/authorized_keys >> /root/.ssh/authorized_keys

在slave1和slave2上执行
ssh master cat /root/.ssh/authorized_keys >> /root/.ssh/authorized_keys

After configuring the password-free login, we can initialize the cluster
hadoop namenode -format

You're done, you can start the cluster. The global environment variables have been configured before. Here, execute the startup command in any path,
start-all.sh

Use the jps command to view the processes on three machines, 4 master, 3 slave1 and slave2

[root@master ~]# jps
2848 Jps
2440 SecondaryNameNode
2265 NameNode
2589 ResourceManager

[root@slave1 ~]# jps
1928 Jps
1820 NodeManager
1727 DataNode

After the startup is successful, you can view the visualization page on the Windows page, but there is some trouble entering the IP address. We can go to the hosts file and add the following configuration. Later, you can access the cluster with the latter domain name
. Modify the hosts under this path File
C:\Windows\System32\drivers\etc
The hosts file can be backed up, modified on the desktop
Insert picture description here and tested on the browser

Insert picture description here Access to
several HDFS machines is actually associated through cluserid, you can see the VERSION file in the following directory

master下
/usr/local/src/hadoop-2.6.1/dfs/name/current
slave下
/usr/local/src/hadoop-2.6.1/dfs/data/current

If the cluster fails, you can pause the cluster first, then delete the following directory (all three machines must do this), and then reformat hadoop namenode -format to
Insert picture description here
install MySQL
Centos7 comes with MySQL, you need to uninstall the built-in MySQL first

查看命令：
rpm -qa | grep mariadb
卸载命令：
rpm -e --nodeps mariadb-libs-5.5.65-1.el7.x86_64

Everyone's centos environment may be different. My installation steps are like this. Note that the installation is in order

1.rpm -Uvh mysql-community-common-5.7.28-1.el7.x86_64.rpm
2.rpm -Uvh mysql-community-libs-5.7.28-1.el7.x86_64.rpm
3.rpm -Uvh mysql-community-client-5.7.28-1.el7.x86_64.rpm
4.rpm -Uvh mysql-community-server-5.7.28-1.el7.x86_64.rpm

Install hive
first configure two configuration files

Put the MySQL driver jar package in this path
mv mysql-connector-java-5.1.25-bin.jar /usr/local/src/apache-hive-1.2.2-bin/lib

Put the jar package in hive into the Hadoop directory
/usr/local/src/hadoop-2.6.1/share/hadoop/yarn/lib

hive jar package path
/usr/local/src/apache-hive-1.2.2-bin/lib

Hive jar package path
Insert picture description here Hadoop path
configuration global configuration file

export JAVA_HOME=/usr/local/src/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib
export HADOOP_HOME=/usr/local/src/hadoop-2.6.1
export HIVE_HOME=/usr/local/src/apache-hive-1.2.2-bin
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin

Let the configuration file take effect immediately
source ~/.bashrc

Detail record: (placed under the building document)
Notes for installing centos7,
Insert picture description here select the minimal installation, and check the development tools

Step 1: How to find the historical version of Hadoop and
Insert picture description here click to download

Insert picture description here Click on the historical version and select 2.6.1

Insert picture description here
Install MySQL details

[root@master mysql]# ll
总用量 252444
-rw-r--r-- 1 root root  45109364 3月  20 23:49 mysql-community-client-5.7.28-1.el7.x86_64.rpm
-rw-r--r-- 1 root root    318768 3月  20 23:49 mysql-community-common-5.7.28-1.el7.x86_64.rpm
-rw-r--r-- 1 root root   4374364 3月  20 23:49 mysql-community-libs-5.7.28-1.el7.x86_64.rpm
-rw-r--r-- 1 root root 208694824 3月  20 23:49 mysql-community-server-5.7.28-1.el7.x86_64.rpm
[root@master mysql]# rpm -qa | grep mariadb
mariadb-libs-5.5.65-1.el7.x86_64
[root@master mysql]# rpm -e --nodeps mariadb-libs-5.5.65-1.el7.x86_64
[root@master mysql]# rpm -qa | grep mariadb
[root@master mysql]# rpm -Uvh mysql-community-common-5.7.28-1.el7.x86_64.rpm
警告：mysql-community-common-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-common-5.7.28-1.e################################# [100%]
[root@master mysql]# rpm -Uvh mysql-community-libs-5.7.28-1.el7.x86_64.rpm
警告：mysql-community-libs-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-libs-5.7.28-1.el7################################# [100%]
[root@master mysql]# rpm -Uvh mysql-community-client-5.7.28-1.el7.x86_64.rpm
警告：mysql-community-client-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-client-5.7.28-1.e################################# [100%]
[root@master mysql]# rpm -Uvh mysql-community-server-5.7.28-1.el7.x86_64.rpm
警告：mysql-community-server-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
错误：依赖检测失败：
	net-tools 被 mysql-community-server-5.7.28-1.el7.x86_64 需要
[root@master mysql]# yum install net-tools -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
正在解决依赖关系
--> 正在检查事务
---> 软件包 net-tools.x86_64.0.2.0-0.25.20131004git.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

===========================================================================================================
 Package                架构                版本                                   源                 大小
===========================================================================================================
正在安装:
 net-tools              x86_64              2.0-0.25.20131004git.el7               base              306 k

事务概要
===========================================================================================================
安装  1 软件包

总下载量：306 k
安装大小：917 k
Downloading packages:
net-tools-2.0-0.25.20131004git.el7.x86_64.rpm                                       | 306 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
警告：RPM 数据库已被非 yum 程序修改。
** 发现 2 个已存在的 RPM 数据库问题， 'yum check' 输出如下：
2:postfix-2.10.1-9.el7.x86_64 有缺少的需求 libmysqlclient.so.18()(64bit)
2:postfix-2.10.1-9.el7.x86_64 有缺少的需求 libmysqlclient.so.18(libmysqlclient_18)(64bit)
  正在安装    : net-tools-2.0-0.25.20131004git.el7.x86_64                                              1/1 
  验证中      : net-tools-2.0-0.25.20131004git.el7.x86_64                                              1/1 

已安装:
  net-tools.x86_64 0:2.0-0.25.20131004git.el7                                                              

完毕！
[root@master mysql]# rpm -Uvh mysql-community-server-5.7.28-1.el7.x86_64.rpm
警告：mysql-community-server-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-server-5.7.28-1.e################################# [100%]

Big data cluster construction (jdk, hadoop, hive, mysql, spark, flume, zookeeper)

Guess you like