cent0S 7 安装 cdh5.13 笔记之三：
安装 hadoop-hdfs
------------------------------------------------------------------------
pkgname | n1 | n2 | n3 | pkgtype
------------------------------------------------------------------------
#### common ####
net-tools (ALL)
gcc (ALL)
perl (ALL)
jdk-8u152-linux-x64.tar.gz (ALL)
zookeeper-server Y Y Y yum
cyrus-sasl ? ? ? yum
cyrus-sasl-devel ? ? ? yum
kafka_2.11-1.0.0.tgz Y Y Y
#### mysql ####
perl Y yum
mysql-community-common-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-libs-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-client-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-server-5.7.20-1.el7.x86_64.rpm
Y rpm
#### hadoop ####
hadoop-hdfs-namenode Y Y N yum
hadoop-hdfs-secondarynamenode* X X X yum (不要安装）
hadoop-hdfs-datanode Y Y Y + yum
hadoop-hdfs-journalnode Y Y Y yum (奇数： 3,5,7,..., 资源消耗低）
hadoop-mapreduce Y Y Y yum
hadoop-hdfs-zkfc Y Y N yum
hadoop-yarn-resourcemanager Y Y N yum （最好单独一台）
hadoop-yarn-nodemanager Y Y Y + yum
hadoop-lzo Y Y Y + yum
hadoop-mapreduce-historyserver Y N N yum
hadoop-libhdfs Y Y N yum
#### hbase ####
hbase-master Y Y N yum
hbase-regionserver Y Y Y yum
#### hive ####
hive-metastore Y N N yum
hive-server2 Y Y Y yum
hive-hbase Y Y Y yum
hive-hcatalog Y Y Y yum
hive-webhcat Y Y Y yum
hive-webhcat-server Y N N yum
#### spark ####
spark-master Y Y N yum
spark-worker Y Y Y yum
spark-history-server Y N N yum
------------------------------------------------------------------------
10 安装hadoop (cdh-5.13)组件
如上表，对pkgtype为yum的执行下面的命令：
yum install -y pkgname
例如， 对于n1, n2执行(Y - 安装, N - 不安装)：
yum install -y hadoop-hdfs-namenode hadoop-hdfs-datanode
对于n3，执行：
yum install -y hadoop-hdfs-datanode
这样就安装好了。重点是配置的过程。
11 安装mysql-server for hive
hive remote模式：
hive-metastore 安装在n1上， 需要在n3上安装mysql-server，下面是安装mysql的步骤：
1) 下载安装包
wget ftp://pub:[email protected]/rpm/mysql-community-common-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-libs-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-client-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-server-5.7.20-1.el7.x86_64.rpm
2) 安装mysql-server
a.首先卸载mariadb
# rpm -qa |grep mariadb mariadb-libs-5.5.56-2.el7.x86_64 # rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
b. 安装mysql-server
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm # yum install -y perl # rpm -ivh mysql-community-server-5.7.20-1.el7.x86_64.rpm
c. 启动 mysql
# service mysqld start
或
# systemctl start mysqld
d. 查看安装过程的临时密码：
# sudo grep 'temporary password' /var/log/mysqld.log
YVo)b8Er=g<g
d. 更改临时密码
# sudo mysql_secure_installation
New password:
Abc1234!
e. 更改mysql数据库文件路径
查看配置文件路径
# mysql --help|grep my.cnf
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
查看默认数据文件路径：
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
| datadir | /var/lib/mysql/
修改到：/data/mysql
# systemctl stop mysqld.service # mv /var/lib/mysql /data/
编辑: /etc/my.cnf， 内容如下：
[client] port = 3306 socket = /data/mysql/mysql.sock #### (必须) [mysqld] port = 3306 datadir=/data/mysql socket=/data/mysql/mysql.sock
再启动mysqld:
# systemctl start mysqld
再次查看数据文件路径：
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
| datadir | /data/mysql/
这样就把数据库文件改到/data/mysql下面了。
最后设置mysql开机启动（待测试）:
# systemctl enable mysqld.service
3) 安装mysql-client
在需要访问mysql服务的机器上安装mysql客户端， 这里是在n1上安装：
首先卸载mariadb
# rpm -qa |grep mariadb mariadb-libs-5.5.56-2.el7.x86_64 # rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
安装mysql-client
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm
4) 在n1 上访问 n3的mysql
# mysql -u root -pAbc1234!mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1130 (HY000): Host 'cent7-n1.pepstack.com' is not allowed to connect to this MySQL server
需要在n3上设置mysql可以远程访问， 回到n3上，执行：
登录到mysql: # mysql -u root -pAbc1234! 设置在n1上可以访问mysql: mysql > grant all privileges on *.* to 'root'@'cent7-n1.pepstack.com' identified by 'Abc1234!' mysql > flush privileges;
这样就可以从n1上访问n3的mysql了。
5) 配置hive使用的mysql数据库
为hive 创建mysql数据库 hive_metastore (hive-schema-1.1.0.mysql.sql):
在n3 (mysql所在的机器上)，执行：
# cd /usr/lib/hive/scripts/metastore/upgrade/mysql/ # mysql -u root -pAbc1234!
为hive创建表：
mysql> create database hive_metastore; mysql> use hive_metastore; mysql> source ./hive-schema-1.1.0.mysql.sql; mysql> quit;
在n3 上，为metastore所在的机器(n1)以hive连接到n3上的mysql做如下配置：
# mysql -u root -pAbc1234!
mysql> create user 'hive'@'cent7-n1.pepstack.com' identified by 'Abc1234!'; mysql> revoke all privileges,grant option from 'hive'@'cent7-n1.pepstack.com'; mysql> grant all privileges on hive_metastore.* to 'hive'@'cent7-n1.pepstack.com'; mysql> flush privileges; mysql> quit;
在n1上，测试访问n3:
# mysql -u hive -pAbc1234! -h cent7-n3.pepstack.com
这样，hive使用的数据库就配置成功了！