Hive集群安装(3个节点)-基于Hadoop

接我上一篇文章《Hadoop集群安装(3个节点)》,写上篇文章以及本篇文章的原因:使用NiFi解决两个需求,

需求一:导出MySQL数据转换为Json串并保存到本地

需求二:导出Hive数据转换为csv并保存到HDFS

需求一还是很容易实现的,只需要安装NiFi(下篇文件介绍安装&使用)、MySQL就可以实现了。

由于是在外做项目(青岛RRS物流),没有在公司那么方便,虽然公司也有机房,有环境,但是资源比较紧张,我就在自己的移动硬盘上安装了3个节点

在做需求二时,发现新搭建的虚拟机集群,只是简易的安装了jdk、NiFi、MySQL,其他的没有了,不具备实现需求二的环境,所以就在3个节点上安装Hadoop集群、hive集群。

下面不废话了,开始介绍hive的安装,顺带介绍一下MySQL的安装

1. 安装hive

安装在hadoop的namenode上(也就是主节点上:node4上),上传安装文件到(可以根据自己的环境,目录可以自定义)/opt/software/apache-hive-2.1.0-bin.tar.gz

解压:

tar –zxvf apache-hive-2.1.0-bin.tar.gz

添加到环境变量

vi /etc/profile
编辑
#hive
export HIVE_HOME=/opt/module/hive/apache-hive-2.1.0-bin
export PATH=$PATH:$HIVE_HOME/bin
保存后使其生效:
source /etc/profile

2. 安装MySQL

安装mysql作为hive的Metastore(主节点安装即可

为了yum顺利进行,配置/etc/resolv.conf,删除原来的nameserver,添加

nameserver 192.168.1.1
nameserver 114.114.114.114

安装perl和numactl

yum install -y perl
yum install -y numactl

卸载已有的mysql-libs

rpm -qa | grep mysql
rpm -qa | grep maria
rpm -e --nodeps 【包名】

上传mysql的jar包:rz

解压:

tar xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar -C /opt/module/mysql/

因为这里不是”.tar.gz”文件,所以使用参数“xvf”,如果是”.tar.gz”文件,使用“zxvf”

rpm安装mysql

rpm -ivh mysql-community-common-5.7.21-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.21-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.21-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm

如果在rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm这一步报错:

(可能是因为我在VMware安装的是最小化版本,所以很多工具没有,需要自己下载)

[root@node4 mysql]# rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm

警告:mysql-community-server-5.7.21-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY

错误:依赖检测失败:

       net-tools 被 mysql-community-server-5.7.21-1.el7.x86_64 需要

下载net-tools即可:

[root@node4 mysql]# yum install net-tools

再次rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm即可成功:

设置开机启动

chkconfig mysqld on

启动mysql服务

service mysqld start

关闭密码校验

echo "validate_password=off" >> /etc/my.cnf

重启mysql服务

service mysqld restart

查看root的初始密码

cat /var/log/mysqld.log | more | grep password

修改 mysql root密码

mysql -uroot -p[查看得到的密码]

[root@localhost ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.21
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> set password='123qwe';
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye

mysql默认不可以远程访问,设置远程访问

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'WITH GRANT OPTION;

上面这句远程访问不需要密码,如果需要密码使用下面这句

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY '123qwe' WITH GRANT OPTION;

使权限生效:

FLUSH PRIVILEGES;

3. 配置hive

1. 在hdfs中新建目录/user/hive/warehouse

hdfs dfs -mkdir /tmp
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/hive
hdfs dfs -mkdir /user/hive/warehouse
hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse

将mysql的驱动jar包mysql-connector-java-5.1.47-bin.jar拷入hive的lib目录下面: /opt/module/hive/apache-hive-2.1.0-bin/lib/ mysql-connector-java-5.1.47.jar

2. 进入hive的conf目录下面复制一下hive-default.xml.template名子命名为:hive-site.xml

cp hive-default.xml.template hive-site.xml
vim hive-site.xml
(1)、<property>
         <name>javax.jdo.option.ConnectionURL</name>
         <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
         <description>JDBC connect string for a JDBC metastore</description>
      </property>

(2)、<property>
         <name>javax.jdo.option.ConnectionDriverName</name>
         <value>com.mysql.jdbc.Driver</value>
         <description>Driver class name for a JDBC metastore</description>
     </property>

(3)、<property>
         <name>javax.jdo.option.ConnectionUserName</name>
         <value>root</value>
         <description>Username to use against metastore database</description>
      </property>

(4)、 <property>
         <name>javax.jdo.option.ConnectionPassword</name>
         <value>123qwe</value>
         <description>password to use against metastore database</description>
       </property>

(5)、<property>
         <name>hive.exec.local.scratchdir</name>
         <value>/opt/module/hive/apache-hive-2.1.0-bin /tmp</value>
         <description>Local scratch space for Hive jobs</description>
      </property>

 (6)、<property>
          <name>hive.downloaded.resources.dir</name>
          <value>/opt/module/hive/apache-hive-2.1.0-bin /tmp/resources</value>
          <description>Temporary local directory for added resources in the remote file system.</description>
       </property>

 (7)、<property>
          <name>hive.querylog.location</name>
          <value>/opt/module/hive/apache-hive-2.1.0-bin /tmp</value>
          <description>Location of Hive run time structured log file</description>
       </property>

(8)、<property>
         <name>hive.server2.logging.operation.log.location</name>
         <value>/opt/module/hive/apache-hive-2.1.0-bin/tmp/operation_logs</value>
         <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
      </property>

3. 复制hive文件到node5、node6节点:

scp –r /opt/module/hive/ node5:/opt/module/
scp –r /opt/module/hive/ node6:/opt/module/

4. 在node4上使用schematool 初始化metastore的schema:

schematool -initSchema -dbType mysql

5. 运行hive

[root@node4 mysql]# hive

mysql/hive字符集问题:hive中文乱码

alter database hive character set latin1;
set character_set_client=latin1;

也可以使用beeline命令,以后有时间再做介绍。

猜你喜欢

转载自blog.csdn.net/Allenzyg/article/details/105762896
今日推荐