Hive2.1.1、Hadoop2.7.3 部署

本文以远程模式安装Hive2.1.1将hive的元数据放置在mysql数据库中。

1 安装mysql数据库

sudo apt-get install mysql-server

用户名:root密码:123456
修改支持远程访问

重启mysql服务使得配置文件生效

sudo service mysql restart

创建hive专用账户

 CREATE USER 'hive'@'%' IDENTIFIED BY '123456';

给hive账户授予所有权限

grant all privileges on *.* to 'hive'@'%' identified by '123456' with grant option;

刷新系统权限表，使配置生效

flush privileges;

2 解压安装hive

cd /usr/local
sudo tar -xvzf apache-hive-2.1.1-bin.tar.gz
sudo mv apache-hive-2.1.1-bin/ hive-2.1.1

配置系统环境变量

sudo gedit .bashrc
export HIVE_HOME=/usr/local/hive-2.1.1
exportPATH=$HIVE_HOME/bin:$HIVE_HOME/lib:$PATH

使得环境变量配置生效

source .bashrc

3 配置hive
3.1 修改conf/hive-env.sh文件

cd /usr/local/hive-2.1.1/conf/
sudo cp hive-env.sh.template hive-env.sh
sudo chown hadoop:hadoop hive-env.sh
sudo gedit hive-env.sh
HADOOP_HOME=/usr/local/hadoop-2.7.3
export HIVE_CONF_DIR=/usr/local/hive-2.1.1/conf
export HIVE_AUX_JARS_PATH=/usr/local/hive-2.1.1/lib

3.2 修改日志属性文件配置日志存储目录
修改hive-log4j2.properties

sudo cp hive-log4j2.properties.template hive-log4j2.properties
sudo chown hadoop:hadoop hive-log4j2.properties
sudo gedit hive-log4j2.properties
property.hive.log.dir = /usr/local/hive-2.1.1/logs

修改llap-cli-log4j2.properties

property.hive.log.dir = /usr/local/hive-2.1.1/logs
property.hive.log.file = llap-cli.log

3.3 修改hive-site.xml配置文件，主要修改如下配置项目

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/usr/local/hive-2.1.1/tmp</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
 <property>
    <name>hive.querylog.location</name>
    <value>/usr/local/hive-2.1.1/logs</value>
    <description>Location of Hive run time structured log file</description>
  </property>
  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/usr/local/hive-2.1.1/logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>
 <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/usr/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>
<property>
    <name>hive.metastore.uris</name>
    <value>thrift://192.168.80.130:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>
<property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
  </property>
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/usr/local/hive-2.1.1/tmp</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.80.130:3306/metastore?createDatabaseIfNotExist=true&amp;useSSL=false</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>123456</value>
    <description>password to use against metastore database</description>
  </property>
<property>
    <name>hive.hwi.listen.host</name>
    <value>0.0.0.0</value>
    <description>This is the host address the Hive Web Interface will listen on</description>
  </property>
  <property>
    <name>hive.hwi.listen.port</name>
    <value>9999</value>
    <description>This is the port the Hive Web Interface will listen on</description>
  </property>
 <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>0.0.0.0</value>
    <description>Bind host on which to run the HiveServer2 Thrift service.</description>
  </property>

4 拷贝mysql连接包到hive主目录下的lib中

sudo mv ~/下载/mysql-connector-java-5.1.40-bin.jar /usr/local/hive-1.2.1/lib/

5 配置Hive的hwi网页访问方式
下载hive-2.1.1源码包

wget http://www-us.apache.org/dist/hive/hive-2.1.1/ apache-hive-1.2.1-src.tar.gz
tar -zxvf apache-hive-2.1.1-src.tar.gz  
cd apache-hive-2.1.1-src  
cd hwi/web  
zip hive-hwi-2.1.1.zip ./*
mv hive-hwi-2.1.1.zip hive-hwi-2.1.1.war
mv hive-hwi-2.1.1.war $HIVE_HOME/lib

拷贝tools包

sudo cp /usr/lib/jdk1.8.0_121/lib/tools.jar /usr/local/hive-2.1.1/lib

删除lib下的ant-1.6.5.jar，否则浏览hwi网页时会显示错误信息，需要刷新两次才能看到网页。
6 初始化hive

schematool -dbType mysql -initSchema

7 启动hive服务
启动metaStore服务

hive --service metastore  &

启动hive web界面

hive --service  hwi &

启动thrift2服务

hive --service hiveserver2 &

启动hive shell

hive

hwi访问网址

http://localhost:9999/hwi/