Hadoop3 cluster construction - virtual machine installation
Hadoop3 cluster building - install hadoop, configure the environment
Hadoop3 cluster construction - configure ntp service
Now to hive.
Hive installation is relatively simple.
Download a package, unzip it, and configure hive-site.xml and hive-env.sh.
1. Download the hive package
Official website: http://mirror.bit.edu.cn/apache/hive/hive-2.3.3/
2. Unzip to the hadoop directory
tar -zxvf apache-hive- 2.3 . 3 -bin. tar .gz #Unzip mv apache-hive- 2.3 . 3 -bin hive2. 3.3 #Modify the directory name for convenience
3. Configure hive environment variables
[hadoop@venn05 ~]$ more .bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi # Uncomment the following line if you don't like systemctl's auto-paging feature: # export SYSTEMD_PAGER= # User specific aliases and functions #jdk export JAVA_HOME=/opt/hadoop/jdk1.8 export JRE_HOME=${JAVA_HOME}/jre export CLASS_PATH=${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH #hadoop export HADOOP_HOME=/opt/hadoop/hadoop3 export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH #hive export HIVE_HOME=/opt/hadoop/hive2.3.3 export HIVE_CONF_DIR=$HIVE_HOME/conf export PATH=$HIVE_HOME/bin:$PATH
4. Create a hive directory on hdfs
The hive working directory is as follows:
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description>
</property>
So create the following directory:
hadoop fs -mkdir -p /user/hive/warehouse #hive library file location
hadoop fs -mkdir -p /tmp/hive/ #hive temporary directory
#授权
hadoop fs -chmod -R 777 /user/hive/warehouse
hadoop fs -chmod -R 777 /tmp/hive
Note: You must authorize, otherwise an error will be reported:
Logging initialized using configuration in jar:file:/opt/hadoop/hive2.3.3/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true Exception in thread "main" java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:720) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:650) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:549) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:750) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:239) at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
5. Modify hive-site.xml
cp hive-default.xml.template hive-site.xml vim hive-site.xml
Modification 1: Cache "${system:java.io.tmpdir}" in hive-site.xml to the specific directory: /opt/hadoop/hive2.3.3/tmp 4
Modification 2: Cache "${system:user.name}" in hive-site.xml to the specific directory: root 3
<property>
<name>hive.exec.local.scratchdir</name>
<value>${system:java.io.tmpdir}/${system:user.name}</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<value>${system:java.io.tmpdir}/${system:user.name}</value>
<description>Location of Hive run time structured log file</description>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
<description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>
Change it to:
<property>
<name>hive.exec.local.scratchdir</name>
<value>/opt/hadoop/hive2.3.3/tmp/root</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/opt/hadoop/hive2.3.3/tmp/${hive.session.id}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<value>/opt/hadoop/hive2.3.3/tmp/root</value>
<description>Location of Hive run time structured log file</description>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/opt/hadoop/hive2.3.3/tmp/root/operation_logs</value>
<description>Top level directory where operation logs are stored if logging functionality is enabled</description>
Configure the metabase mysql:
mysql> CREATE USER ' hive ' @ ' % ' IDENTIFIED BY ' hive ' ; #Create a hive user Query OK, 0 rows affected (0.00 sec) mysql > GRANT ALL ON *.* TO ' hive ' @ ' % ' ; #authorization Query OK, 0 rows affected (0.00 sec)
Modify the database configuration:
<!-- mysql 驱动 -->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<!-- 链接地址 -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://venn05:3306/hive?createDatabaseIfNotExist=true</value>
<description>
JDBC connect string for a JDBC metastore.
To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description><property><!-- username-->
</property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>Username to use against metastore database</description>
</property>
<!-- 密码 -->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<description>password to use against metastore database</description>
</property>
6. Modify hive-env.sh
cp hive-env.sh.template hive-env.sh vim hive-env.sh
Add the following at the end:
export HADOOP_HOME=/opt/hadoop/hadoop3 export HIVE_CONF_DIR=/opt/hadoop/hive2.3.3/conf export HIVE_AUX_JARS_PATH=/opt/hadoop/hive2.3.3/lib
7. Upload the mysql driver package
Upload to: $HIVE_HOME/lib
8. Initialize hive
schematool -initSchema -dbType mysql
9. Start hive
hive
get it