Continue the last tutorial, this time install hbase and hive. Previously, some students reported that there is no hbase installation package in Baidu network disk. This time I added it. There is already in the network disk link of the previous blog. The most detailed Hadoop+Hbase+Hive fully distributed environment construction tutorial (1) .
One, hbase2.1.1 installation
On the master node:
$ tar -zxvf hbase-2.1.1-bin.tar.gz -C /opt
$ cd /opt/hbase-2.1.1/conf/
$ vi hbase-env.sh
Add the following sentence to the end of the file
export JAVA_HOME=/opt/jdk1.8.0_102
$ vi hbase-site.xml
amend as below:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,slave1,slave2</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/fay/zookeeper</value>
</property>
</configuration>
$ vi regionservers
amend as below:
# 删掉localhost
slave1
slave2
$ vi backup-masters
This file is newly created, add the following sentence
slave1
Then is to configure environment variables:
$ vi ~/.bashrc
export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:
$ source ~/.bashrc
Then give the configuration file and the entire directory to the other two nodes:
$ scp -r hbase-2.1.1/ fay@slave1:/opt
$ scp -r hbase-2.1.1/ fay@slave2:/opt
$ scp ~/.bashrc fay@slave1:/home/fay
$ scp ~/.bashrc fay@slave1:/home/fay
Source the other two nodes
$ source ~/.bashrc
and then start hbase when Hadoop is up
# 启动hadoop
$ start-dfs.sh
$ start-yarn.sh
# 启动hbase
$ start-hbase.sh
Then the master node:
[fay@master opt]$ jps
39825 Jps
38898 ResourceManager
39398 HQuorumPeer #hbase的zookeeper实例
38487 NameNode
39495 HMaster # hbase master
38702 SecondaryNameNode
slave node:
[fay@slave1 ~]$ jps
16481 DataNode
16598 NodeManager
16886 HRegionServer # hbase regionserver
16775 HQuorumPeer
17003 HMaster # 这个是slave1，因为前面配置了backup-masters,slave2没有这个
17278 Jps
Pay attention to the time synchronization problem here. Startup may be time-consuming. Do not cancel with ctrl+c, as it may be abnormal. Ok, test it
[fay@master opt]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase-2.1.1/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.1, rb60a92d6864ef27295027f5961cb46f9162d7637, Fri Oct 26 19:27:03 PDT 2018
Took 0.0047 seconds
hbase(main):001:0> list
TABLE
wuxia
1 row(s) # 这里我之前创建了表，初始状态应该是0
Took 1.8662 seconds
=> ["wuxia"]
hbase(main):002:0> create 'hello','world'
Created table hello
Took 3.1974 seconds
=> Hbase::Table - hello
No exception, basically hbase is set up
Two, hive2.3.4 build
1. Install mysql 8.0.13 (you can use the built-in derby without installation)
Uninstall the mariadb-lib that comes with the system to
view the mariadb version
rpm -qa|grep mariadb
Uninstalling mariadb
rpm -e mariadb-libs-5.5.56-2.el7.x86_64 --nodeps
mysql has dependencies, the installation order is common, lib, client, server
sudo rpm -ivh mysql-community-common-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-client-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-server-8.0.13-1.el7.x86_64.rpm
Start mysql:
sudo systemctl start mysqld
Set the root password:
[fay@master ~]$ cat /var/log/mysqld.log
2018-11-11T09:12:06.253251Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.13) initializing of server in progress as process 20113
2018-11-11T09:12:09.192846Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: pR+ndHtvO6#d
The above pR+ndHtvO6#dis the initial root password, use this password to log in to mysql
[fay@master ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 8.0.13 MySQL Community Server - GPL
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
Then create a fay user and give him permissions
mysql> CREATE USER 'fay'@'master' IDENTIFIED BY '你的密码';
mysql> create database hive_metedata
mysql> grant all privileges on *.* to 'fay'@'master';
mysql> flush privileges;
2. Install hive
Unzip to the /optfolder:
$ tar -zxvf apache-hive-2.3.4-bin.tar.gz -C /opt
$ cd /opt/apache-hive-2.3.4-bin/conf/
modify hive-env.sh to
configure hadoop_home and other environment variables, there are at the end of the file
[fay@master conf]$ cp hive-env.sh.template hive-env.sh
[fay@master conf]$ vi hive-env.sh
...
# HADOOP_HOME=${bin}/../../hadoop
export HADOOP_HOME=/opt/hadoop-2.8.5
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/apache-hive-2.3.4-bin/conf
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/apache-hive-2.3.4-bin/lib
export JAVA_HOME=/opt/jdk1.8.0_102
Modify hive-site.xml
[fay@master conf]$ cp hive-default.xml.template hive-site.xml
[fay@master conf]$ vi hive-site.xml
# 文件太长了，就copy几个重要需要改的地方，找到这些配置项改
<property>
<name>hive.exec.local.scratchdir</name>
<value>/home/fay/hive/tmp/fay</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/home/fay/hive/tmp/${hive.session.id}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>你的密码</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://master:3306/hive_metedata?createDatabaseIfNotExist=true</value>
<description>
JDBC connect string for a JDBC metastore.
To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
Note : If javax.jdo.option.ConnectionDriverName is com.mysql.jdbc.Driver, put the mysql-connector jar package into the ${hive_home}/lib folder, so the mysql-connector jar package may have Optional
Then all of the configuration file ${system:java.io.tmpdir}is changed to /home/fay/hive/tmp(if not the file is created), and gives read and write access to this folder will ${system:user.name}changefay
Then add environment variables:
[fay@master lib]$ vi ~/.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HIVE_HOME=/opt/apache-hive-2.3.4-bin
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin
source ~/.bashrc
Create hdfs folder and grant permissions
$ hadoop fs -mkdir -p /user/hive/
$ hadoop fs -mkdir -p /user/hive/warehouse
$ hadoop fs -chmod 777 /user/hive/
$ hadoop fs -chmod 777 /user/hive/warehouse
Initialize hive, and then start hive:
[fay@master conf]$ schematool -dbType mysql -initSchema
[fay@master conf]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.4-bin/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
Then test the function of hive, which is similar to sql. I don't need to introduce more here. So far, the whole environment is set up. If you encounter a problem halfway, there are still many solutions on the Internet. Later, if I press this one time, I will adjust the environment again if there is a problem. The environment is more familiar, I have also built the environment nearly 10 times, I used to use the same version, this time I plan to use the latest version, there are still many pitfalls in the middle.