HBase integrated deployment and other related components (Hive Phoenix) of

A, HBase deploy

1.1 , Zookeeper normal deployment

First, to ensure the normal deployment Zookeeper cluster, and start it:

/opt/module/zookeeper-3.4.5/bin/zkServer.sh start

1.2 , Hadoop normal deployment

Normal deployment of Hadoop cluster and start:

/opt/module/hadoop-2.8.4/sbin/start-dfs.sh

/opt/module/hadoop-2.8.4/sbin/start-yarn.sh

1.3 , HBase decompression

HBase to extract the specified directory:

tar -zxf /opt/software/hbase-1.3.1-bin.tar.gz -C /opt/module/

1.4 , HBase profile

HBase need to modify the corresponding configuration file.

hbase-env.sh modify the content:

export JAVA_HOME=/opt/module/jdk1.8.0_121

export HBASE_MANAGES_ZK=false

Tip: If you are using JDK8 above, comment out the lines 45-47 hbase-env.sh, otherwise a warning will be reported

hbase-site.xml content:

<name>hbase.rootdir</name>

<value>hdfs://master:9000/hbase</value>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.master.port</name>

</property>

<name>hbase.zookeeper.quorum</name>

<value>master:2181,slave1:2181,slave2:2181</value>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/opt/module/zookeeper-3.4.10/zkData</value>

</property>

<name>hbase.master.maxclockskew</name>

</property>

regionservers：

master

slave1

slave2

For 1.5 , HBase need to rely Jar package (extra, no configuration)

Since HBase need to rely on Hadoop, so replace jar package under the HBase lib directory to resolve compatibility issues:

1) delete the original jar:

rm -rf /opt/module/hbase-1.3.1/lib/hadoop-*

rm -rf /opt/module/hbase-1.3.1/lib/zookeeper-3.4.10.jar

2) Copy the new jar, jar involved are:

hadoop-annotations-2.8.4.jar

hadoop-auth-2.8.4.jar

hadoop-client-2.8.4.jar

hadoop-common-2.8.4.jar

hadoop-hdfs-2.8.4.jar

hadoop-mapreduce-client-app-2.8.4.jar

hadoop-mapreduce-client-common-2.8.4.jar

hadoop-mapreduce-client-core-2.8.4.jar

hadoop-mapreduce-client-hs-2.8.4.jar

hadoop-mapreduce-client-hs-plugins-2.8.4.jar

hadoop-mapreduce-client-jobclient-2.8.4.jar

hadoop-mapreduce-client-jobclient-2.8.4-tests.jar

hadoop-mapreduce-client-shuffle-2.8.4.jar

hadoop-yarn-api-2.8.4.jar

hadoop-yarn-applications-distributedshell-2.8.4.jar

hadoop-yarn-applications-unmanaged-am-launcher-2.8.4.jar

hadoop-yarn-client-2.8.4.jar

hadoop-yarn-common-2.8.4.jar

hadoop-yarn-server-applicationhistoryservice-2.8.4.jar

hadoop-yarn-server-common-2.8.4.jar

hadoop-yarn-server-nodemanager-2.8.4.jar

hadoop-yarn-server-resourcemanager-2.8.4.jar

hadoop-yarn-server-tests-2.8.4.jar

hadoop-yarn-server-web-proxy-2.8.4.jar

zookeeper-3.4.10.jar

Scream: These corresponding version of the package should be replaced jar hadoop version you are currently using, specific conditions.

Find jar package, for example:

find /opt/module/hadoop-2.8.4/ -name hadoop-annotations*

Then find the jar package copied to the HBase lib directory.

1.6 , HBase flexible connection Hadoop configuration (additional, no configuration)

ln -s /opt/module/hadoop-2.8.4/etc/hadoop/core-site.xml /opt/module/hbase-1.3.1/conf/core-site.xml

ln -s /opt/module/hadoop-2.8.4/etc/hadoop/hdfs-site.xml /opt/module/hbase-1.3.1/conf/hdfs-site.xml

1.7, configure the environment variables

vi / etc / profile

export HBASE_HOME=/opt/module/hbase-1.3.1

export PATH=$HBASE_HOME/bin:$PATH

source /etc/profile

1.8, HBase remote scp to other cluster

scp -r /opt/module/hbase-1.3.1/ slave1:/opt/module/

scp -r /opt/module/hbase-1.3.1/slave2:/opt/module/

1.9, start HBase services

Start mode 1 :

bin/hbase-daemon.sh start master

bin/hbase-daemon.sh start regionserver

Screaming Tip: If the time between the cluster nodes are not synchronized, it will lead to regionserver not start throwing ClockOutOfSyncException exception.

Start mode 2 :

bin/start-hbase.sh

Corresponding stop service:

bin/stop-hbase.sh

2.0, page view Hbse

After a successful start, you can : access HBase management page "host port" way, such as:

http://master:16010

Two, HBase and Hive integrated use

Preparing the Environment

Because we follow may also affect HBase Hive while operating, it is required to hold the operation of HBase Hive Jar, then the next copy Hive Jar package depends (or in the form of soft-wired). Remember also the zookeeper was admitted to the jar package lib directory of the hive.

# Environment Variables / etc / profile

$ export HBASE_HOME=/opt/module/hbase-1.3.1

$ export HIVE_HOME=/opt/module/apache-hive-1.2.2-bin

#Shell execution

$ ln -s $HBASE_HOME/lib/hbase-common-1.3.1.jar $HIVE_HOME/lib/hbase-common-1.3.1.jar

$ ln -s $HBASE_HOME/lib/hbase-server-1.3.1.jar $HIVE_HOME/lib/hbase-server-1.3.1.jar

$ ln -s $HBASE_HOME/lib/hbase-client-1.3.1.jar $HIVE_HOME/lib/hbase-client-1.3.1.jar

$ ln -s $HBASE_HOME/lib/hbase-protocol-1.3.1.jar $HIVE_HOME/lib/hbase-protocol-1.3.1.jar

$ ln -s $HBASE_HOME/lib/hbase-it-1.3.1.jar $HIVE_HOME/lib/hbase-it-1.3.1.jar

$ ln -s $HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar $HIVE_HOME/lib/htrace-core-3.1.0-incubating.jar

$ ln -s $HBASE_HOME/lib/hbase-hadoop2-compat-1.3.1.jar $HIVE_HOME/lib/hbase-hadoop2-compat-1.3.1.jar

$ ln -s $HBASE_HOME/lib/hbase-hadoop-compat-1.3.1.jar $HIVE_HOME/lib/hbase-hadoop-compat-1.3.1.jar

While the hive-site.xml modification zookeeper properties, as follows:

<name>hive.zookeeper.quorum</name>

<value>bigdata11,bigdata12,bigdata13</value>

<description>The list of ZooKeeper servers to talk to. This is only needed for read/write locks.</description>

</property>

<name>hive.zookeeper.client.port</name>

<description>The port of ZooKeeper servers to talk to. This is only needed for read/write locks.</description>

</property>

Note : need to replace the hive under the lib directory hive-hbase-handler-1.2.1.jar

Three, HBase and phoenix integrated use

1.phoenix installation package unzip replacement directory

tar -zxvf apache-phoenix-4.14.1-HBase-1.2-bin.tar.gz -C /opt/module

mv apache-phoenix-4.14.1-HBase-1.2-bin phoenix-4.14.1

Environment variables vi / etc / profile

# In the last two lines add the following phoenix configuration

export PHOENIX_HOME=/opt/module/phoenix-4.14.1

export PATH=$PATH:$PHOENIX_HOME/bin

# The environment variable configuration to take effect

source /etc/profile

The master node phoenix packet transmitted from the node

$ scp -r phoenix-4.14.1 root@slave1:/opt/module

$ scp -r phoenix-4.14.1 root@slave2:/opt/module

Copy hbase-site.xml (Note) have three

cp hbase-site.xml /opt/module/phoenix-4.14.1/bin/

The following two jar package, /opt/module/phoenix-4.14.1 the directory, copy to hbase lib directory, directory /opt/module/hbase-1.3.1/lib/

(Note) every three

phoenix-4.14.1-HBase-1.3-server.jar

phoenix-core-4.14.1-HBase-1.3.jar

2, start Phoenix

After the restart under hbase configured .

sqlline.py master:2181