Hbase principle the concept of literacy

A, Hbase Profile

1. What is Hbase

Hbase prototype is google's BigTable paper, the paper received the inspired idea, as currently hadoop subprojects to develop maintenance, support for structured data storage.

Hbase is a high reliability (HDFS stored on, there is a copy of the mechanisms), high-performance, column-oriented, non-relational databases (similar Redis), scalable, distributed storage system (because it is stored in the HDFS), using hbase technology to build large-scale structure of the database storage cluster on inexpensive PC server.

Hbase goal is to store and process large data, more specifically, just use common hardware, can handle large data made up of thousands of rows and columns thereof.

Hbase distributed storage framework is built on hdfs, but Hbase achieve random read and write on hdfs changed to address the things that are not supported hdfs

2, Hbase features

A, mass storage

B, column storage

Here columnar storage in fact, that is column family is stored, Hbase can have a lot of columns according to column family column family to store the data below, the column family must be specified when creating the table

Hbase columns and mysql column is not a thing, Hbase column is his data

C, easily extended

Scalability Hbase mainly in two aspects, one is based on the ability to sort the upper extension (RegionServer, equivalent Datanode, processing read and write requests), one based on extended storage (HDFS)

Add RegionServer through horizontal machines, horizontal expansion enhance Hbase upper processing capacity, improve service more Hbase Region ability.

Note: the role RegionServer the management Region) (similar to the mysql table concept), following a read request of the client access, this latter will be described in detail by laterally adding datanode machine, for expansion of the storage layer to enhance Bhbase of literacy storage capacity and enhance the back-end storage

D, sparse

Sparse mainly for flexibility in hbase column, the column family, you can specify any number of columns, column data in the case of empty, will not take up storage space, and here mysql and other databases are not the same, mysql If each field has no value, then this field is null, not empty, and take up storage space

3, Hbase architecture

The following architecture diagram Hbase

And a HMaster Hbase HRegionServer composition, also depends on the availability of HMaster zk, of the Namenode similar hdfs;

HRegionServer hdfs equivalent of datanode, the actual read request processing node;

a、Zookeeper

HBase do Hmaster high availability through zk, RegionServer monitoring, configuration and metadata entry cluster maintenance work, particularly work into the

By zk to ensure that the cluster is running only one master, master if abnormal, will have a new master through a competitive mechanism to provide services

Zk RegionServer monitored by the state, when there is an abnormality RegionServer, notifies the master in the form of callback logout information RegionServer

Zk uniform entry address stored metadata;

b、HMaster

For the allocation Region RegionServer

Maintain load balancing cluster is assigned Region

Maintain metadata cluster information

Found that the failure of the Region, and the failure of Region assign to a normal RegionServer

When RegionServer failure, Hlog hdfs the block and the corresponding coordinate data recovery

C、HRegionServer

HRegionServer direct docking read and write requests of users, is the real work of nodes, his features are summarized below

Management master assigned to Region

Processing read and write requests from a client

Interactive, data storage and is responsible for the underlying hdfs to hdfs

Split Region is responsible for the increases

Responsible for Storefile merger work

D HDFS

Hdfs hbase to provide the ultimate underlying data storage service

And provide metadata table data stored in the underlying distributed service

Multiple copies of data to ensure high reliability and availability

E, Hlog

A HRegionServer only one Hlog, Hlog equivalent edits the file hdfs save Hbase change log, when Hbase to write data, the data is not directly written to disk, he will remain for some time in memory (time i data the threshold may be set amount). However, the data stored in the memory may have a higher probability of data loss, in order to solve this problem, the data is first written in a file called Hlog, Hlog stored on disk, also located on hdfs, then write to memory , so when a system failure or loss of memory, the data can be rebuilt from the log file

F、Region

Corresponds to Region in mysql table, there may be a plurality of HRegionServer Region, have a plurality Region HRegionServer; if the data table is too large, split, according to the average amount of data segmentation, a table of all HBase Region will correspond to one or more, when the content of the table is small, it corresponds to a a table Region, if the table is large, then this will Region segmentation, Region segmentation simultaneously splitting the Region of all Store.

G、Store

Store the equivalent of column families, the popular talk is a family column, in the hbase, want to create a column, you must specify the column family, that is, a column must belong to a column family. A table can have multiple column families, a store corresponds to a column family, hbase official does not recommend more than one column family, a column family can engage hundreds of columns, is sufficient. However, if a HRegion is then sliced, cut breakdown is aromatic, so even if only a HRegion a list, the segmentation will correspond to a plurality of Region Store, strore are assigned to a plurality of other storage nodes HRegionServer

H, MemStore

MemStore is a family of data columns in memory, write data, and will be written to memory, as long as the memory write is successful, the returns.

I、StoreFile

StoreFile, data in memory insecurity, but is limited in size, so it is necessary to in-memory data written to disk to store Hfile format on hdfs. Every memstore brush to form a storefile, so many storefile would, however small, because the memory itself is not large, behind storeFile will merge, but this is only the internal consolidation of a column family StoreFile merger, will not cross the combined column family

J, HFILE

This is the actual physical files stored on the disk of the original data is actually stored files, storefile Hfile is stored in the form of hdfs

Two, Hbase installation

1, you must first install zk

2, and then you want to install hdfs

3, and finally install hbase

4, extract, modify the configuration file

Here focus on that modify the configuration file, the front did not say, because I use ambari tools in actual use to install

First modify hbase-env.sh

Java configuration environment variable

export JAVA_HOME=/usr/lib/jvm/java

Configuring zk, Hbase also strongly dependent on the zookeeper, and whether you want to enable their zookeeper. If was true, if the external zookeeper, and false

export HBASE_MANAGES_ZK=false

Configuring hbase-site.xml

<property>
      <name>hbase.rootdir</name>
      <value>/apps/hbase/data</value>
    </property>

Whether a cluster configuration hbase enabled

    <property>
      <name>hbase.cluster.distributed</name>
      <value>true</value>
    </property>

Set Hbase service port number, not the web port number, web port number is 16010

  <property>
      <name>hbase.master.info.port</name>
      <value>16010</value>
    </property>

    <property>
      <name>hbase.master.port</name>
      <value>16000</value>
    </property>

Zk configured to connect

   <property>
      <name>hbase.zookeeper.quorum</name>
      <value>abdi1,abdi2,abdi3</value>
    </property>

Zk parent directory to store data, mainly in order to distinguish between multiple clusters hbase

    <property>
      <name>zookeeper.znode.parent</name>
      <value>/hbase-unsecure</value>
    </property>

配置regionservers文件

指定RegionServer的节点

由于hbase是强依赖于hdfs的，需要拷贝hdfs的配置文件到hbase的conf目录

我们一般情况会这样操作，创建一个软链接，链接到hdfs的core-site.xml和hdfs-site.xml中，就是让hbase知道我要连接哪个hadoop集群

但是在ambari安装的hbase的配置文件中没有找到相应的配置，但是在hbase启动的时候有加载hdfs的环境变量

启动hbase，可以看到有Hmaster和HRegionServer的java进程

ambari的web页面显示效果如下

注意：Hbase的Master和RegionServer安装是一样的，只是看我们是否要启动master

Hbase的web页面，采用16010端口

三、Hbase的简单shell操作

1、进入hbase shell

[root@abdi2 bin]# /usr/hdp/current/hbase-client/bin/hbase shell

2、查看当前有哪些表：list

hbase(main):003:0> list
TABLE                                                                                                                                                                                                                                                                         
0 row(s)
Took 0.2713 seconds                                                                                                                                                                                                                                                           
=> []
hbase(main):004:0>

3、创建表操作。这里的列族是必须要指定的，就是和mysql的列一样:create "student","info"

hbase(main):004:0> create "student","info"
Created table student
Took 1.3445 seconds                                                                                                                                                                                                                                                           
=> Hbase::Table - student
hbase(main):005:0> 
hbase(main):006:0> list
TABLE                                                                                                                                                                                                                                                                         
student                                                                                                                                                                                                                                                                       
1 row(s)
Took 0.0055 seconds                                                                                                                                                                                                                                                           
=> ["student"]

4、插入数据。Hbase中的数据没有什么类型，比如字符串，hash等，全部是字节:put "student","1001","info:name","laowang"

hbase(main):007:0> put "student","1001","info:name","laowang"
Took 0.1217 seconds                                                                                                                                                                                                                                                           
hbase(main):008:0> put "student","1001","info:age","18"
Took 0.0038 seconds                                                                                                                                                                                                                                                           
hbase(main):009:0> put "student","1001","info:sex","male"
Took 0.0049 seconds                                                                                                                                                                                                                                                           
hbase(main):010:0> put "student","1002","info:name","laoluo"
Took 0.0036 seconds                                                                                                                                                                                                                                                           
hbase(main):011:0> put "student","1002","info:age","20"
Took 0.0035 seconds