[Reprint] depth analysis HBase architecture

Depth analysis HBase architecture

https://zhuanlan.zhihu.com/p/30414252

 

 

Original link ( HTTPS: // mapr.com/blog/in-depth- look-HBase-Architecture / )

 

HBase architecture

 

Physically, HBase system has three types of daemons, namely Region server, Master server and zookeeper.

Region server is responsible for actually read or write data. When accessing the data, the client and the HBase Region server communicate directly.

Master server management location Region, DDL (add and delete table structure)

Zookeeper responsible for the maintenance and record the entire HBase cluster state.

 

All HBase data is stored in HDFS. Each Region server put their data stored in HDFS. If a server is both Region server is HDFS's Datanode. So this Region server to which the data will be stored in a copy local HDFS, the accelerated access speed.

 

However, if a new migration to the Region server, this region server and no local copy of the data. HBase run until compaction, will put a copy of migrating to the local Datanode above.

 

Name node location information stored in HDFS This all block files

 

HBase Region server

 

HBase table divided into a plurality of regions according Row Key Region, a Region contains all the data within this region. Server manages the plurality Region Region, is responsible for all read and write operations in this region of Region Server. Region up a Server 1000 can manage the region.

 

 

HBase Master server

 

HBase Maste region is mainly responsible for the distribution and operation of DDL (such as create and delete tables), etc.

HBase Master features:

• Coordination Region server
• In a cluster in the data recovery or dynamically adjust the load assigned to a Region in a Region Server
• Cluster state control, monitoring of all RegionServer
• DDL provides related API, the new (create), delete (delete) and update (update) table structure.

 

 

ZooKeeper: cluster "Property" Administrator

 

Zookeepper is a distributed no central metadata storage service. Zookeeper detect and record HBase cluster server status information. If the zookeeper found the server is down, it will notify Hbase the master node requires at least three in a production environment deployment zookeeper server, to meet the core algorithm zookeeper of Paxos minimum requirements.

 

Annotation: As shown, there zookeeper zookeeper status report embodiment three servers, region server and the master node through the heartbeat

 

ZooKeeper, Master Region server and work together

 

Zookeepr responsible for maintaining the memberlist cluster, which server line, which server downtime by zookeeper detection and management. Region server, standby Master node active connection Zookeeper, maintain a Session connection,

This session requires regularly send heartbeat, to explain their online zookeeper, and no downtime.

 

ZooKeeper有一个Ephemeral Node(临时节点)的概念, session连接在zookeeper中建立一个临时节点(Ephemeral Node), 如果这个session断开, 临时节点被自动删除.

所有Region server都尝试连接Zookeeper, 并在这个session中建立一个临时节点(Ephemeral node). HBase的master节点监控这些临时节点的是否存在, 可以发现新加入region server和判断已经存在的region server宕机.

为了高可用需求, HBase的master也有多个, 这些master节点也同时向Zookeeper注册临时节点(Ephemeral Node). Zookeeper把第一个成功注册的master节点设置成active状态, 而其他master node处于inactive状态.

 

如果zookeeper规定时间内, 没有收到active的master节点的heartbeat, 连接session超时, 对应的临时节也自动删除. 之前处于Inactive的master节点得到通知, 马上变成active状态, 立即提供服务.

同样, 如果zookeeper没有及时收到region server的heartbeat, session过期, 临时节点删除. HBase master得知region server宕机, 启动数据恢复方案.

 

HBase的第一次读写流程

 

HBase的第一次读写流程

 

HBase把各个region的位置信息存储在一个特殊的表中, 这个表叫做Meta table.

Zookeeper里面存储了这个Meta table的位置信息.

 

HBase的访问流程:

1. 客户端访问Zookeep, 得到了具体Meta table的位置

2. 客户端再访问真正的Meta table, 从Meta table里面得到row key所在的region server

3. 访问rowkey所在的region server, 得到需要的真正数据.

 

客户端缓存meta table的位置和row key的位置信息, 这样就不用每次访问都读zookeeper.

如果region server由于宕机等原因迁移到其他服务器. Hbase客户端访问失败, 客户端缓存过期, 再重新访问zookeeper, 得到最新的meta table位置, 更新缓存.

HBase Meta Table

 

Meta table存储所有region的列表

Meta table用类似于Btree的方式存储

Meta table的结构如下:

- Key: region的开始row key, region id

- Values: Region server

 

译注: 在google的bigtable论文中, bigtable采用了多级meta table, Hbase的Meta table只有2级

 

 

Region Server的结构

 

Region Server运行在HDFS的data node上面, 它有下面4个部分组成:

• WAL: 预写日志(Write Ahead Log)是一HDFS上的一个文件, 如果region server崩溃后, 日志文件用来恢复新写入的的, 但是还没有存储在硬盘上的数据.
• BlockCache: 读取缓存, 在内存里缓存频繁读取的数据, 如果BlockCache满了, 会根据LRU算法(Least Recently Used)选出最不活跃的数据, 然后释放掉
• MemStore: 写入缓存, 在数据真正被写入硬盘前, Memstore在内存中缓存新写入的数据. 每个region的每个列簇(column family)都有一个memstore. memstore的数据在写入硬盘前, 会先根据key排序, 然后写入硬盘.
• HFiles: HDFS上的数据文件, 里面存储KeyValue对.

 

HBase的写入流程(1)

 

当hbase客户端发起Put请求, 第一步是将数据写入预写日志(WAL):

• 将修改的操作记录在预写日志(WAL)的末尾
• Write-ahead log (the WAL) is used when the region server crashes, recovery of the data memstore

 

WAL total write to the end of the file, writing is sequential write speed faster

 

 

Hbase writing process (2)

 

Data written to write-ahead log (WAL), and stored after memstore, the user returns to write successfully.

 

 

HBase MemStore

 

MemStore memory according to the order Key, Key-Value for storage, a Memstore cluster corresponds to a column (column family). Also in HFile inside, all of the Key-Value is ordered according to the Key Store.

 

 

HBase Region Flush

 

 

Annotation: Original inside Flush consciousness is, the buffer memory dump of data from the hard disk, which is similar to flush the toilet (Flush the toilet), the data is compared to water, the accumulation of water will look into the sewers want when data is written to the hard disk cache. Flush and very similar to the English as well as un-plug, such as a water bath, as long as the un-plug the bathtub stopper, bath water began to flow into the sewer, but also the analogy cache data is written to the hard disk

 

When Memstore accumulated enough data, Region server data is written Memstore HDFS, stored as a HFile. Each cluster column (column family) for a plurality of HFile, HFile which each data is actually stored.

These HFile are full when Memstore later, Flush HDFS file to the notice HBase cluster limits the column (column family) because the number of clusters for each column (column family) corresponds to a Memstore [... Translation: too more memstore take up too much memory ].

当Memstore的数据Flush到硬盘时, 系统额外保存了最后写入操作的序列号(last written squence number), 所以HBase知道有多少数据已经成功写入硬盘. 每个HFile都记录这个序号, 表明这个HFile记录了多少数据和从哪里继续写入数据.

在region server启动后, 读取所有HFile中最高的序列号, 新的写入序列号从这个最高序列号继续向上累加.

 

 

HBase HFile

 

HFile中存储有序的Key-Value对. 当Memstore满了之后, Memstore中的所有数据写入HDFS中,形成一个新的HFile. 这种大文件写入是顺序写, 因为避免了机械硬盘的磁头移动, 所以写入速度非常快.

 

HBase HFile Structure

 

HFile存储了一个多级索引(multi-layered index), 查询请求不需要遍历整个HFile查询数据, 通过多级索引就可以快速得到数据(工作机制类似于b+tree)

 

• Key-Value按照升序排列
• Key-Value存储在以64KB为单位的Block里
• 每个Block有一个叶索引(leaf-index), 记录Block的位置
• 每个Block的最后一个Key(译注: 最后一个key也是最大的key), 放入中间索引(intermediate index)
• 根索引(root index)指向中间索引

 

尾部指针(trailer pointer)在HFile的最末尾, 它指向元数据块区(meta block), 布隆过滤器区域和时间范围区域. 查询布隆过滤器可以很快得确定row key是否在HFile内, 时间范围区域也可以帮助查询跳过不在时间区域的读请求.

 

译注: 布隆过滤器在搜索和文件存储中有广泛用途, 具体算法请参考https://china.googleblog.com/2007/07/bloom-filter_7469.html

 

 

HFile索引

 

当打开HFile后, 系统自动缓存HFile的索引在Block Cache里, 这样后续查找操作只需要一次硬盘的寻道.

 

 

 

HBase的混合读(Read Merge)

 

我们发现HBase中的一个row里面的数据, 分配在多个地方. 已经持久化存储的Cell在HFile, 最近写入的Cell在Memstore里, 最近读取的Cell在Block cache里. 所以当你读HBase的一行时, 混合了Block cache, memstore和Hfiles的读操作

 

1. 首先, 在Block cache(读cache)里面查找cell, 因为最近的读取操作都会缓存在这里. 如果找到就返回, 没有找到就执行下一步
2. 其次, 在memstore(写cache)里查找cell, memstore里面存储里最近的新写入, 如果找到就返回, 没有找到就执行下一步
3. 最后, 在读写cache中都查找失败的情况下, HBase查询Block cache里面的Hfile索引和布隆过滤器, 查询有可能存在这个cell的HFile, 最后在HFile中找到数据.

 

 

HBase Minor Compaction

 

HBase自动选择较小的HFile, 将它们合并成更大的HFile. 这个过程叫做minor compaction. Minor compaction通过合并小HFile, 减少HFile的数量.

HFile的合并采用归并排序的算法.

译注: 较少的HFile可以提高HBase的读性能

 

 

 

HBase Major Compaction

Major compaction指一个region下的所有HFile做归并排序, 最后形成一个大的HFile. 这可以提高读性能. 但是, major compaction重写所有的Hfile, 占用大量硬盘IO和网络带宽. 这也被称为写放大现象(write amplification)

Major compaction可以被调度成自动运行的模式, 但是由于写放大的问题(write amplification), major compaction通常在一周执行一次或者只在凌晨运行. 此外, major compaction的过程中, 如果发现region server负责的数据不在本地的HDFS datanode上, major compaction除了合并文件外, 还会把其中一份数据转存到本地的data node上.

 

Region = 一组连续key

快速的复习region的概念:

• 一张表垂直分割成一个或多个region, 一个region包括一组连续并且有序的row key, 每一个row key对应一行的数据.
• 每个region最大1GB(默认)
• region由region server管理
• 一个region server可以管理多个region, 最多大约1000个region(这些region可以属于相同的表,也可以属于不同的表)

 

 

Region的拆分

 

最初, 每张表只有一个region, 当一个region变得太大时, 它就分裂成2个子region. 2个子region, 各占原始region的一半数据, 仍然被相同的region server管理. Region server向HBase master节点汇报拆分完成.

如果集群内还有其他region server, master节点倾向于做负载均衡, 所以master节点有可能调度新的region到其他region server, 由其他region管理新的分裂出的region.

 

 

 

负载均衡

 

最初, 一个Region server上的region一分为二, 但是考虑到负载均衡, master node会把新region调度到其他服务器上. 然而, 新region所在的region server在本地data node上没有数据, 所有操作都是操作远程HDFS上面的数据. 直到这个Region server运行了major compaction之后, 才有一份副本落在本地datanode中.

 

译注: HFile和WAL都是存储在HDFS中, 这里说的把副本存储在本地是指: 由于HDFS是一种聪明的FS, 如果他发现要求写入文件的客户端恰好也是HDFS的data node, 那么在分配哪三台服务器存储副本时, 会优先在发请求的客户端存储数据, 这样就可以让Region server管理的数据虽然是3份, 但是其中一份就在本地服务器上, 优化了访问路径.

具体可以参考这篇文章http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html, 里面详述了HDFS如何实现这种本地化的存储. 换句话说, 如果region server没有和HDFS的data node部署在同一台服务器, 就无法实现上面说的本地存储

 

 

HDFS的数据复制(1)

 

所有读写都是操作primary node. HDFS自动复制所有WAL和HFile的数据块到其他节点. HBase依赖HDFS保证数据安全. 当在HDFS里面写入一个文件时, 一份存储在本地节点, 另两份存储到其他节点

 

 

HDFS的数据复制(2)

 

预写日志(WAL) 和 HFile都存在HDFS里面, 可以保证数据的可靠性, 但是HBase memstore里的数据都在内存中, 如果系统崩溃后重启, Hbase如何恢复Memstore里面的数据?

Annotation: memstore seen from the data in FIG memory, there is no multiple copies

 

 

HBase disaster recovery

 

When the region server goes down, the collapse of the region server management region can no longer provide services, HBase to monitor abnormal, start the recovery process, restore the region.

Zookeeper region server found the heartbeat stops, judging region server downtime and notify the master node. Hbase master node assignment that region after the region server downtime, the collapse of the region server management to other region server. HBase from a pre-write file (WAL ) in memstore recovery in the data.

HBase master know the old region which is reassigned to a new region server. Master the crash of the Region server has a write-ahead log (WAL) split into multiple Participation in pre Each region server failure recovery replay of the write log (WAL), rebuild the lost memstore.

 

 

 

Data Recovery

 

Write-ahead log (WAL) records of each operation HBase, each operating on behalf of a Put or Delete Delete operation. All operations in write-ahead log (WAL) arranged in chronological order, the oldest operating header record, the latest operation in the end of the file.

How to restore memstore years, but have not written HFile data? Re-execute the write-ahead log (WAL) can be. Sequentially executed write-ahead log (WAL) in operation from front to back, rebuild memstore data. Finally, Flush memstore data to HFile, complete the recovery.

 

Apache Hbase architecture advantages

Strong consistency model

- When writing to return, to ensure that all reads the same value as read

Automatic extensions

- When data growth is too large, automatically splitting region

- the use of dispersion and backup data HFDS

Built-in auto-reply

- write-ahead log (WAL)

Eco-integrated Hadoop

- running on HBase map reduce

 

Apache HBase problems ...

• Business continuity reliability:
• 重放预写日志慢
• 故障恢复既慢又复杂
• Major compaction容易引起IO风暴(写放大)