what is
every db holds its storage level,either memory or fs.this is similar to,hbase,it has HFile as underlying data structure which will be stored in fs or dfs.
exquisite data structure also must be matched appropriate algoriths.of course ,different requirements will lead to heavily varied db design styles.
like some other index tools,HFile is a indexed storage structure mainly for fast access.the idea of HFle is from TFile ,sure,the former has a bit differences improved, and its internal structures will like below figures:
*note:this file format(not Data block) is for HFile v1,the part File Info is placed after Meta Index for v2.
whye
as described above,hbase uses special file instead of existances in hadoop (eg. SequenceFile) is to speedup the readings by rowkeys,as you can see some index blocks in previous figures:
a.construct a reversed index query,speed up for reading(mainly)
b.supplys a hadoop-independent style to read/write file format without considering dfs's compability
| file | structure | compression | embed in |
| SequenceFile | 3 file types: 1)uncompressed 2)record-comprssed(only values be compressed) 3)block-compressed(both keys and vlaues are compressed,similar to HFile v2) |
yes | hadoop |
| HFile v1 | similar to v2,but some block indexes are different wth that. | yes | hbase |
| HFile v2 | 1)indexed reversed style 2)most data(except trailer) in file are compressed if configure compressor |
yes | hbase new file format for (94+) |
how to
there is a process model in the flush:
1.iterate all keyvaues in snapshot to write to memory buffer 2.generate a new data block if over a 'block-size' which in set when creating table 3.repeat 1 & 2 until no more data from snapshot 4.flush to memory compressed stream 5.flush to stream to outputStream (hfile stream) and clear the tmp buffer to avoid huage memory usage 6.similar to data block flush,flush meta block,data index,meta index,file-info and last trailer
why places extra parts of index and trailer to the last of a hfile?
i think some points are abvious:
a. as the index or trailer have some stats about the block and index,flushing memory buffer to hfile per 'block'(part of hfile) will min the memory overhead by region.
b. this can avoid going far away from the top of hfile offset to locate a special data block offset
others
using the or.apache.hadoop.hbase.io.hfile.HFile tool ,u wil look at the the details of it like below:
hbase hfile -f path-to-file
for a compressed file occupied size '18644309' will results like this:
Stats: Key length: count: 496573 min: 48 max: 53 mean: 49.44025551127427 Val length: count: 496573 min: 1 max: 915 mean: 34.053575204451306 Row size (bytes): count: 28930 min: 814 max: 3190 mean: 1570.45855513308 Row size (columns): count: 28930 min: 12 max: 21 mean: 17.16463878326996 Key of biggest row: 94678d0589778ade561378ac26dfd791Key length count--number of keys(composite keys,ie. rowkey+fml+col+ts+type) val length --same as above row size (bytes) count--number of rows belong to this hfile;and the 'mean' is the average bytes of total rows size row size (columns) --same as above,but for 'mean',it is average qualifies(columns) of total rows so we know,a actual uncompressed row size is:
18644309 / compact-ratio(.42) / 28930 = 1534 ~ 1570(mean)and
key length count = val length count = 496573 ~ (28930 * 17.16 = 496438.8)
and the key length count is just same the entries when flush log shows:
2014-11-04 00:24:08,299 INFO [regionserver60020.cacheFlusher] Store.java:817 Added hdfs://xxx:54310/hbase/gggg/f95f2a0fa04b5c13747bafa22bd610d9/f1/0a5ac38c99da4244ac6925beb51c96b8 , entries=496275, sequenceid=96412594, filesize=18.0m
TODO something to check
-cache data on write:set in blockcache also when writing
-data block encoding:use some encodes like fast diff
-data block /meta block index structure
ref:
hbase-memstore flush -1 overview
hbase guide