hbase problem summary

1 java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: offset (0) + length (8) exceed the capacity of the array: 4

It appears when doing a simple incr operation. The reason is that the int length vlen=4 was put in the previous put, and the increase operation is not applicable. It can only be changed to long type vlen=8

2 When writing data to column org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: 10.xx.xx.37:60020, or org.apache.hadoop.hbase .NotServingRegionException: Region is not online: For these two errors, Regions in Transition appears in the master-status for more than ten minutes, and it has been in the PENDING_OPEN state, causing the request to block. At present, the 10.xx.xx.37 machine is offline, and it runs stably overnight, and there is no blockage caused by split. Suspect it is a machine problem. Hmaster's log shows that the region server keeps open and close without any split or flush.

The full name of RIT is region in transcation. Each time the hbase master performs an open or a close operation on the region, a record will be inserted into the RIT of the master, because the operation of the master on the region must be kept atomic, and the open and close of the region are performed by the Hmaster It is completed with the assistance of the region server. Therefore, in order to meet the coordination, rollback, and consistency of these operations. Hmaster adopts the RIT mechanism and combines the state of Node in Zookeeper to ensure the security and consistency of operations.

OFFLINE, // region is in an offline state
PENDING_OPEN, // sent rpc to server to open but has not begun
OPENING, // server has begun to open but not yet done
OPEN, // server opened region and updated meta
PENDING_CLOSE, // sent rpc to server to close but has not begun
CLOSING, // server has begun to close but not yet done
CLOSED, // server closed region and updated meta
SPLITTING, // server started split of a region
SPLIT // server completed split of a region

It is further found that the problem of load balance is that the region server is repeatedly opened and closed. Refer to abloz.com/hbase/book.html#regions.arch.assignment to restart the region server and it is normal

The region not on line appeared again in the code running later, which was thrown by NotServingRegionException because "Thrown by a region server if it is sent a request for a region it is not serving."

Why keep requesting an offline region? And this kind of error is concentrated in 3 regions out of 150. Tracking the server log, the region will be closed by CloseRegionHandler, and it will be reopened after about 20 minutes. After closing, the region requested by the client is still the closed region?

3 The setting switch does not take effect if it is not written to hbase

The code was first launched, and a switch was added. If there is a problem with hbase, the switch will be turned off. But there is a problem and it is found that the program is stuck. At present, it is believed that the reason is the continuously increasing retry mechanism, 60 seconds timeout, 10 retry times in 1-32 seconds, in case of a problem, the switch is useless.

Need to configure rpc timeout parameter and retry time to solve it

4 flush , split , compact lead to stop-the-world

A long flush split operation caused the hbase server to fail to respond to requests. You need to adjust the size of the region and test the number of flushes obtained

5 hbase parameter settings

hbase.regionserver.handler.count

Considering the io capability of the sas disk, set it to 50

hbase.hregion.memstore.block.multiplier

When the size of the memstore is a multiplier multiple of hbase.hregion.memstore.flush.size, block reading and writing for flushing, the default is 2

6 region server crush

The reason for the Regionserver crash is that the connection between the Regionserver and the zookeeper is timed out because the GC time is too long.

The timeout inside Zookeeper is as follows:

minSessionTimeout in milliseconds, the default is 2 times tickTime.

maxSessionTimeout in milliseconds, the default is 20 times tickTime.

(tickTime is also a configuration item. It is the minimum time unit for the internal control time logic of the Server)

If the sessionTimeout sent by the client exceeds the range of min-max, the server will automatically intercept it as min or max, and then create a new Session object for the client.

The default tickTime is 2s, that is, the maximum timeout of the client is 40s. Even if the zookeeper.session.timeout of the regionserver is set to 60s, it is useless.

change:

Modify the tickTime of the zookeeper cluster to 9s, the maximum timeout is 180s, and modify the zookeeper.session.timeout to 120s, so as to avoid the timeout caused by the GC.
Add the parameter hbase.regionserver.restart.on.zk.expire to true. The function of changing the parameter is to restart the regionserver after the timeout between the regionserver and the zookeeper, instead of turning off the regionserver.

7 Code problems lead to deadlocks

A query in the slow query log of the master reached 2 hours, which eventually caused the server to respond slowly and could not cope with large writes. The reason for the investigation is that the getColumns operation takes out hundreds of thousands of data without paging; the change program has about 500 pages, and there is no problem at present.

8 operation too slow

2012-07-26 05:30:39,141 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":69315,"ts":9223372036854775807,"client":"10.75.0.109:34780","starttimems":1343251769825,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"trackurl_status_list","families":{"sl":[{"timestamp":1343251769825,"qualifier":"zzzn1VlyG","vlen":0}]},"row":""}
It took 69315s to delete a row of data

And the magic is that row is "", row cannot be set to null, but empty strings can be added. did a round of testing
It takes 700ms to delete a non-existent column with an empty row-key

It takes 5ms to delete an existing column with an empty row-key

Deleting any column with a non-empty row-key takes 3ms

I don't know if it's a bug, and I don't know how to pass an empty row-key in. The current strategy is to avoid operations on empty row-keys on the code side.

9 responseTooSlow

2012-07-31 17:52:06,619 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":1156438,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3dbb29e5), rpc version=1, client version=29, methodsFingerPrint=-1508511443","client":"10.75.0.109:35245","starttimems":1343727170177,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}

引用hbase说明：The output is tagged with operation e.g. (operationTooSlow) if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged (responseTooSlow) and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself.

The current practice is to cancel the batch delete operation on multiple columns of a key to avoid blocking, and no new problems have been found.

10 output error

2012-07-31 17:52:06,812 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@61574be4, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"c":["ALL"]},"maxVersions":1,"row":"zOuu6TK"}), rpc version=1, client version=29, methodsFingerPrint=-1508511443 from 10.75.0.151:52745: output error

11 rollbackMemstore problem

Logs like this appear more frequently:
2012-08-07 10:21:49,887 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: rollbackMemstore rolled back 0 keyvalues from start:0 to end:0

方法解释为：Remove all the keys listed in the map from the memstore. This method is called when a Put has updated memstore but subequently fails to update the wal. This method is then invoked to rollback the memstore.

It's very strange that the start and end index are both 0

Loop in method: for (int i = start; i < end; i++) {

So empty data, empty rollback. need further investigation

12 A new region server is online, resulting in region not on line

Requesting a region to the wrong region server

13 Requesting a region that does not exist, and re-establishing the tablepool does not work

Requested timestamp 1342510667

The latest region rowkey related timestamp 1344558957

Finally found that the property that maintains the region location table is in the HConnectionManager

get Get, delete Delete, incr Increment are handled in the ServerCallable class withRetries

Scenario 1 If there is an error (SocketTimeoutException ConnectException RetriesExhaustedExcetion), then clear the regionServer location

Scenario 2 If numRetries is set to 1, the loop will only be executed once, connect(tries!=0) is connect(false), that is, reload=false, location update will not be performed, and it will be re-acquired when numRetries>1

Get Gets List, put Put or Puts List, and delete Deletes List will call the processBatch of HConnectionManager to process. When there is a problem with the results of batch get, put, and delete operations, refresh the regionServer location

Set numRetries to > 1 times, here is 3 times, to solve the problem

14 zookeeper.RecoverableZooKeeper(195): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

This happened when I was doing a test on a single machine. Whether I started hbase from ide or bin, I could connect normally from the shell, but I couldn't connect from the test program. The zookeeper port is 2181, and the client port should have nothing to do with zookeeper.

Finally, change the configuration from port 21818 to port 2181 and it works fine. This change should be done in a stand-alone environment.

<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>Property from ZooKeeper's config zoo.cfg.
    The port at which the clients will connect.
    </description>
  </property>

hbase-site.xml configuration

hbase.tmp.dir

The tmp directory of the local file system is generally set to the local mode, but it is better to set it, because many files will be set to the following by default.

Online configuration

< property >

< name >hbase.tmp.dir</ name >

< value >/mnt/dfs/11/hbase/hbase-tmp</ value >

</ property >

Defaults:

${java.io.tmpdir}/hbase-${user.name}

Write to the system's /tmp directory

hbase.rootdir

All RegionServers in the HBase cluster share the directory, which is used to persist HBase data. Generally, the file directory of hdfs is set, such as hdfs://namenode. example.org:9000/hbase

Online configuration

< property >

< name >hbase.rootdir</ name >

< value >hdfs://mycluster/hbase</ value >

</ property >

Defaults:

${hbase.tmp.dir}/hbase

hbase.cluster.distributed

Cluster mode, distributed or stand-alone mode, if set to false, the HBase process and the Zookeeper process are in the same JVM process.
Online configuration is true
Default: false

hbase.zookeeper.quorum

The URL configuration of the zookeeper cluster, multiple hosts are separated by commas (,)

Online configuration

< property >

< name >hbase.zookeeper.quorum</ name > < value >inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org</ value >

</ property >

Default: localhost

hbase.zookeeper.property.dataDir

Configuration in ZooKeeper's zoo.conf. where snapshots are stored
Online configuration: /home/hadoop/zookeeperData
Default: ${hbase.tmp.dir}/zookeeper

zookeeper.session.timeout

Client and zk connection timeout
Online configuration: 1200000 (20min)
Default: 180000 (3min)

hbase.zookeeper.property.tickTime

The time interval between the client and zk sending heartbeats
Online configuration: 6000 (6s)
Default: 6000

hbase.security.authentication

HBase cluster security authentication mechanism, the current version only supports kerberos security authentication.
Online configuration: kerberos
Default: empty

hbase.security.authorization

Whether HBase has enabled the security authorization mechanism
Online configuration: true
Default: false

hbase.regionserver.kerberos.principal

Principal name of regionserver's kerberos authentication (consists of three parts: service or user name, instance name, and domain name)
Online configuration: hbase/[email protected]
Default: none

hbase.regionserver.keytab.file

regionserver keytab file path
Online configuration: /home/hadoop/etc/conf/hbase.keytab
Default: none

hbase.master.kerberos.principal

The principal name of the master's kerberos authentication (consists of three parts: service or user name, instance name, and domain name)
Online configuration: hbase/[email protected]
Default: none

hbase.master.keytab.file

master keytab file path
Online configuration: /home/hadoop/etc/conf/hbase.keytab
Default: none

hbase.regionserver.handler.count

The number of threads that the regionserver processes IO requests
Online configuration: 50
Default configuration: 10

hbase.regionserver.global.memstore.upperLimit

The RegionServer process block performs flush triggering conditions: the sum of the memstores of all regions on the node reaches upperLimit*heapsize
Online configuration: 0.45
Default configuration: 0.4

hbase.regionserver.global.memstore.lowerLimit

A condition for the RegionServer process to trigger flush: the sum of the memstores of all regions on the node reaches lowerLimit*heapsize
Online configuration: 0.4
Default configuration: 0.35

hbase.client.write.buffer

The client writes the buffer. When autoFlush is set to false, the client will only flush when the buffer is full.
Online configuration: 8388608 (8M)
Default configuration: 2097152 (2M)

hbase.hregion.max.filesize

The region size of a single ColumnFamily, if it exceeds the set value according to the ConstantSizeRegionSplitPolicy policy, it will be automatically split
Online configuration: 107374182400 (100G)
Default configuration: 21474836480 (20G)

hbase.hregion.memstore.block.multiplier

If the multiple of the size of the memstore reaches this value, all write requests will be blocked for self-protection.
Online configuration: 8 (the memory is large enough and can be adjusted appropriately, in this case, the client needs to make adjustments)
Default configuration: 2

hbase.hregion.memstore.flush.size

The size of memstore, when this value is reached, it will be flushed to the external storage device
Online configuration: 104857600 (100M)
Default: 134217728 (128M)

hbase.hregion.memstore.mslab.enabled

Whether to enable the mslab solution to reduce Full GC caused by memory fragmentation and improve overall performance
Online configuration: true
Default configuration: true

hbase.regionserver.maxlogs

The number of hlogs of regionserver
Online configuration: 128
Default configuration: 32

hbase.regionserver.hlog.blocksize

The upper limit of the hlog size, when the value is reached, block and roll off
Online configuration: 536870912 (512M)
Default configuration: block size of hdfs configuration

hbase.hstore.compaction.min

The minimum number of storefiles entering the minor compact queue
Online configuration: 10
Default configuration: 3

hbase.hstore.compaction.max

The maximum number of files in a single minor compact
Online configuration: 30
Default configuration: 10

hbase.hstore.blockingStoreFiles

When the number of storefiles in a region reaches this value, block write and wait for compact
Online configuration: 100 (the production environment can be set very large)
Default configuration: 7

hbase.hstore.blockingWaitTime

block waiting time
Online configuration: 90000 (90s)
Default configuration: 90000 (90s)

hbase.hregion.majorcompaction

The cycle that triggers major compact
Online configuration: 0 (turn off major compact)
Default configuration: 86400000 (1d)

hbase.regionserver.thread.compaction.large

The number of threads in the large compact thread pool
Online configuration: 5
Default configuration: 1

hbase.regionserver.thread.compaction.small

The number of threads in the small compact thread pool
Online configuration: 5
Default configuration: 1

hbase.regionserver.thread.compaction.throttle

The critical point at which compact (major and minor) requests enter the large and small compact thread pools
Online configuration: 10737418240 (10G)
Default configuration: 2 * this.minFilesToCompact * this.region.memstoreFlushSize

hbase.hstore.compaction.max.size

The maximum size of the storefile file in the minor compact queue
Online configuration: 21474836480 (20G)
Default configuration: Long.MAX_VALUE

hbase.rpc.timeout

RPC request timeout time
Online configuration: 300000 (5min)
Default configuration: 60000 (10s)

hbase.regionserver.region.split.policy

split operation default strategy
Online configuration: org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy (adopt the old strategy, control split by yourself)
Default configuration: org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy (under the premise that the region does not reach maxFileSize, if the fileSize reaches regionCount * regionCount * flushSize, the split operation is performed)

hbase.regionserver.regionSplitLimit

Maximum number of regions on a single RegionServer
Online configuration: 150
Default configuration: 2147483647

hbase-env.sh configuration

Specify the system operating environment

export JAVA_HOME= /usr/lib/jvm/java-6-sun/ #JDK HOME

export HBASE_HOME= /home/hadoop/cdh4/hbase-0 .94.2-cdh4.2.1 # HBase 安装目录

export HBASE_LOG_DIR= /mnt/dfs/11/hbase/hbase-logs #日志输出路径

JVM parameter tuning

export HBASE_OPTS="-verbose:gc -XX:+PrintGCDetails -Xloggc:${HBASE_LOG_DIR} /hbase-gc .log -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime \

-server -Xmx20480m -Xms20480m -Xmn10240m -Xss256k  -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -XX:MaxTenuringThreshold=15 \

-XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection \

-XX:+CMSClassUnloadingEnabled  -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSMaxAbortablePrecleanTime=5000     \

"

.Each Region has a Memstore, the default size of Memstore is 128MB, which can be changed by hbase.hregion.memstore.flush.size;
Regions will gradually increase with the split operation. In order to control the sum of Memstores and cause OOM errors, in the old version of hbase, it is controlled by hbase.regionserver.global.memstore.upperLimit and hbase.regionserver.global.memstore.lowerLimit, in the new version Use hbase.regionserver.global.memstore.size and hbase.regionserver.global.memstore.lowerLimit control;
When HEAP_SIZE=4G in Hbase-env.sh, the old version Hbase.regionserver.global.memstore.upperLimit (default HEAP_SIZE*0.4)=1.6G, hbase.regionserver.global.memstore.lowerLimit (default HEAP_SIZE*0.35)=1.4G , the new version hbase.regionserver.global.memstore.size(default HEAP_SIZE*0.4)=1.6G and Hbase.regionserver.global.memstore.lowerLimit(hbase.regionserver.global.memstore.size*HEAP_SIZE*0.95)=1.52G;
When the sum of Memstores reaches the first critical value, the largest one among all memstores will be selected for flushing, and writing will not be blocked at this time;
The sum of the memstores reaches the second critical value, which will block all reads and writes, and flush all the current memstores.
Each Region has a BlockCache, and the sum of BlockCache is set to HEAP_SIZE multiplied by 0.4 by default, which is set by hfile.block.cache.size by default;
All read requests are searched in BlockCache first. Some values in the basic Memstore are also found in BlockCache. If they are not found, they are searched in Hfile.
By default, hbase stipulates that the sum of the maximum sum of Memstore (hbase.regionserver.global.memstore.size defaults to 0.4) and the maximum sum of BlockCache (hfile.block.cache.size defaults to 0.4) cannot be greater than 0.8, because HEAP_SIZE of 0.2 needs to be reserved For other operations, please refer to the hbase source code Org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil.java file for details.