HBase performance optimization

1. Modify the Linux configuration
The maximum number of open files in the Linux system is generally 1024 by default. If you do not modify the concurrency, an error of "Too Many Open Files" will appear, making the entire HBase inoperable. You can use ulimit -n command to modify, or modify the parameters of /etc/security/limits.conf and /proc/sys/fs/file-max, how to modify can go to Google keyword "linux limits.conf"

2. Modify JVM configuration
Modify the configuration parameters in the hbase-env.sh file—
HBASE_HEAPSIZE 4000 #The size of the JVM heap used by
HBase—HBASE_OPTS "‐server -XX:+UseConcMarkSweepGC"JVM #GC option
—HBASE_MANAGES_ZK false #Whether to use Zookeeper for distributed management

3, - Modify HBase configuration -
zookeeper.session.timeout
- Default value: 3 minutes (180000ms)
Description: The connection timeout time between RegionServer and Zookeeper. When the timeout expires, the ReigonServer will be removed from the RS cluster list by Zookeeper. After the HMaster receives the removal notification, it will rebalance the regions responsible for this server and let other surviving RegionServers take over.
Tuning:
This timeout determines Whether the RegionServer can failover in time. Set to 1 minute or less to reduce the extended failover time due to wait timeouts.
However, it should be noted that for some online applications, the RegionServer itself has a very short time from downtime to recovery (flashes such as network failures, crash, etc., operation and maintenance can quickly intervene). Because when the ReigonServer is officially removed from the RS cluster, the HMaster starts to balance (let other RSs recover according to the WAL log recorded by the faulty machine). When the faulty RS is restored by manual intervention, this balancing action is meaningless, but will make the load uneven and bring more burdens to the RS. Especially those with fixed allocation of regions.

4. Modify the HBase configuration: hbase-site.xml -
hbase.regionserver.handler.count
- default value: 10
Description: The number of IO threads for request processing by RegionServer.
Tuning:
The tuning of this parameter is closely related to memory.
Fewer IO threads are suitable for processing Big PUT scenarios with high memory consumption for a single request (large-capacity single PUT or scan with a large cache set, both belong to Big PUT) or scenarios where the memory of ReigonServer is relatively tight.
More IO threads are suitable for scenarios with low memory consumption per request and very high TPS requirements. When setting this value, monitor the memory as the main reference.
It should be noted here that if the server has a small number of regions and a large number of requests fall on one region, the read-write lock caused by the flush triggered by the fast filling of the memstore will affect the global TPS, not the higher the number of IO threads, the better.
During stress testing, enable Enabling RPC-level logging to monitor the memory consumption and GC status of each request at the same time, and finally adjust the number of IO threads reasonably through the results of multiple stress testing.

5. Modify HBase configuration
—
—hbase.hregion.max.filesize
—
Default value: 256M
Description: The maximum storage space of a single Reigon on the current ReigonServer. When a single Region exceeds this value, the Region will be automatically split into smaller regions.
Tuning:
Small regions are friendly to split and compaction, because split regions or storefiles in compact small regions are fast and have low memory usage. The disadvantage is that split and compaction will be frequent.
In particular, the continuous split and compaction of a large number of small regions will cause large fluctuations in the cluster response time. Too many regions will not only bring trouble to management, but also cause some HBase bugs.
Generally, anything below 512 is considered a small region.
Large regions are not suitable for frequent split and compaction, because doing a compaction and split will cause a long pause, which has a great impact on the read and write performance of the application. In addition, a large region means a larger storefile, which is also a challenge to memory when compacting.
Of course, large regions also have their place. If in your application scenario, the traffic volume at a certain point in time is low, then doing compaction and split at this time can not only complete the split and compaction smoothly, but also ensure most of the time.
Since split and compaction affect performance so much, there are There is no way to remove it?
Compaction is unavoidable, but split can be adjusted from automatic to manual.
As long as the value of this parameter is increased to a value that is difficult to achieve, such as 100G, automatic split can be disabled indirectly (RegionServer will not split regions that do not reach 100G).
Then cooperate with the tool RegionSplitter to manually split when split is required.
Manual split is much more flexible and stable than automatic split. On the contrary, the management cost does not increase much, and it is recommended to use online real-time systems. Smooth read and write performance.
In terms of memory, a small region is more flexible in setting the size of the memstore, while a large region cannot be too large or too small. Too large a region will cause the app's IO wait to increase during flushing, and if it is too small, it will affect the read performance due to too many store files.

6. - Modify HBase configuration -
hbase.regionserver.global.memstore.upperLimit/lowerLimit
-
default value: 0.4/0.35
upperlimit description: hbase.hregion.memstore.flush.size This parameter is used when the size of all memstores in a single Region When the sum exceeds the specified value, flush all memstores in the region. The flush of the RegionServer is processed asynchronously by adding the request to a queue, simulating the production and consumption mode. Then there is a problem here. When the queue is too late to consume and a large number of backlog requests are generated, it may cause a sharp increase in memory, and the worst case is to trigger OOM.
The function of this parameter is to prevent excessive memory usage. When the total memory occupied by memstores of all regions in ReigonServer reaches 40% of the heap, HBase will force block all updates and flush these regions to release the memory occupied by all memstores.
lowerLimit description: Same as upperLimit, but lowerLimit does not flush all memstores when the memory occupied by memstores in all regions reaches 35% of the Heap. It will find a region with the largest memory usage in the memstore and do individual flushes. At this time, write updates will still be blocked. lowerLimit is a remedial measure before all regions are forced to flush and cause performance degradation. In the log, the performance is "** Flush thread woke up with memory above low water."
Tuning: This is a Heap memory protection parameter, and the default value is suitable for most scenarios.
Parameter adjustment will affect reading and writing. If the writing pressure is large and often exceeds this threshold, reduce the read cache hfile.block.cache.size to increase the threshold, or when there is a large heap margin, do not modify the read cache size.
If it does not exceed this threshold under high pressure, it is recommended that you lower this threshold appropriately and then do the pressure test to ensure that the number of triggers is not too many, and then when there is more Heap margin, increase hfile.block .cache.size improves read performance.
There is also a possibility that hbase.hregion.memstore.flush.size remains unchanged, but RS maintains too many regions. It is necessary to know that the number of regions directly affects the size of the occupied memory.

7. - Modify HBase configuration -
hfile.block.cache.size 
-
Default value: 0.2
Description: The percentage of the Heap's size occupied by the read cache of storefile, 0.2 means 20%. This value directly affects the performance of data read.
Tuning: Of course, the bigger the better, if the writing is much less than the reading, it is no problem to open it to 0.4-0.5. If the reading and writing are more balanced, it is about 0.3. If you write more than you read, let it be the default. When setting this value, you should also refer to "hbase.regionserver.global.memstore.upperLimit", which is the maximum percentage of memstore in the heap. One of the two parameters affects reading and the other affects writing. If the two values ​​add up to more than 80-90%, there is a risk of OOM, so set it carefully.
- The memory of Regionserver on HBase is divided into two parts, one part is used as Memstore, which is mainly used for writing; the other part is used as BlockCache, which is mainly used for reading.
- The write request will be written to the Memstore first, and the Regionserver will provide a Memstore for each region. When the Memstore is full of 64MB, it will start flushing to the disk. When the total size of the Memstore exceeds the limit (heapsize * hbase.regionserver.global.memstore.upperLimit * 0.9), the flush process will be started forcibly, starting from the largest Memstore and flushing until it falls below the limit.
- The read request goes to the Memstore to check the data. If it cannot be found, it goes to the BlockCache. If it cannot be found, it goes to the disk to read, and the read result is put into the BlockCache. Since BlockCache adopts the LRU strategy, after the BlockCache reaches the upper limit (heapsize * hfile.block.cache.size * 0.85), the elimination mechanism will be activated to eliminate the oldest batch of data.
—There is a BlockCache and N Memstores on a Regionserver, and the sum of their sizes cannot be greater than or equal to heapsize * 0.8, otherwise HBase cannot be started. The default BlockCache is 0.2 and Memstore is 0.4. For systems that focus on read response time, BlockCache can be set larger, such as BlockCache=0.4, Memstore=0.39, to increase the cache hit rate.

8. - Modify HBase configuration -
hbase.hstore.blockingStoreFiles  
-
default value: 7
Description: When flushing, when there are more than 7 storefiles in the Store (Coulmn Family) in a region, block all write requests for compaction, to reduce the number of storefiles.
Tuning: Block write requests will seriously affect the response time of the current regionServer, but too many storefiles will also affect read performance. From a practical point of view, in order to obtain a smoother response time, the value can be set to infinite. If you can tolerate large peaks and valleys in the response time, you can adjust it by default or according to your own scene.

9. - Modify HBase configuration -
hbase.hregion.memstore.block.multiplier 
-
default value: 2
Description: When the memory size of the memstore in a region exceeds twice the size of hbase.hregion.memstore.flush.size, block the All requests of the region are flushed and the memory is released.
Although we set the total memory size of memstores occupied by the region, such as 64M, imagine that at the last 63.9M, I put a 200M data, and the size of the memstore will instantly skyrocket to exceed the expected hbase.hregion Several times of .memstore.flush.size. The function of this parameter is to block all requests when the size of the memstore increases to more than 2 times hbase.hregion.memstore.flush.size, preventing further expansion of the risk.
Tuning: The default value of this parameter is quite reliable. If you estimate that your normal application scenarios (excluding exceptions) will not have burst writes or a controllable amount of writes, then keep the default values. If under normal circumstances, your write request volume will often increase to several times normal, then you should increase this multiple and adjust other parameter values, such as hfile.block.cache.size and hbase.regionserver.global.memstore .upperLimit/lowerLimit to reserve more memory to prevent HBase server OOM.

10. - Modify HBase configuration
- -
hbase.hregion.memstore.mslab.enabled 
-
Default value: true
Description: Reduce Full GC caused by memory fragmentation and improve overall performance.
Tuning: Arena Allocation is a GC optimization technology that can effectively reduce Full GC caused by memory fragmentation, thereby improving the overall performance of the system. This article introduces the principle of Arena Allocation and its application in Hbase-MSLAB.
Enable MSLAB:
--hbase.hregion.memstore.mslab.enabled=true // Enable MSALB
—hbase.hregion.memstore.mslab.chunksize=2m // chunk size, the larger the memory, the better the continuity, but the average memory utilization will be reduced
—hbase.hregion.memstore.mslab.max.allocation=256K // pass The object allocated by MSLAB cannot exceed 256K, otherwise it will be allocated directly on the Heap, 256K is big enough