foreword
This article belongs to the column "Summary of Big Data Abnormal Problems" . This column is original by the author. Please indicate the source for the citation. Please point out the deficiencies and mistakes in the comment area, thank you!
Problem 1: A large number of empty Regions appear in the business table
application background
The Rowkey design of the business flow table is strictly increasing according to the lexicographical order, and the bucket mechanism is used to write data from multiple processes to one or more exclusive buckets.
cause problems
As the data is continuously added to the HBase table, the region of the table is continuously split. After the split, the previous region will no longer have data written to it, and the data will continue to be written into the newly generated region.
After a period of time, when all the data in a Region expires and the RegionServer triggers a major compaction, the data in the Region will be physically deleted, and the Region will become a useless empty Region.
When there are too many empty Regions in the cluster, it will bring trouble to the cluster management, and the meta table will continue to grow, increasing the pressure on the HMaster node.
solution
HBase supports online merging of Regions. We only need to routinely check the status of the Regions in the table. If there are multiple consecutive empty Regions, we can merge these empty Regions into one Region.
Question 2: The RPC connection of the business HBase client is abnormally closed
application background
The business system data is synchronized from MySQL to HBase, and the frequency of synchronous storage is directly related to the frequency of use by users of the business system.
cause problems
We have observed from the business HBase client logs that the RPC connection is always closed abnormally during certain periods of time.
solution
Adjust the HBase client parameters, adjust the RPC timeout from 60s to 180s, and at the same time, the client is forced to execute a flush commit every minute or when 1000 records are full, and synchronize HBase from the DB once to ensure that the RPC long connection will not be automatically closed.
Question 3: The I/O utilization rate of a single RegionServer is always 100%, and the HBase cluster request volume is 0
application background
In order to increase the data storage rate, the business HBase client uses the putlist interface to reduce the number of RPC connections between the client and the server.
cause problems
When a single RegionServer has a disk abnormality and the IO usage of a certain disk is always 100%, the entire cluster becomes unavailable, and the number of cluster requests is directly 0.
solution
Increase the monitoring of abnormal disk IO for cluster machines, and if the disk usage rate of a partition is too high, the machine will be automatically removed from the cluster.