Daily operation and maintenance of hbase

From: http://my.oschina.net/beiyou/blog/76456

 

First, the basic command:

    建表:create 'testtable','coulmn1','coulmn2'

     You can also add coulmn attributes when creating a table, such as: create 'testtable',{NAME => 'coulmn1', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'LZO ', TTL => '30000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'coulmn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => ' 30', COMPRESSION => 'LZO', TTL => '30000', IN_MEMORY => 'true'} (the attributes are versions: set the number of historical versions, TTL: expiration time, COMPRESSION: compression method, when configuring lzo condition)

    Drop table: drop 'testtable' (disable the table before dropping the table, command disable 'testtable')

    Enable and disable tables: enable 'testtable' and disable 'testtable'

    Other basic commands: describe 'testtable' (view table structure), alert to modify table structure, list to list all tables.

Second, the order of daily maintenance

    1, major_compact 'testtable', usually the production environment will close the automatic major_compact (hbase.hregion.majorcompaction in the configuration file is set to 0), select a time window with few users at night to manually major_compact, if the hbase update is not too frequent, you can do it in a week Do major_compact once for all tables. You can watch the number of all storefiles after doing a major_compact. If the number of storefiles increases to nearly double the storefile after major_compact, you can do a major_compact for all tables. It takes a long time and the operation should be done as much as possible. Avoid peak periods.

    2. Flush 'testtable' to refresh all memstores to hdfs. Usually, if it is found that the memory usage of the regionserver is too large, causing many thread blocks in the regionserver of the machine, you can perform a flush operation, which will cause a sharp increase in the number of hbase storefiles. This operation should be avoided as much as possible. In another case, when hbase is migrating, if you choose to copy files, you can stop writing first, then flush all tables and copy files.

    3, balance_switch true or balance_switch flase, configure whether the master balances the number of regions of each regionserver. When we need to maintain or restart a regionserver, the balancer will be closed, which will make the region unevenly distributed on the regionserver. At this time, manual Turn on balance.

Third, restart a regionserver

    bin/graceful_stop.sh --restart --reload --debug nodename

    This operation is to smoothly restart the regionserver process, which will not affect the service. It will first migrate all regions on the regionserver that need to be restarted to other servers, then restart, and finally migrate the previous region back, but we modify In a configuration, you can restart each machine in this way. This command will shut down the balancer, so finally we need to execute balance_switch true in the hbase shell. For hbase regionserver restart, do not kill the process directly, which will cause the zookeeper. If session.timeout is interrupted for a long time, do not restart the regionserver through bin/hbase-daemon.sh stop. If you are not lucky, if the -ROOT- or .META. table is above, all requests will fail.

Fourth, shut down a regionserver offline

    bin/graceful_stop.sh --stop  nodename

    As above, the system will migrate all regions before shutting down, then stop the process, and finally we need to manually balance_switch true to enable the master's region balance.

Fifth, check whether the region is normal and repair it

    bin/hbase hbck (check)

    bin/hbase hbck -fix (fix)

    It will return whether all regions are mounted normally. If there is no normal mounting, you can use the next command to fix it. If it still cannot be repaired, you need to see why the log failed and handle it manually.

Six, hbase migration

    1, copytable method

    bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=zookeeper1,zookeeper2,zookeeper3:/hbase 'testtable'

    目前0.92之前的版本的不支持多版本的复制,0.94已经支持多个版本的复制。当然这个操作需要添加hbase目录里的conf/mapred-site.xml,可以复制hadoop的过来。

    2,Export/Import

    bin/hbase org.apache.hadoop.hbase.mapreduce.Export testtable /user/testtable [versions] [starttime] [stoptime]

    bin/hbase org.apache.hadoop.hbase.mapreduce.Import testtable  /user/testtable

    跨版本的迁移,我觉得是一个不错的选择,而且copytable不支持多版本,而export支持多版本,比copytable更实用一些。

    3,直接拷贝hdfs对应的文件

    首先拷贝hdfs文件,如bin/hadoop distcp hdfs://srcnamenode:9000/hbase/testtable/ hdfs://distnamenode:9000/hbase/testtable/

    然后在目的hbase上执行bin/hbase org.jruby.Main bin/add_table.rb /hbase/testtable

    生成meta信息后,重启hbase

    这个操作是简单的方式,操作之前可以关闭hbase的写入,执行flush所有表(上面有介绍),再distcp拷贝,如果hadoop版本不一致,可以用hftp接口的方式,我推荐使用这种方式,成本低。

    先总结这么多,有空我再详细写一下region坏时怎么修复。

Guess you like

Origin http://10.200.1.11:23101/article/api/json?id=326922120&siteId=291194637