After HBASE is started, the reason for the rapid shutdown of the Hmaster process is simple analysis and simple solutions

In the process of building a big data cluster, HBASE is an important tool for using noSQL (non-relational database), but Hmaster crashes after using start-hbase.sh to start. Today is Saturday, I will come to find out.

problem recurring

ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted

Then Hmaster does not have JPS in the process, but when start-hbase.sh is used, it will prompt that there is this process, and it needs to be killed first.

According to the information collected on the Internet, the general reasons are as follows:

1. The core-site.xml and hdfs-site.xml files in hdfs are not copied to hbase /conf folder Insufficient allocation
3. Due to the data generated during hdfs startup and hbase being out of sync

We see that the first and third reasons are related to hdfs, so I first build a pseudo-distribution (using the file system that comes with linux as storage, and the zookeeper component that comes with it for management).

The construction steps are as follows:

#!/bin/bash
INSTALL_PATH=/opt/hbase
INSTALL_TAR=$(find ./ -name hbase*)
tar -zxvf ${INSTALL_TAR}
mv ./hbase* ./hbase
#手动cp hbase-env.sh和hbase-site.xml覆盖conf文件夹下同名
#删除tar和配置
echo "##HBASE_HOME @ `date` @ by YuanYihan" >>/etc/profile
echo "export HBASE_HOME=/opt/hbase">>/etc/profile
echo "export PATH=\$PATH:\$HBASE_HOME/bin">>/etc/profile
source /etc/profile
#0。ip网络hosts，jdk
#1.上传tar安装包并解压；
#2、hbase-env.sh中配置JAVA_HOME
#3、copy 配置
echo "hbase:::@ by YuanYihan#done!"

In two of the configuration files, the places I modified are:

########hbase-env.sh#######第29行左右添加source语句（懒得写javahome了）####
# The java implementation to use.  Java 1.6 required.
source /etc/profile
#export JAVA_HOME=/usr/java/jdk1.6.0/

########hbase-env.sh#######第117行左右定义了文件夹（tmp文件我认为不靠谱）####
# The directory where pid files are stored. /tmp by default.
export HBASE_PID_DIR=$HBASE_HOME/pids

########hbase-env.sh#######第124行左右定义ZK的使用，当然默认就是true####
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=true

<!-- hbase-site.xml文件中定义了几个目录文件 -->
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
  <name>hbase.rootdir</name>
  <value>file:///var/yyh/hbase/hbase_local/rootdir</value>
 </property>
 <property>
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/var/yyh/hbase/hbase_local/zookeeper</value>
 </property>
</configuration>

Local test conclusion

The file system built in this way relies on the file system of linux itself, and does not rely on the HDFS file system. As a result, the test is still hmaster flashback, so it can be judged: at least the flashback of my computer has nothing to do with the HDFS cluster.

Then in-depth test results are as follows:

1) I checked the problem this morning and found that the active NN and Hmaster are on the same computer, so the Hmaster is easy to hang.

2) Forcibly kill the first NN (kill zkfc), and then there is no problem after the first hmaster (will not be hanged).

Based on the above two characteristics, I think that the impact of insufficient memory resources is relatively large.

proposal of a solution

Considering factors such as memory, the solution is to improve the structure of the cluster so that it becomes the following structural relationship to reduce the device load of the first virtual machine:

new cluster structure

virtual machine	hdfs and ZK roles	Yarn and HIVE characters	Hbase role	external port
yyh1		HIVEC	HM, HR, HZK (included)	60010
yyh2	NN, DN, JN, ZK, ZKFC	NM,HIVEC		50070，8020
yyh3	DN, JN, ZK	NM,RM，HIVES		8088，10000，9083
yyh4	DN, JN, ZK	NM,MYSQL,HIVEC，RM		3306，8088

After the cluster is built, use the following steps to start the cluster:

Startup sequence for new clusters

The new cluster startup sequence is as follows: Step
1: [Auto script: firewall is closed, ntp is timed] [Mysql in yyh4] --- This part does not require operation
2: zookeeper [yyh2,3,4] zkServer.sh start --- status --stop
3: Hapdoop: [yyh2] start-all.sh { start-dfs.sh start-yarn.sh}----stop-all.sh
4: RM: [yyh3, yyh4] yarn-daemon. sh start resourcemanager---
5: HIVES: [If yyh1-3 is to be used, please use yyh3] hive --service metastore other hive or hiveservice2 and the rest BEELINE come over
6: HBASE: [yyh1] start-hbase.sh then hbase shell

New cluster test results

HDFS：

YARN:

HIVE：

HBASE: