Hadoop2.6+Hbase1.0 cluster construction

HBase installation

http://hbase.apache.org/

There are also three modes of HBase installation: stand-alone mode, pseudo-distributed mode and fully distributed mode. Only the fully distributed mode is introduced here. The premise is that the Hadoop cluster and Zookeeper have been installed and run correctly.

Step 1: Download the installation package, unzip it to a suitable location, and assign permissions to the hadoop user (the account that runs hadoop)

The download here is hbase-0.94.6, the Hadoop cluster uses 1.0.4, extract it to /usr/local and rename it to hbase

sudo cp hbase-0.94.6.tar.gz /usr/local

sudo tar -zxf hbase-0.94.6.tar.gz

sudo mv hbase-0.94.6 hbase

sudo chown -R hadoop:hadoop hbase

Step 2: Configure the relevant files

(1) Configure hbase-env.sh, which is in /usr/local/hbase/conf

Set the following values:

export JAVA_HOME=/usr/local/java/jdk1.6.0_27 #Java installation path

export HBASE_CLASSPATH=/usr/local/hadoop/conf #HBase class path

export HBASE_MANAGES_ZK=true #HBase is responsible for starting and closing Zookeeper

(2) Configure hbase-site.xml, which is located in /usr/local/hbase/conf

<name>hbase.master</name>

<value>master:6000</value>

</property>

<name>hbase.master.maxclockskew</name>

</property>

<name>hbase.rootdir</name>

<value>hdfs://master:9000/hbase</value>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.zookeeper.quorum</name>

<value>master</value>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/home/${user.name}/tmp/zookeeper</value>

</property>

<name>dfs.replication</name>

</property>

Among them, hbase.master is the server and port number that specify the running HMaster;

hbase.master.maxclockskew is used to prevent regionserver startup failure due to time inconsistency between HBase nodes. The default value is 30000;

hbase.rootdir specifies the storage directory of HBase;

hbase.cluster.distributed sets the cluster in distributed mode;

hbase.zookeeper.quorum sets the host name of the Zookeeper node, and its value must be an odd number;

hbase.zookeeper.property.dataDir sets the Zookeeper directory, the default is /tmp, and dfs.replication sets the number of data backups. If the number of cluster nodes is less than 3, it needs to be modified. This test is a node, so it is modified to 1.

(3) Configure regionservers, the file is located in /usr/local/hbase/conf

Set the machine running HBase. The configuration of this file is similar to the slaves in hadoop. One line specifies one machine. This test only uses one machine, and you can set the master.

(4) Set the HBase environment variable, the file is located in /etc/profile

Add at the end of the file:

#hbase Env

export HBASE_HOME=/usr/local/hbase

export PATH=$PATH:$HBASE_HOME/bin

Make it effective: source /etc/profile

Step 3: Run the test

After starting hadoop, enter $HBASE_HOME/bin/start-hbase.sh in the terminal to view the running process:

Close: stop-hbase.sh

Reference 1:

Hbase fully distributed cluster installation configuration (Hbase1.0.0, Hadoop2.6.0)

http://blog.csdn.net/wuwenxiang91322/article/details/44684655

1. Install the software

OS: hundreds6.5

Hadoop：hadoop2.6.0

Hbase:hbase.1.0.0

JDK: jdk1.7.0_51

Cluster machines:

192.168.153.130 （hadoop01 purpose）

192.168.153.131 (hadoop02 datanode)

192.168.153.132 (hadoop03 datanode)

2. Installation steps

The following installation steps assume that the JDK has been installed. If it is not installed, please find the JDK installation tutorial by yourself.

2.1 Install hadoop

(1) Download hadoop version (hadoop2.6.0), download address: http://www.apache.org/dyn/closer.cgi/hadoop/common/ Select the corresponding version to download under this path.

(2) Unzip hadoop-2.6.0.tar.gz

tar zxvf hadoop-2.6.0.tar.gz -C /usr/local/soft/

Unzip hadoop to /usr/local/soft/ directory.

(3) Add hadoop to the environment variable

vim ~/.bashrc Open the .bashrc file and add HADOOP_HOME

exportJAVA_HOME=/usr/local/soft/jdk1.7.0_79

export PATH=$HADOOP_HOME/bin:$PATH

(4) Modify the configuration file (under the $HADOOP_HOME/etc/hadoop directory)

Placement core-site.xml

<name>fs.defaultFS</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/usr/local/soft/hadoop-2.6.0/data/tmp</value>

</property>

</configuration>

Configure hdfs-site.xml:

<name>dfs.namenode.name.dir</name>

<value>/usr/local/soft/hadoop-2.6.0/data/namenode</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/usr/local/soft/hadoop-2.6.0/data/datanode</value>

</property>

<name>dfs.replication</name>

</property>

</configuration>

Configure the mapred-site.xml file

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop01:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop01:19888</value>

</property>

<name>mapreduce.cluster.temp.dir</name>

<value>/usr/local/soft/hadoop-2.6.0/data/mr_temp</value>

</property>

</configuration>

Configure yarn-site.xml file

<name>yarn.resourcemanager.hostname.rm1</name>

<value>hadoop01</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<value>/usr/local/soft/hadoop-2.6.0/data/yarn_log</value>

</property>

</configuration>

Modify the slaves file and add the datanode node hostname to the slaves file

hadoop01

hadoop02

hadoop03

(5) Format the cluster and execute the following command on the hadoop01 node:

hadoop purpose -format

(6) Start the cluster and execute the following command on the hadoop01 node:

start-all.sh

(7) Check whether hadoop is successfully installed through the browser:

Enter: http://hadoop01:50070

Enter: http://hadoop01:8088/

2.2 Install Hbase

(1) Download the hbase version

Download address: http://www.apache.org/dyn/closer.cgi/hbase/, select the corresponding version to download under this path, this installation download hadoop1.0.0 version

(2) Unzip hbase-1.0.0-bin.tar.gz

tar zxvf hbase-1.0.0-bin.tar.gz –C /usr/local/soft/

(3) Add hbase to the environment variable

export HBASE_HOME=:/usr/local/soft/hbase-1.0.0

export PATH=$HBASE_HOME/bin:$PATH

(4) Modify the configuration file

Modify hbase-env.sh

exportJAVA_HOME=/home/hadoop001/thirdparty/jdk1.7.0_51

Modify hbase-site.xml

<name>hbase.rootdir</name>

<value>hdfs://hadoop01:8020/hbase</value>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.zookeeper.property.quorum</name>

<value>hadoop01,hadoop02,hadoop03</value>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/usr/local/soft/hadoop/zookeeper</value>

</property>

<name>hbase.master.maxclockskew</name>

</property>

</configuration>

Remark:

In the above configuration file, the first attribute specifies the storage directory of the local hbase;

The second attribute specifies the operating mode of hbase, and true represents the fully distributed mode; the third and fourth attributes are about the configuration of the Zookeeper cluster. My zookeeper is installed on hadoop01, hadoop02 and hadoop03.

Modify regionservers and add the following to the regionservers file:

hadoop02

hadoop03

replicate to the other two nodes

scp -r hadoop01 \hadoop02:/usr/local/soft

scp -r hadoop01 \hadoop03:/usr/local/soft

(5) Start hbase

Make sure hdfs has been started when starting hbase. Execute on the master node:

start-hbase.sh $HBASE_HOME/bin/start-hbase.sh

After the startup is successful, the following processes will be added to the cluster:

NameNode node:

Pictures omitted...

Datanode node:

Pictures omitted...

View via browser:

Enter: http://hadoop01:16010

Pictures omitted...

2.3 Hbase shell test

(1) Execute the $HBASE_HOME/bin/hbase shell command:

(2) Create the testtable table

create 'testtable:colfaml'

(3) put data

The hbase shell is implemented based on Ruby, so you can mix the hbase shell with Ruby code during use. For this example, refer to the introduction to the shell on page p-66 of the HBase Authoritative Guide.

for i in 'a'..'z' do for j in 'a'..'z' do \

put 'testtable' , "row-#{i}#{j}", "colfaml:#{j}" , "#{j}" end end (4) View inserted data

scan 'testtable'

Screenshots of some of the results:

Pictures omitted...

3. Common parameters (to be continued)

4. Remarks

At present, all the configurations installed are the simplest configuration, and parameter optimization is not considered. The purpose of this tutorial is to let Hbase run first, and continue to do the optimization later.

bug:

1. No HRegionServer, HMaster

2. The hbase service of centos-zookeeper-01 cannot be started

Hadoop2.6+Hbase1.0 cluster construction

Guess you like