HBase的分布式环境搭建

其他 2019-03-09 15:41:09 阅读次数: 0

最近工作涉及到Hbase的使用，搭建环境是第一步，折腾了2天，终于搞定了，特意记录下来。

1. 主机

搭建分布式环境,准备3台主机

主机名 IP
node1 1.1.12.23
node2 1.1.12.45
node3 1.1.12.48

2. 软件准备

  • jdk-8u161-linux-x64.tar.gz
  • hadoop-2.7.1.tar.gz
  • zookeeper-3.3.6.tar.gz
  • hbase-1.2.6-bin.tar.gz

3. 安装JDK

将jdk解压到目录/usr/java/并设置java环境变量

root@node1:/usr/java# tar -zxvf jdk-8u161-linux-x64.tar.gz
root@node1:/usr/java# ls
jdk1.8.0_161
root@node1:/usr/java# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_161/
export JRE_HOME=JAVA_HOME/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
root@node1:/usr/java# source /etc/profile
root@node1:/usr/java# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

4. 安装Hadoop

Hbase在分布式环境下依赖于HDFS,需要先安装Hadoop。

1. 创建用户和组（每台主机）

添加hadoop用户和组，主要原因是为了控制安全权限。

root@node1:~#  groupadd hadoop
root@node1:~#  useradd -m -g hadoop hadoop
root@node1:~#  passwd hadoop
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
root@node1:~# su - hadoop
hadoop@node1:~$

2. 准备目录并设置权限(每台主机)

hadoop@node1:~$ mkdir app   #hadoop,hbase,zookeeper的安装目录
hadoop@node1:~$ mkdir data   #hadoop,hbase,zookeeper的数据目录
hadoop@node1:~$ ls -l
total 20
drwxr-xr-x 5 hadoop hadoop 4096 4月  11 16:19 app
drwxrwxr-x 5 hadoop hadoop 4096 4月  11 16:09 data
-rw-r--r-- 1 hadoop hadoop 8980 4月  20  2016 examples.desktop

若app与data目录的权限不是hadoop,则应设置权限:

root@node1:~# chown -R hadoop:hadoop /home/hadoop/app/   #data目录类似

3. 配置hosts(每台主机)

root@node1:~# vim /etc/hosts
127.0.0.1       localhost
1.1.12.23       node1
1.1.12.45       node2
1.1.12.48       node3

4. SSH 免密登录 (每台主机)

hadoop@node1:~$ ssh-keygen -t rsa

一路回车,然后在当前用户目录下的.ssh目录中会成成公钥文件(id_rsa.pub)和私钥文件(id_rsa).
分发公钥

扫描二维码关注公众号，回复： 5474747 查看本文章 
hadoop@node1:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@node1

5.  服务器功能规划

  node1 node2 node3
NameNode yes    
DataNode yes yes yes
ResourceManager   yes  
NodeManager yes yes yes
Zookeeper yes yes yes

4.1 在node1上安装hadoop

1. 解压安装包

root@node1:/home/hadoop/app# tar -zxvf hadoop-2.7.1.tar.gz
root@node1:/home/hadoop/app# ls
hadoop-2.7.1 
root@node1:/home/hadoop/app# chown -R hadoop:hadoop hadoop-2.7.1/
root@node1:/home/hadoop/app# ls -l
total 12
drwxr-xr-x 10 hadoop hadoop 4096 4月  11 14:11 hadoop-2.7.1

2. 配置Hadoop的java环境:hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_161

3. 配置core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>   #NameNode地址
        <value>hdfs://node1:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>  #临时目录地址
        <value>/home/hadoop/data/tmp</value>
    </property>
</configuration>

4. 配置hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.data.dir</name>   #NameNode的数据存储位置
        <value>file:///home/hadoop/data/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>   #DataNode的数据储存位置
        <value>file:///home/hadoop/data/dfs/data</value>
    </property>
</configuration>

5. 配置slaves(即DataNode)

node1
node2
node3

6. 配置yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.hostname</name>  #resourcemanager地址
        <value>node2</value>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>  #是否启用日志聚集功能
        <value>true</value>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-seconds</name> #聚集日志在hdfs中最多保存多长时间
        <value>106800</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name> #nodemanager分配的内存。不应小于1024Mb
        <value>1024</value>
    </property>
</configuration>

以上并没有配置mapreduce,若有需要可参考:点击打开链接或:点击打开链接

将其他两台主机也按照上述步骤进行配置,或直接远程复制。

4.2 格式化与启动集群

1. 在NameNode上格式化:

hadoop@node1:~/app/hadoop-2.7.1$ bin/hdfs namenode -format

2. 启动集群

hadoop@node1:~/app/hadoop-2.7.1$ ./sbin/start-dfs.sh 
Starting namenodes on [node1]
node1: starting namenode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-namenode-node1.out
node1: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node1.out
node3: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node3.out
node2: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-secondarynamenode-node1.out
hadoop@node1:~/app/hadoop-2.7.1$ jps
36579 NameNode
36742 DataNode
37065 Jps
36939 SecondaryNameNode
hadoop@node1:~/app/hadoop-2.7.1$ ./sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-node1.out
node1: starting nodemanager, logging to /home/hadoop/app/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-node1.out
node2: starting nodemanager, logging to /home/hadoop/app/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-node2.out
node3: starting nodemanager, logging to /home/hadoop/app/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-node3.out
hadoop@node1:~/app/hadoop-2.7.1$ jps
36579 NameNode
36742 DataNode
37399 Jps
36939 SecondaryNameNode
37261 NodeManager

切换到resourcemanager(即node2):

hadoop@node2:~/app/hadoop-2.7.1$ ./sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-node2.out
hadoop@node2:~/app/hadoop-2.7.1$ jps
23944 ResourceManager
23771 NodeManager
23595 DataNode
24172 Jps

5. 安装Zookeeper

分别在3台机器上执行以下步骤。

1. 解压安装包

hadoop@node1:~/app$  tar -zxvf zookeeper-3.3.6.tar.gz
hadoop@node1:~/app$ ls
hadoop-2.7.1  zookeeper-3.3.6

2.配置zoo.cfg

hadoop@node1:~/app/zookeeper-3.3.6/conf$  cp zoo_sample.cfg zoo.cfg
hadoop@node1:~/app/zookeeper-3.3.6/conf$  vim zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/hadoop/data/zdata
# the port at which the clients will connect
clientPort=2181
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

3. 创建myid文件

hadoop@node1:~/data$ touch zdata/myid
hadoop@node1:~/data$ echo 1 >> zdata/myid
hadoop@node1:~/data$ cat zdata/myid 
1

其他两台机器的myid分别设置为2、3

4. 启动zookeeper

hadoop@node1:~/app/zookeeper-3.3.6$ bin/zkServer.sh start
JMX enabled by default
Using config: /home/hadoop/app/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
hadoop@node1:~/app/zookeeper-3.3.6$ jps
38181 QuorumPeerMain
38204 Jps

 

6. 安装HBase

1. 服务器规划

  node1 node2 node3
Master yes    
RegionServer   yes yes

2. 解压安装包

hadoop@node1:~/app$ tar -zxvf hbase-1.2.6-bin.tar.gz
hadoop@node1:~/app$ ls
hadoop-2.7.1  hbase-1.2.6  zookeeper-3.3.6

3. 配置java环境

hadoop@node1:~/app/hbase-1.2.6$ vim conf/hbase-env.sh
# The java implementation to use.  Java 1.7+ required.
 export JAVA_HOME=/usr/java/jdk1.8.0_161/

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=true

4. 配置hbase-site.xml

<configuration>
    <property>
        <name>hbase.rootdir</name>   #根目录,应来源于hdfs的namenode地址
        <value>hdfs://node1:9000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>  #采用分布式环境
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name> #zookeeper的数据存放路径,应与zookeeper配置一致
        <value>/home/hadoop/data/zdata</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>    #zookeeper集群的地址
        <value>node1,node2,node3</value>
    </property>
</configuration>

5. 配置regionservers

node2
node3

在其他主机上执行上述步骤。

6. 启动HBase

（1）启动hdfs（namenode）

hadoop@node1:~/app/hbase-1.2.6$ start-dfs.sh 
Starting namenodes on [node1]
node1: starting namenode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-namenode-node1.out
node1: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node1.out
node2: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node2.out
node3: starting datanode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-datanode-node3.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.7.1/logs/hadoop-hadoop-secondarynamenode-node1.out
hadoop@node1:~/app/hbase-1.2.6$ jps
38913 SecondaryNameNode
38551 NameNode
38716 DataNode
39037 Jps

（2）启动hbase（master）

hadoop@node1:~/app/hbase-1.2.6$ ./bin/start-hbase.sh 
starting master, logging to /home/hadoop/app/hbase-1.2.6/bin/../logs/hbase-hadoop-master-node1.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
node1: starting regionserver, logging to /home/hadoop/app/hbase-1.2.6/bin/../logs/hbase-hadoop-regionserver-node1.out
node2: starting regionserver, logging to /home/hadoop/app/hbase-1.2.6/bin/../logs/hbase-hadoop-regionserver-node2.out
node3: starting regionserver, logging to /home/hadoop/app/hbase-1.2.6/bin/../logs/hbase-hadoop-regionserver-node3.out
node1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
node1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
node2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
node2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
node3: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
node3: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
hadoop@node1:~/app/hbase-1.2.6$ jps
38913 SecondaryNameNode
39346 HRegionServer
39189 HMaster
38181 QuorumPeerMain
38551 NameNode
38716 DataNode
39663 Jps

（4）web页面

HBase Web UI: http://node1:16010

HDFS Web UI：http://node1:50070

7. 客户端连接

1. 客户端hosts配置

1.1.12.23       node1
1.1.12.45       node2
1.1.12.48       node3

2. 将hbase-site.xml放到工程资源目录下src/main/resources

<configuration>
       <property>
       	<name>hbase.zookeeper.quorum</name>
       	<value>node1,node2,node3</value>
       </property>
</configuration>

3. 示例

public class App 
{
    static Configuration conf;
	
    public static void main( String[] args )
    {
       conf = HBaseConfiguration.create();
       try {
    	   Connection conn = ConnectionFactory.createConnection(conf);
    	   Admin admin = conn.getAdmin();
    	   if (!admin.tableExists(TableName.valueOf("mytable"))) {
    	       HTableDescriptor desc = new HTableDescriptor(TableName.valueOf("mytable"));
		    desc.addFamily(new HColumnDescriptor("c1"));
		    admin.createTable(desc);
			}
    	       Table table = conn.getTable(TableName.valueOf("mytable"));
    	       //插入数据：row1,c1:a,value1
    	       Put put = new Put(Bytes.toBytes("row1"));
    	       put.addColumn(Bytes.toBytes("c1"), Bytes.toBytes("a"), Bytes.toBytes("value1"));
    	       table.put(put);
    	       table.close();
    	       System.out.println("sucessful put data...");
    	       conn.close();
		} catch (Exception e) {
		    e.printStackTrace();
		}
    }
}

猜你喜欢

转载自blog.csdn.net/wubinghai/article/details/79931931
今日推荐
NetBSD 禁止提交由 AI 生成的代码
Apache Doris 2.0.10 版本正式发布！
开源日报 | 大模型开战；大模型独角兽被曝卖身；周鸿祎建议谷歌开源所有产品；最大开源AI社区提供1000万美元共享GPU
开源日报 | Chrome内置Gemini的意义不在于Gemini；中国AI追随之路的五大误区；ECharts创始人“下海”养鱼；谷歌I/O开发者大会什么都有，只是没有惊喜
微软回应中国区AI团队“打包赴美”传闻
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
周排行
static方法和非static方法的区别（java）
如何查找计算机专业paper
java.lang.ClassFormatError: Incompatible magic value 0 in class file com/sitecha
跳跃游戏II
stm32_之【建立工程】
TeaWeb v0.0.9 发布，统计底层优化、主机监控功能改进
事件分发 -----控制字体大小
JavaScript DOM练习（动态表格添加） December 25，2019
JSF Scope & CDI
实现从零搭建一个登录注册页面（附源代码）
每日归档
更多
2024-05-19(0)
2024-05-18(4)
2024-05-17(34)
2024-05-16(6)
2024-05-15(24)
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022