CentOS7上Apache impala2.7集成apache kudu1.3

其他 2018-11-14 10:30:54 阅读次数: 0

转自：https://blog.csdn.net/jssg_tzw/article/details/73549284

Apache impala集成apache kudu

1.环境准备

1.1.服务器配置

ID IP address Hostname Notes
1 192.168.56.181 apollo.dt.com Hadoop Master
2. Spark Master
3.Kudu Master 
4. Impala Master 5. JDK 
6. Scala
2 192.168.56.182 artemis.dt.com Hadoop Slave
2. Spark Slave
3.Kudu tserver 
4. Impalad 5. JDK 
6. Scala
3 192.168.56.183 uranus.dt.com Hadoop Slave
2. Spark Slave
3.Kudu tserver 
4. Impalad 5. JDK 
6. Scala

1.2.环境配置

1.3. impala 安装包下载

2.下载安装依赖包

[root@apollo impala2.7]# yum -y install cyrus-sasl-plain lsb ntp

3.安装impala

3.1.安装 bigtop-utils (主从机都要安装)

[root@apollo impala2.7]# rpm -ivh bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm
warning: bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:bigtop-utils-0.7.0+cdh5.9.0+0-1.c################################# [100%]

3.2.安装 impala-kudu-2.7.0 (主从机都要安装)

# 要加上选项--nodeps否则安装会因为缺少依赖包无法通过
[root@apollo impala2.7]# rpm -ivh impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm --nodeps
warning: impala-kudu-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-2.7.0+cdh5.9.0+0-1.cd################################# [100%]

3.3.安装 impala-kudu-catalog (主机安装)

[root@apollo impala2.7]# rpm -ivh impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-catalog-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-catalog-2.7.0+cdh5.9.################################# [100%]

3.4.安装 impala-kudu-state (主机安装)

[root@apollo impala2.7]# rpm -ivh impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-state-store-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-state-store-2.7.0+cdh################################# [100%]

3.5.主机安装 impala-kudu-server (主从机都要安装)

[root@apollo impala2.7]# rpm -ivh impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-server-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-server-2.7.0+cdh5.9.0################################# [100%]

3.6.安装impala-kudu-shell (主从机都要安装)

[root@apollo impala2.7]# rpm -ivh impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-shell-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-shell-2.7.0+cdh5.9.0+################################# [100%]

3.7.安装 impala-kudu-udf-devel(主从机都要安装)

[root@apollo impala2.7]# rpm -ivh impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm
warning: impala-kudu-udf-devel-2.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.11.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-kudu-udf-devel-2.7.0+cdh5.################################# [100%]

4.配置impala (主从机都要配置)

4.1. 配置 vim /etc/default/bigtop-utils里的JDK

# export JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_77

4.2.配置vim /etc/default/impala (主从机都要配置)

IMPALA_CATALOG_SERVICE_HOST=apollo.dt.com
IMPALA_STATE_STORE_HOST=apollo.dt.com

5.配置三台机器时间同步（主从机都要配置)

#主机：apollo.dt.com
[root@apollo kudu]# systemctl restart ntpd
#从机：artemis.dt.com
[root@artemis kudu]# systemctl restart ntpd
#从机：uranus.dt.com
[root@uranus kudu]# systemctl restart ntpd

#主机apollo.dt.com设置开机启动
[root@apollo kudu]# systemctl enable ntpd
#从机artemis.dt.com设置开机启动
[root@artemis kudu]# systemctl enable ntpd
#从机uranus.dt.com设置开机启动
[root@uranus kudu]# systemctl enable ntpd

6.把hadoop配置文件core-site.xml和hdfs-site.xml拷到/etc/impala/conf.dist/目录下并作如下改动

6.1 拷贝core-site.xml和hdfs-site.xml (主从机都要配置)

#主机apollo.dt.com
[root@apollo .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[root@apollo .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/
#从机artemis.dt.com
[root@artemis .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[root@artemis .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/

#从机uranus.dt.com
[root@uranus .ssh]# cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf.dist/
[root@uranus .ssh]# cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf.dist/

6.2.具体配置（在hadoop原有内容上增加以下内容）

  • core-site.xml
<property>
        <name>dfs.client.read.shortcircuit</name>
        <value>true</value>
</property>
<property>
        <name>dfs.client.read.shortcircuit.skip.checksum</name>
        <value>false</value>
</property>
<property>
        <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
        <value>true</value>
</property>
  • hdfs-site.xml
<!--impala configuration -->
<property>
        <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
        <value>true</value>
</property>
<property>
        <name>dfs.block.local-path-access.user</name>
        <value>impala</value>
</property>
<property>
        <name>dfs.client.file-block-storage-locations.timeout.millis</name>
        <value>60000</value>
</property>

7. 设置scoket path

在目录/var/run/目录下新建目录hadoop-hdfs (主从机都要配置)。 
注意：该文件夹可能已经存在，应当确认用impala是否有权限进行读写。 
如果已经存在，将用户impala加入该文件所属的组，并修改该文件组的权限即: chown -R 775 hadoop-hdfs/

#主机：apollo.dt.com
[root@apollo .ssh]# cd /var/run/ #进入目录/var/run/
[root@apollo run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[root@apollo run]# chown -R 775 hadoop-hdfs/ #设置权限

#从机：artemis.dt.com
[root@artemis .ssh]# cd /var/run/ #进入目录/var/run/
[root@artemis run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[root@artemis run]# chown -R 775 hadoop-hdfs/ #设置权限

#从机：uranus.dt.com
[root@uranus .ssh]# cd /var/run/ #进入目录/var/run/
[root@uranus run]# mkdir hadoop-hdfs #创建目录hadoop-hdfs
[root@uranus run]# chown -R 775 hadoop-hdfs/ #设置权限

8. 权限配置（所有机器都应当配置）

如果想要 impala 和 YARN 合作，需要把 impala 用户加入 hdfs 组，相关的可以了解的是Llama项目. 
impala 在执行 DROP TABLE 操作时，需要把文件移到到 hdfs 的回收站，所以你需要创建一个 hdfs 的目录 /user/impala，并将其设置为impala 用户可写。同样的，impala 需要读取 hive 数据仓库下的数据，故需要把 impala 用户加入 hive 组。

#主机：apollo.dt.com
[root@apollo hadoop]# usermod -G hdfs,hadoop impala
[root@apollo hadoop]# groups impala
impala : impala hdfs hadoop

#从机：artemis.dt.com
[root@artemis hadoop]# usermod -G hdfs,hadoop impala
[root@artemis hadoop]# groups impala
impala : impala hdfs hadoop

#从机：uranus.dt.com
[root@uranus hadoop]# usermod -G hdfs,hadoop impala
[root@uranus hadoop]# groups impala
impala : impala hdfs hadoop

Extension: 创建impala在hdfs上的目录并设置权限

[hadoop@apollo sbin]$ hadoop fs -mkdir -p /user/impala
[hadoop@apollo sbin]$ hadoop fs -chown impala /user/impala

8.启动impala

#主机apollo.dt.com
[root@apollo impala]# service impala-state-store restart --kudu_master_hosts=apollo.dt.com:7051
[root@apollo impala]# service impala-catalog restart --kudu_master_hosts=apollo.dt.com:7051
[root@apollo impala]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#从机：artemis.dt.com
[root@artemis .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051
#从机：uranus.dt.com
[root@uranus .ssh]# service impala-server restart --kudu_master_hosts=apollo.dt.com:7051

9. 登录impala-shell

[root@apollo impala]# impala-shell
Starting Impala Shell without Kerberos authentication
Connected to apollo.dt.com:21000
Server version: impalad version 2.7.0-cdh5-IMPALA_KUDU-cdh5 RELEASE (build 48f1ad385382cd90dbaed53b174965251d91d088)
***********************************************************************************
Welcome to the Impala shell. Copyright (c) 2015 Cloudera, Inc. All rights reserved.
(Impala Shell v2.7.0-cdh5-IMPALA_KUDU-cdh5 (48f1ad3) built on Thu Aug 18 12:15:44 PDT 2016)

Run the PROFILE command after a query has finished to see a comprehensive summary
of all the performance and diagnostic information that Impala gathered for that
query. Be warned, it can be very long!
***********************************************************************************
[apollo.dt.com:21000] >

10.在impala-shell端创建表

[apollo.dt.com:21000] >  CREATE TABLE dept (
 id BIGINT,
 name STRING
 ) distribute by range(name) split rows(('it'),('op'),('hr'))
 TBLPROPERTIES(
   'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',
   'kudu.table_name' = 'dept',
   'kudu.master_addresses' = 'apollo.dt.com:7051',
   'kudu.key_columns' = 'id,name'
 );

11.在kudu控制台查看impala-shell创建的表

这里写图片描述

猜你喜欢

转载自blog.csdn.net/lukabruce/article/details/82783167
今日推荐
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
周排行
BPM为企业带来的实际利益
好程序员web前端分享css常用属性缩写
Java文件下载（excel）
css样式的动态添加及显示和隐藏等零碎用法
axios全局配置以及拦截器
使用Logstash来实时同步MySQL和log日志数据到ES
C++获取当前时间（年月日、时分秒、毫秒）
Odoo产品分析 (四) -- 工具板块(11) -- 网站即时聊天(1)
Java环境配置正确，但是java、javac、java -version均返回“不是内部或外部命令，也不是可运行的程序或批处理文件”？
01 官网下载各种CentOS教程（超详细版）
每日归档
更多
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022