CDH 6.2.0安装

其他 2020-08-04 16:26:52 阅读次数: 0

CDH安装文档

  1. 概述
    本文档主要内容为 cdh 生态环境各个应用搭建部署文档，CDH包括zookper、hadoop、saprk、hive、scala。

  2. 软件环境
    集群软件环境确定如下：
    软件名称 版本
    OS Centos 7.3
    spark 2.4.0-cdh6.2.1
    python python 2.7.13
    java 1.8
    hadoop 3.0.0-cdh6.2.1
    mariadb 5.5.64
    hive hive-2.1.1-cdh6.2.1
    scala scala-2.11.12

  3. 集群情况
    测试集群:
    主机名 业务IP地址 角色
    K8s-1 172.16.93.11 HDFSBalancer DFSZKFailoverController HDFSHttpFS NameNode HDFSNFSGateway HIVEGateway HIVEMetastoreServer HIVEServer2 HIVEWebchatServer ActiveMonitor AlertPublish EventServer HostMonitor ServiceMonitor SparkGateway SparkHistoryServer JobHistoryServer Master ResourceManager Zookeeper
    K8s-2 172.16.93.12 HDFSDataNode DFZKFailoverController NameNode HiveGateway SparkGateway NodeManager
    K8s-3 172.16.93.13 DataNode JournalNode HiveGateway SparkGateway NodeManager
    K8s-4 172.16.93.14 DataNode JournalNode HiveGateway SparkGateway NodeManager
    K8s-5 172.16.93.15 DataNode JournalNode HiveGateway SparkGateway NodeManager

JournalNode：hadoop元数据同步存储节点。
Namnode：hadoop 主节点，负责集群数据存储管理。
DataNode：hadoop 数据节点，存储hadoop数据块。
DFSZKFailoverController：依赖zookeeper的HA组件，负责namenode的active/standby
ResourceManager：Yarn 资源调度，管理NodeManager节点
NodeManager：是ResourceManager的代理，监控本身节点资源使用情况，并向RM汇报
Master: spark主节点，spark 资源调度，管理Worker节点。（采用spark by yarn，无 master。）
Worker: spark工作节点，spark 任务执行。（采用spark by yarn，无worker。）
QuorumPeerMain：zookeeper进程，元数据同步节点通信管理。

  1. CDH平台搭建
    4.1配置host（所有机器）
    每个节点编辑/etc/hosts文件，添加如下内容
    172.16.93.11 k8s-1
    172.16.93.12 k8s-2
    172.16.93.13 k8s-3
    172.16.93.14 k8s-4
    172.16.93.15 k8s-5

4.2关闭防火墙（所有机器）

#stop and disable firewalld service
systemctl stop firewalld
systemctl disable firewalld
#disable selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/'  /etc/selinux/config
setenforce 0

4.3配置系统参数（所有机器）

sysctl -w vm.swappiness=10
echo "vm.swappiness=10" >>/etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

4.4配置免密

ssh-keygen -t rsa     #在cm-server生成密钥对
for num in `seq 12 15`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done

4.5安装java sdk（所有机器）

rpm -qa | grep jdk | xargs rpm -e –nodeps
rpm -ivh /var/www/html/cloudera-repos/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
#update java_home
vim /etc/profile
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
CLASSPATH=$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME CLASSPATH
#refresh profile
source /etc/profile

4.6安装http

yum -y install httpd
systemctl start httpd

4.7下载相关包

mkdir -p /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cm6/6.2.0/redhat7/ -P /var/www/html/cloudera-repos
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.2.0/redhat7/ -P /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.2.0/redhat7/ -P /var/www/html/cloudera-repos
chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6
mkdir -p /opt/cloudera/parcel-repo
cd /opt/cloudera/parcel-repo
wget https://archive.cloudera.com/cdh6/6.2.1/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel
wget https://archive.cloudera.com/cdh6/6.2.1/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha1
wget https://archive.cloudera.com/cdh6/6.2.1/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha256
wget https://archive.cloudera.com/cdh6/6.2.1/parcels/manifest.json
sha1sum CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel | awk ‘{ print $1 }’ > CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

4.8添加yum源

#所有机器上配制源
vim /etc/yum.repos.d/cloudera-repo.repo
[cloudera-repo]
name=cloudera-repo
baseurl=http://172.16.93.11/cloudera-repos/cm6/6.2.0/redhat7/yum/
enabled=1
gpgcheck=0
#所有机器操作
yum makecache
#检查是否成功
yum search cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
#manager上安装
sudo yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

4.9安装数据库

yum install -y MariaDB-server MariaDB-client
#开机自启动mariaDB
systemctl enable mariadb
#Start the MariaDB server:
systemctl start mariadb
#secure_installation（可选）
/usr/bin/mysql_secure_installation
#Installing the MySQL JDBC Driver for MariaDB
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
#Creating Databases for Cloudera Software
mysql -u root –p
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie'; 
GRANT ALL ON scm.* TO 'scm'@'localhost' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'localhost' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'localhost' IDENTIFIED BY 'hue';
GRANT ALL ON hive.* TO 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL ON sentry.* TO 'sentry'@'localhost' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'localhost' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'localhost' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'localhost' IDENTIFIED BY 'oozie';
#检查数据库创建
show databases;
#检查用户授权
SHOW GRANTS FOR 'scm'@'%';
SHOW GRANTS FOR 'amon'@'%';
SHOW GRANTS FOR 'rman'@'%';
SHOW GRANTS FOR 'hue'@'%';
SHOW GRANTS FOR 'hive'@'%';
SHOW GRANTS FOR 'sentry'@'%';
SHOW GRANTS FOR 'nav'@'%';
SHOW GRANTS FOR 'navms'@'%';
SHOW GRANTS FOR 'oozie'@'%';
#Set up the Cloudera Manager Database in shell 
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  scm scm
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  amon amon
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  rman rman
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  hue hue
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  hive hive
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  sentry sentry
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  nav nav
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  navms navms
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql  oozie oozie

4.10启动CDH服务

systemctl start cloudera-scm-server
cd /var/www/html/cloudera-repos/cm6/6.2.0
wget https://archive.cloudera.com/cm6/6.2.0/allkeys.asc

5．CDH配置
用户名密码admin/admin
在这里插入图片描述

可以忽略这个自动TLS自动安装，点击“继续”跳到下一个页面
在这里插入图片描述

输入主机ip或者名称，点击搜索找到主机后点击继续172.16.93.[11-15]
在这里插入图片描述

这里默认设置，下一步即可
在这里插入图片描述

这里也是默认设置
在这里插入图片描述

进入下一步安装jdk，因为之前我们已安装过jdk，这里可以不装，点击“继续“
在这里插入图片描述
在这里插入图片描述
选择需要安装的服务，我们选择了hdfs/hive/spark/yarn/zookeeper
在这里插入图片描述
各个节点选择角色
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

Q&A

Canary 测试无法为 /tmp/.cloudera_health_monitoring_canary_files 创建父目录。
# hdfs dfsadmin -safemode get
Safe mode is OFF
# hdfs dfs -ls /tmp
Found 3 items
d--------- - hdfs supergroup 0 2019-02-18 23:57 /tmp/.cloudera_health_monitoring_canary_files
# hdfs dfs -chmod 777 /tmp/.cloudera_health_monitoring_canary_files

用户权限不够
groupadd supergroup
usermod -a -G supergroup mapred

猜你喜欢

转载自blog.csdn.net/nickyu888/article/details/107686512
今日推荐
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
【转】spring中对控制反转和依赖注入的理解
tms webcore 安装和使用
java程序员进阶相关书籍
SpringMVC接受请求参数、
如何保存训练好的机器学习模型
MyEclipse、Eclipse设置项目JDK的三个地方
商超行业微信小程序开发定制一般多少钱 （行业技术人员解读）
Markdown编辑器语言——30分钟入门到到精通
Linux系统下MongoDB的简单安装与基本操作
Power Strings
每日归档
更多
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022