CDH5搭建

其他 2020-03-03 10:26:12 阅读次数: 0

本文介绍了CDH5的搭建步骤及遇到的问题和解决方案。

参考自：https://www.cnblogs.com/zhangleisanshi/p/7575579.html及https://www.cnblogs.com/pojishou/archive/2017/01/12/6267616.html

准备工作

本案例使用公司现成的三台机器，如需自己搭建和配置虚拟机，可参考别的教程。

  • 三节点的集群，查看机器配置如下：

IP地址

主机别名

说明

可用内存

可用硬盘

10.1.4.18

b3

主节点master(namenode),datanode

4.3GB

44GB

10.1.4.19

b4

datanode

4.9GB

44GB

10.1.4.20

b5

datanode

4.8GB

44GB
  • 机器系统信息：
[root@bi-zhaopeng03 songzehao]# cat /proc/version 
Linux version 3.10.0-229.el7.x86_64 ([email protected]) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42 UTC 2015
[root@bi-zhaopeng03 songzehao]# uname -a
Linux bi-zhaopeng03 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

搭建工作

JDK安装

因为hadoop是由Java写的，很多启动程序都需要java的环境。JDK安装没什么特别的介绍，安装完查看信息。

[root@bi-zhaopeng03 songzehao]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
[root@bi-zhaopeng03 songzehao]# echo $JAVA_HOME
/usr/local/java

相关安装介质获取

  • Mysql：

https://dev.mysql.com/get/Downloads/MySQL-5.5/mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz

  • JDBC驱动：

https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz

  • cloudera manager包：

http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz

  • CDH包：

http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel

http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1

http://archive.cloudera.com/cdh5/parcels/5.7.2/manifest.json

配置hostname(每台机器)

[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=bi-zhaopeng03
[root@bi-zhaopeng03 songzehao]# vi /etc/hosts
10.1.4.18  bi-zhaopeng03 b3 （说明：b3也是hostname的别名）
10.1.4.19  bi-zhaopeng04 b4
10.1.4.20  bi-zhaopeng05 b5

关闭selinux(每台机器)

配置selinux需重启后生效

[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/selinux
SELINUX=disabled
[root@bi-zhaopeng03 songzehao]# sestatus -v
SELinux status: disabled

 关闭防火墙(每台机器)

[root@bi-zhaopeng03 songzehao]# systemctl stop firewalld
[root@bi-zhaopeng03 songzehao]# systemctl disable firewalld
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
[root@bi-zhaopeng03 songzehao]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
    Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
    Active: inactive (dead)

NTP网络时间同步 (每台机器)

首先安装ntp服务

[root@bi-zhaopeng03 songzehao]# yum -y install ntp
或
[root@bi-zhaopeng03 songzehao]# yum install -y ntpdate

然后同步时间，如下是CentOS系统使用NTP来从一个时间服务器同步，把当前时区调整为上海就是+8区，想改其他时区也可以去看看/usr/share/zoneinfo目录

[root@bi-zhaopeng03 songzehao]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
[root@bi-zhaopeng03 songzehao]# ntpdate us.pool.ntp.org

启动和查看ntp服务

[root@bi-zhaopeng03 songzehao]# systemctl start ntpd  //启动ntp服务
[root@bi-zhaopeng03 songzehao]# systemctl status ntpd //查看ntp服务状态

SSH无密码登录配置 (每台机器)

检验机器之间是否已配置好SSH免密登录，可ssh hostname测试，我这里的三台机器已被配置好。下面贴出其他小伙伴的操作示范

[root@cdh1 /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1d:e9:b4:ed:1d:e5:c6:a7:f3:23:ac:02:2b:8c:fc:ca root@cdh1
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|           .     |
|          +     .|
|         + +   + |
|        S + . . =|
|       .   . . +.|
|  . o   o   o +  |
|  .o o . .   o + |
|   Eo..   ... . o|
+-----------------+
[root@cdh1 /]# ssh-copy-id 192.168.42.129
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.42.129'"
and check to make sure that only the key(s) you wanted were added.

MySQL安装

可参考文首的文章使用rpm方式安装，这里参考百度经验：https://jingyan.baidu.com/article/a378c9609eb652b3282830fd.html，解压安装（如下仅为解压操作，省略的部分配置步骤安装百度经验即可）

[root@bi-zhaopeng03 songzehao]# tar mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz
[root@bi-zhaopeng03 songzehao]# cp mysql-5.5.62-linux-glibc2.12-x86_64 /usr/local/mysql -r
...

安装完查看信息

[root@bi-zhaopeng03 songzehao]# mysql --help |grep Distrib
mysql  Ver 14.14 Distrib 5.5.62, for linux-glibc2.12 (x86_64) using readline 5.1

修改密码解压JDBC驱动（需在各节点创建目录/usr/share/java/，并将驱动包重命名后放进去）

[root@bi-zhaopeng03 songzehao]# mysql -u root -p          //登录进行去更改密码
mysql> SET PASSWORD=PASSWORD('123456');
mysql> update user set host='%' where user='root' and host='localhost';   //允许mysql远程访问
Query OK, 1 row affected (0.05 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

[root@bi-zhaopeng03 songzehao]# chkconfig mysql on   //配置开机启动
[root@bi-zhaopeng03 songzehao]# tar -zcvf mysql-connector-java-5.1.44.tar.gz // 解压mysql-connector-java-5.1.44.tar.gz得到mysql-connector-java-5.1.44-bin.jar
[root@bi-zhaopeng03 songzehao]# mkdir /usr/share/java // 在各节点创建java文件夹
[root@bi-zhaopeng03 songzehao]# cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar //将mysql-connector-java-5.1.44-bin.jar拷贝到/usr/share/java路径下并重命名为mysql-connector-java.jar

创建数据库

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
 create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
grant all on *.* to root@"%" Identified by "123456";

安装Cloudera-Manager

1.解压cm tar包到所有服务器指定目录（或者在主节点解压好，然后通过scp到各个节点同一目录下）

[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager
[root@bi-zhaopeng03 songzehao]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager

2.创建cloudera-scm用户（所有节点）

[root@bi-zhaopeng03 songzehao]# useradd --system --home=/opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3.在主节点创建cloudera-manager-server的本地元数据保存目录 

[root@bi-zhaopeng03 songzehao]# mkdir /var/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera-manager

4.配置从节点cloudera-manger-agent指向主节点服务器

[root@bi-zhaopeng04 ~]# vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini
将server_host改为CMS所在的主机名即b3
# Hostname of the CM server.
server_host=b3

5.主节点中创建parcel-repo仓库目录

[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcel-repo
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
[root@bi-zhaopeng03 songzehao]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo
注意：其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉

 6.创建parcels目录（所有节点）

[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcels
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
解释：Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/opt/cloudera/parcels目录中

 7.初始脚本配置数据库scm_prepare_database.sh（在主节点上）

[root@bi-zhaopeng03 songzehao]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指：
mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。
-b3：数据库建立在b3主机上面,也就是主节点上面。
-uroot：root身份运行mysql。-123456：mysql的root密码是***。
--scm-host b3：CMS的主机，一般是和mysql安装的主机是在同一个主机上,最后三个参数是：数据库名，数据库用户名，数据库密码。

此处执行可能会报权限不够的错，参考以下或另行处理即可 

update user set PASSWORD=PASSWORD('123456') where user='root';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'b3' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;

此处执行可能报以下错：

[root@bi-zhaopeng03 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
JAVA_HOME=/usr/local/java
Verifying that we can write to /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-server
2019-06-05 11:21:14,867 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://b3/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Can't create database 'scmdbn'; database exists
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
        at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
        at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
        at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
        at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
        at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
2019-06-05 11:21:14,870 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Stack Trace:
java.sql.SQLException: Can't create database 'scmdbn'; database exists
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
        at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
        at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
        at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
        at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
        at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
--> Error 1, giving up (use --force if you wish to ignore the error)

可尝试drop掉数据库scmdbn

drop database if exists scmdbn;

也可尝试参考此解决方案：https://bbs.csdn.net/topics/391836526

重新执行初始化CM5数据库命令
/opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -p123456 --scm-host localhost scmdbn scmdbu scmdbp
再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-server start 启动服务端。
再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-agent start 启动客户端。

8.启动主节点

[root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-server on
[root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-server
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
[root@bi-zhaopeng03 songzehao]# service cloudera-scm-server start
//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-server restart

 

9.启动cloudera-scm-agent（所有节点）

[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent
[root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
[root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-agent on
[root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-agent
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
[root@bi-zhaopeng03 songzehao]# service cloudera-scm-agent start
//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-agent restart

在浏览器安装CDH

面对一个陌生环境的搭建，前面肯定有各种想到想不到查到查不到的问题，要把握住两点，

  • 要相信所有问题最终都能被解决掉；
  • 细心查看日志（/opt/cloudera-manager/cm-5.7.2/log/cloudera-scm-server/ cloudera-scm-server.log）和数据库脚本（/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh）内容，分析出错原因。

等待主节点完成启动就在浏览器中进行操作了，进入b3:7180 默认使用admin admin登录，在浏览器中使用操作安装，可直接参考https://www.cnblogs.com/zhangleisanshi/p/7575579.html与https://www.cnblogs.com/pojishou/archive/2017/01/12/6267616.html

（1）注意在【群集设置 - 选择您要再群集上安装的CDH5服务】步骤里，选择要安装的服务组合，量力而行，我的机器配置较低，所以暂时只选择了“核心Hadoop”组合，之后需要其他组件另行安装即可。

（2）注意在【群集安装 - 检查主机正确性】步骤里，若报错“检查器在所有主机上运行失败.”或“该群集的主机上未安装 Java”，查看server日志，应该会报错“JAVA_HOME is not set and could not be found”。这时候，我们的jdk肯定是正常安装了，所以去查看/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh，可以发现它认定的JAVA8_HOME中不包含“/usr/local/java”，因此一种方式是软链接到它指定的目录，另一种是在它的集合中加进去“/usr/local/java”，如下：

local JAVA8_HOME_CANDIDATES=(
  '/usr/local/java'
  '/usr/java/jdk1.8'
  '/usr/java/jre1.8'
  '/usr/lib/jvm/j2sdk1.8-oracle'
  '/usr/lib/jvm/j2sdk1.8-oracle/jre'
  '/usr/lib/jvm/java-8-oracle'
)

最终此步骤成功

 （3）注意在【群集设置 - 数据库】步骤里，若报错“A package was not selected.”，如下：

莫名其妙，日志也没有追加新的信息，返回几步重新再点过来就好了。 

安装成功

  1. hadoop配置文件目录： /etc/hadoop/conf（包括core-site.xml，hdfs-site.xml等）
  2. hdfs文件系统地址：cat /etc/hadoop/conf/core-site.xml，查找fs.defaultFS = hdfs://bi-zhaopeng03:8020
  3. cloudera-manager的web管理地址：主节点ip:7180 
  4. hdfs的web管理地址：主节点ip:50070
  5. yarn的web管理地址：主节点ip:8088
  6. hue的web管理地址：主节点ip:8888

测试工作

在集群的一台机器上执行以下模拟圆周率Pi（π）的示例程序：

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

通过YARN的Web管理界面（http://10.1.4.18:8088）也可以看到MapReduce的执行状态： 

查看MapReduce应用程序的完整日志：

[root@bi-zhaopeng04 ~]# sudo -u hdfs yarn logs -applicationId application_1559721784881_0001

MapReduce执行过程中终端的输出如下：

[root@bi-zhaopeng04 ~]# sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
Number of Maps  = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
19/06/06 14:08:23 INFO client.RMProxy: Connecting to ResourceManager at bi-zhaopeng03/10.1.4.18:8032
19/06/06 14:08:23 INFO input.FileInputFormat: Total input paths to process : 10
19/06/06 14:08:23 INFO mapreduce.JobSubmitter: number of splits:10
19/06/06 14:08:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559721784881_0001
19/06/06 14:08:24 INFO impl.YarnClientImpl: Submitted application application_1559721784881_0001
19/06/06 14:08:24 INFO mapreduce.Job: The url to track the job: http://bi-zhaopeng03:8088/proxy/application_1559721784881_0001/
19/06/06 14:08:24 INFO mapreduce.Job: Running job: job_1559721784881_0001
19/06/06 14:08:29 INFO mapreduce.Job: Job job_1559721784881_0001 running in uber mode : false
19/06/06 14:08:29 INFO mapreduce.Job:  map 0% reduce 0%
19/06/06 14:08:34 INFO mapreduce.Job:  map 10% reduce 0%
19/06/06 14:08:37 INFO mapreduce.Job:  map 20% reduce 0%
19/06/06 14:08:40 INFO mapreduce.Job:  map 30% reduce 0%
19/06/06 14:08:43 INFO mapreduce.Job:  map 40% reduce 0%
19/06/06 14:08:46 INFO mapreduce.Job:  map 50% reduce 0%
19/06/06 14:08:49 INFO mapreduce.Job:  map 60% reduce 0%
19/06/06 14:08:52 INFO mapreduce.Job:  map 70% reduce 0%
19/06/06 14:08:55 INFO mapreduce.Job:  map 80% reduce 0%
19/06/06 14:08:58 INFO mapreduce.Job:  map 90% reduce 0%
19/06/06 14:09:06 INFO mapreduce.Job:  map 90% reduce 30%
19/06/06 14:13:30 INFO mapreduce.Job:  map 90% reduce 0%
19/06/06 14:13:33 INFO mapreduce.Job:  map 100% reduce 0%
19/06/06 14:13:38 INFO mapreduce.Job:  map 100% reduce 100%
19/06/06 14:13:38 INFO mapreduce.Job: Job job_1559721784881_0001 completed successfully
19/06/06 14:13:38 INFO mapreduce.Job: Counters: 50
        File System Counters
                FILE: Number of bytes read=91
                FILE: Number of bytes written=1310718
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2680
                HDFS: Number of bytes written=215
                HDFS: Number of read operations=43
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=3
        Job Counters 
                Killed reduce tasks=1
                Launched map tasks=10
                Launched reduce tasks=2
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=17328
                Total time spent by all reduces in occupied slots (ms)=272280
                Total time spent by all map tasks (ms)=17328
                Total time spent by all reduce tasks (ms)=272280
                Total vcore-seconds taken by all map tasks=17328
                Total vcore-seconds taken by all reduce tasks=272280
                Total megabyte-seconds taken by all map tasks=17743872
                Total megabyte-seconds taken by all reduce tasks=278814720
        Map-Reduce Framework
                Map input records=10
                Map output records=20
                Map output bytes=180
                Map output materialized bytes=340
                Input split bytes=1500
                Combine input records=0
                Combine output records=0
                Reduce input groups=2
                Reduce shuffle bytes=340
                Reduce input records=20
                Reduce output records=0
                Spilled Records=40
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=589
                CPU time spent (ms)=5390
                Physical memory (bytes) snapshot=4883341312
                Virtual memory (bytes) snapshot=31219437568
                Total committed heap usage (bytes)=4787273728
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1180
        File Output Format Counters 
                Bytes Written=97
Job Finished in 315.215 seconds
Estimated value of Pi is 3.14800000000000000000

机器比较慢，执行了315s，不过结果也是成功计算了出来，说明我们的CDH5已搭建成功。

DayDayUp丶
发布了62 篇原创文章 · 获赞 22 · 访问量 7万+
私信 关注

猜你喜欢

转载自blog.csdn.net/songzehao/article/details/91044032
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
Java自定义时间格式
同步整形电路
在开发中最最最常用的字符串的属性大集合
Linux 查看端口占用并杀掉
Java基础四：ArrayList
多线程之死锁就是这么简单
mysql 基础命令集
awk 命令详解
Centos6.3编译安装nginx+php步骤
OCR （Optical Character Recognition，光学字符识别）
每日归档
更多
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022