Centos7下Hadoop3.x源码编译(Zstd问题已解决)

其他 2019-04-17 11:20:57 阅读次数: 0

工具

组件 版本 下载地址
JDK 1.8.0_131 https://www.oracle.com/technetwork/java/javase/downloads/index.html
Hadoop 3.1.1 http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz
Maven 3.6.0 http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz
CMake 3.13.1 https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz
ProtocolBuffer 2.5.0 https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q

编译完成生成的软件包

软件包名 下载地址 描述
hadoop-3.1.1.tar.gz https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw CentOS7.4下hadoop3.1.1源码编译
native-hadoop-3.1.1-with-sztd.tar.gz https://download.csdn.net/download/weixin_43858859/10832098 hadoop3.1.1源码编译生成的native，含zstd

编译所需要工具和软件包
下载hadoop-3.1.1-src.tar.gz源码包，解压并查看BUILDING.txt

[hadoop@hadoop src]$cat BUILDING.txt
...
Requirements:
* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)
...

软件安装

安装JDK
下载jdk-8u131-linux-x64.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/java目录，安装jdk

[root@hadoop ~]# mkdir /usr/local/maven
[root@hadoop ~]#

解压jdk

[hadoop@hadoop src]$ tar -zxvf /home/hadoop/src/jdk-8u131-linux-x64.tar.gz -C /usr/java/

修改属主

[hadoop@hadoop ~]$ chown -R hadoop:hadoop /usr/java

配置环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD java environment
export JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证JDK

[hadoop@hadoop ~]$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[hadoop@hadoop ~]$

安装Maven
下载apache-maven-3.6.0-bin.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/maven目录，安装maven

[root@hadoop ~]# mkdir /usr/local/maven  ##创建maven安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/maven  ##修改目录属主

解压maven

[hadoop@hadoop src]$ tar -zxvf apache-maven-3.6.0-bin.tar.gz -C /usr/local/maven/

配置maven环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD maven environment
export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.0
export PATH=$MAVEN_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证Maven

[hadoop@hadoop ~]$ mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
Maven home: /usr/local/maven/apache-maven-3.6.0
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
[hadoop@hadoop ~]$

配置国内maven仓库源

[hadoop@hadoop conf]$ vim /usr/local/maven/apache-maven-3.6.0/conf/settings.xml 
...
<mirrors>
    <mirror>
        <id>nexus-aliyun</id>
        <mirrorOf>central</mirrorOf>
        <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>
</mirrors>
...

安装工具包（root）
安装gcc make 工具

[root@hadoop ~]# yum install -y gcc* make

安装压缩工具

[root@hadoop ~]# yum -y install snappy*  bzip2* lzo* zlib*  lz4* gzip*

安装一些基本工具

[root@hadoop ~]# yum -y install openssl* svn ncurses* autoconf automake libtool

安装openssl-devel

[root@hadoop ~]# yum -y install openssl-devel

安装zstd
安装扩展源 epel

[root@hadoop ~]# yum -y install epel-release

安装zstd

[root@hadoop ~]# yum -y install *zstd*

安装CMake
CMake 3.1 或者更新版本，此处我们使用3.13.1（cmake-3.13.1.tar.gz）
下载并上传cmake-3.13.1.tar.gz到/home/hadoop/src目录下
解压cmake

[hadoop@hadoop src]$ tar -zxvf cmake-3.13.1.tar.gz

切换到源码包目录

[hadoop@hadoop src]$ cd cmake-3.13.1

检查依赖

[hadoop@hadoop cmake-3.13.1]$ ./bootstrap

编译安装cmake（root）
使用root用户编译安装

[hadoop@hadoop cmake-3.13.1]$ su root
Password: 
[root@hadoop cmake-3.13.1]# make && make install

查看cmake版本

[hadoop@hadoop ~]$ cmake -version
cmake version 3.13.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[hadoop@hadoop ~]$

安装ProtocolBuffer
下载protobuf-2.5.0.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/protobuf目录，安装ProtocolBuffer

[root@hadoop ~]# mkdir /usr/local/protobuf  ##创建ProtocolBuffer安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/protobuf  ##修改目录属主

解压ProtocolBuffer

[hadoop@hadoop src]$ tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/src/

切换到/home/hadoop/src/protobuf-2.5.0目录

[hadoop@hadoop ~]$ cd /home/hadoop/src/protobuf-2.5.0
[hadoop@hadoop protobuf-2.5.0]$

检查库
/usr/local/protobuf是protobuf安装目录

[hadoop@hadoop protobuf-2.5.0]$ ./configure --prefix=/usr/local/protobuf

编译安装protobuf

[hadoop@hadoop protobuf-2.5.0]$ make && make install
...
make[3]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
[hadoop@hadoop protobuf-2.5.0]$

配置protobuf环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD protobuf  environment
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证protobuf

[hadoop@hadoop protobuf]$ protoc --version
libprotoc 2.5.0

编译Hadoop

下载hadoop-3.1.1-src.tar.gz并上传至/home/hadoop/src
解压hadoop源码包

[hadoop@hadoop src]$ tar -zxvf hadoop-3.1.1-src.tar.gz

进入到hadoop-3.1.1-src目录

[hadoop@hadoop src]$ cd hadoop-3.1.1-src

修改pom文件（42 行）
将hadoop-annotations的值由provide改成compile

[hadoop@hadoop src]$ vim hadoop-common-project/hadoop-auth/pom.xml 
...
<dependencies>
    <dependency>
      <!-- Used, even though 'mvn dependency:analyze' doesn't find it -->
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-annotations</artifactId>
      <scope>compile</scope>
</dependency>
...

务必保持网络畅通，如果出现缺少某个文件，则要先清理maven(使用命令 mvn clean) 再重新编译。

[root@hadoop hadoop-3.1.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar  ##使用root编译

出现下面结果则成功编译
在这里插入图片描述
编译完成后的包文件位置
/home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

...
[INFO] --- exec-maven-plugin:1.3.1:exec (tar) @ hadoop-dist ---
$ tar cf hadoop-3.1.1.tar hadoop-3.1.1
$ gzip -f hadoop-3.1.1.tar

Hadoop dist tar available at: /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

[INFO] 
...

hadoop native检查

进入到编译好软件包目录下

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$

检查native

[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 18:54:46,459 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 18:54:46,476 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 18:54:46,603 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 18:54:46,803 INFO util.ExitUtil: Exiting with status 1: ExitException

问题解决(zstd)

安装zstd需要先安装扩展源，否则会报找不到zstd包，如下：

[root@hadoop ~]# yum -y install *zstd*
No package zstd available.
Error: Nothing to do
[root@hadoop ~]#

没有安装zstd出现的问题
检查native时，缺失zstd

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 07:35:12,808 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 07:35:12,826 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 07:35:12,851 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 07:35:12,967 INFO util.ExitUtil: Exiting with status 1: ExitException
[hadoop@hadoop bin]$

转载：https://blog.csdn.net/weixin_43858859/article/details/84673249

猜你喜欢

转载自blog.csdn.net/weixin_42868638/article/details/89315768
今日推荐
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
“百模大战”必有一战 | 2024中国“百模大战”竞争格局分析
最强开源大模型 Llama 3 上架 Gitee AI
虽然老乡鸡开源的不是代码，但背后的原因却让人很暖心
富文本编辑器 Quill 2.0 重磅发布，特性、可靠性与开发者体验大幅提升
周排行
SVN同步出现问题
解决 nginx 出现 413 Request Entity Too Large 的问题
第一节 区块链服务BaaS的总体架构以及基本模块设计的一种方案
ITeye 2013年度盘点——社区赠书书单
IDEA / git 和github 的新手使用教程 史上最简单的 IntelliJ IDEA 教程 史上最简单的 GitHub 教程
测试工程方法：测试用例设计综合策略
Spark优化(三)：对多次使用的RDD进行持久化
使用STM32 ST-LINK Utility 设置读保护后不能运行
exgcd 解同余方程ax=b(%n)
Android使用脚本进行多渠道打包
每日归档
更多
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)
2024-04-15(42)
2024-04-14(0)
2024-04-13(119)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022