Hadoop-6-源码编译
1、准备环境
【0】Hadoop源码
本文是基于hadoop-2.7.7-src版本进行编译打包的
下载并解压缩
由于2.7.7源码在编译打包的时候,会下载apache-tomcat-6.0.48,所以可以提前下载,这样可以减少编译执行的时间
分别在hadoop-2.7.7-src/hadoop-common-project/hadoop-kms
和hadoop-2.7.7-src/hadoop-hdfs-project/hadoop-hdfs-httpfs
目录里创建子目录downloads
,并把apache-tomcat-6.0.48.tar.gz复制到目录里
【1】下载压缩包
-
jdk-8u211-linux-x64 我这里选择的是JDK8,你也可以选择JDK7
-
1.10.6 release - requires minimum of Java 8 at runtime
1.9.14 release - requires minimum of Java 5 at runtime注意所选择的JDK版本,如果选择JDK7,则使用1.9.14版本
解压缩JDK、Maven、Ant、Protobuf和GoogleTest。其中,GoogleTest需要解压缩到Protobuf目录里,并修改目录名为gtest
配置好JDK、Maven和Ant的环境变量
在Protobuf目录里,分别执行命令:
sh autogen.sh
./configure
make
make install
查看版本
$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
$ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /export/apache-maven-3.6.1
Java version: 1.8.0_211, vendor: Oracle Corporation, runtime: /export/jdk1.8.0_211/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "amd64", family: "unix"
$ ant -version
Apache Ant(TM) version 1.10.6 compiled on May 2 2019
$ protoc --version
libprotoc 2.5.0
配置Maven的镜像,这样下载jar包时,可以快一些
<mirror>
<id>alimaven</id>
<mirrorOf>central</mirrorOf>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
</mirror>
<mirror>
<id>repo1</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo1.maven.org/maven2/</url>
</mirror>
<mirror>
<id>repo2</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo2.maven.org/maven2/</url>
</mirror>
【2】下载依赖包
yum install -y autoconf automake libtool glibc-headers gcc-c++ cmake openssl-devel ncurses-devel
2、编译打包
【1】单纯地在Linux64位系统下编译,没有添加任何工具
① 运行环境
-
笔记本电脑
处理器:Intel® Core™ i5-7200U CPU @ 2.50GHz 2.71 GHz
-
虚拟机
处理器:处理器数量2,每个处理器的内核数量2
操作系统:CentOS release 6.8 (Final)
② 执行命令
mvn clean
mvn package -Pdist,native -DskipTests -Dtar
如果在下载jar包的过程中,系统假死,则Ctrl + C
停止编译,重新执行命令即可
③ 执行结果
main:
[exec] $ tar cf hadoop-2.7.7.tar hadoop-2.7.7
[exec] $ gzip -f hadoop-2.7.7.tar
[exec]
[exec] Hadoop dist tar available at: /hadoop-2.7.7-src/hadoop-dist/target/hadoop-2.7.7.tar.gz
[exec]
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.7.7:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 9.443 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.441 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 5.180 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 4.112 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.241 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.501 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 7.762 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.587 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 5.265 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 3.194 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:11 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.243 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.756 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.042 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:36 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.569 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 4.641 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.899 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.039 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.032 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 26.082 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 20.657 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.044 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 7.394 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 13.435 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.603 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 5.085 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.511 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 3.605 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 4.714 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 2.540 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.107 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.499 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 3.788 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.378 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.100 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.367 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.601 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 2.966 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 6.613 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 3.880 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 4.363 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.570 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.993 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 2.233 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 3.210 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 6.003 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.686 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.086 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.175 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.858 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.420 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.098 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 5.602 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.521 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 10.019 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 26.678 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.604 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.603 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.638 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.282 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.021 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 26.823 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:55 min
[INFO] Finished at: 2019-08-21T11:57:53+08:00
[INFO] ------------------------------------------------------------------------
中途假死过一次,再次执行命令后,成功!
hadoop-2.7.7-src/hadoop-dist/target/
目录结构
drwxr-xr-x. 2 root root 4096 8月 21 11:57 antrun
drwxr-xr-x. 3 root root 4096 8月 21 11:57 classes
-rw-r--r--. 1 root root 1864 8月 21 11:57 dist-layout-stitching.sh
-rw-r--r--. 1 root root 637 8月 21 11:57 dist-tar-stitching.sh
drwxr-xr-x. 9 root root 4096 8月 21 11:57 hadoop-2.7.7
-rw-r--r--. 1 root root 198929775 8月 21 11:57 hadoop-2.7.7.tar.gz
-rw-r--r--. 1 root root 26520 8月 21 11:57 hadoop-dist-2.7.7.jar
-rw-r--r--. 1 root root 399070206 8月 21 11:57 hadoop-dist-2.7.7-javadoc.jar
-rw-r--r--. 1 root root 24049 8月 21 11:57 hadoop-dist-2.7.7-sources.jar
-rw-r--r--. 1 root root 24049 8月 21 11:57 hadoop-dist-2.7.7-test-sources.jar
drwxr-xr-x. 2 root root 4096 8月 21 11:57 javadoc-bundle-options
drwxr-xr-x. 2 root root 4096 8月 21 11:57 maven-archiver
drwxr-xr-x. 3 root root 4096 8月 21 11:57 maven-shared-archive-resources
drwxr-xr-x. 3 root root 4096 8月 21 11:57 test-classes
drwxr-xr-x. 2 root root 4096 8月 21 11:57 test-dir