Hadoop-6-source code compilation
1. Prepare the environment
[0] Hadoop source code
This article is compiled and packaged based on hadoop-2.7.7-src version
Download and unzip
Since the source code of 2.7.7 is compiled and packaged, apache-tomcat-6.0.48 will be downloaded , so it can be downloaded in advance, which can reduce the time of compilation and execution
Create subdirectories in the hadoop-2.7.7-src/hadoop-common-project/hadoop-kmsand hadoop-2.7.7-src/hadoop-hdfs-project/hadoop-hdfs-httpfsdirectories respectively downloads, and copy apache-tomcat-6.0.48.tar.gz to the directories
[1] Download the compressed package
-
jdk-8u211-linux-x64 I choose JDK8 here, you can also choose JDK7
-
1.10.6 release - requires minimum of Java 8 at runtime
1.9.14 release - requires minimum of Java 5 at runtimePay attention to the selected JDK version , if you choose JDK7, use version 1.9.14
Unzip JDK, Maven, Ant, Protobuf and GoogleTest. Among them, GoogleTest needs to be unzipped into the Protobuf directory, and the directory name is changedgtest
Configure the environment variables of JDK, Maven and Ant
In the Protobuf directory, execute the commands separately:
sh autogen.sh
./configure
make
make install
View version
$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
$ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /export/apache-maven-3.6.1
Java version: 1.8.0_211, vendor: Oracle Corporation, runtime: /export/jdk1.8.0_211/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "amd64", family: "unix"
$ ant -version
Apache Ant(TM) version 1.10.6 compiled on May 2 2019
$ protoc --version
libprotoc 2.5.0
Configure the mirror of Maven so that it can be faster when downloading the jar package
<mirror>
<id>alimaven</id>
<mirrorOf>central</mirrorOf>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
</mirror>
<mirror>
<id>repo1</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo1.maven.org/maven2/</url>
</mirror>
<mirror>
<id>repo2</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo2.maven.org/maven2/</url>
</mirror>
[2] Download dependency package
yum install -y autoconf automake libtool glibc-headers gcc-c++ cmake openssl-devel ncurses-devel
2. Compile and package
[1] Simply compile under Linux64 bit system without adding any tools
① Operating environment
-
laptop
Processor: Intel® Core™ i5-7200U CPU @ 2.50GHz 2.71 GHz
-
virtual machine
Processor: The number of processors is 2, the number of cores per processor is 2
Operating system: CentOS release 6.8 (Final)
② execute the order
mvn clean
mvn package -Pdist,native -DskipTests -Dtar
If the system hangs while downloading the jar package, Ctrl + Cstop compiling and re-execute the command
③ Implementation results
main:
[exec] $ tar cf hadoop-2.7.7.tar hadoop-2.7.7
[exec] $ gzip -f hadoop-2.7.7.tar
[exec]
[exec] Hadoop dist tar available at: /hadoop-2.7.7-src/hadoop-dist/target/hadoop-2.7.7.tar.gz
[exec]
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.7.7:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 9.443 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.441 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 5.180 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 4.112 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.241 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.501 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 7.762 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.587 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 5.265 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 3.194 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:11 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.243 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.756 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.042 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:36 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.569 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 4.641 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.899 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.039 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.032 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 26.082 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 20.657 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.044 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 7.394 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 13.435 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.603 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 5.085 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.511 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 3.605 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 4.714 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 2.540 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.107 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.499 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 3.788 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.378 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.100 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.367 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.601 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 2.966 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 6.613 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 3.880 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 4.363 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.570 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.993 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 2.233 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 3.210 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 6.003 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.686 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.086 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.175 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.858 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.420 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.098 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 5.602 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.521 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 10.019 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 26.678 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.604 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.603 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.638 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.282 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.021 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 26.823 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:55 min
[INFO] Finished at: 2019-08-21T11:57:53+08:00
[INFO] ------------------------------------------------------------------------
Fake death once halfway, after executing the command again, success!
hadoop-2.7.7-src/hadoop-dist/target/Directory Structure
drwxr-xr-x. 2 root root 4096 8月 21 11:57 antrun
drwxr-xr-x. 3 root root 4096 8月 21 11:57 classes
-rw-r--r--. 1 root root 1864 8月 21 11:57 dist-layout-stitching.sh
-rw-r--r--. 1 root root 637 8月 21 11:57 dist-tar-stitching.sh
drwxr-xr-x. 9 root root 4096 8月 21 11:57 hadoop-2.7.7
-rw-r--r--. 1 root root 198929775 8月 21 11:57 hadoop-2.7.7.tar.gz
-rw-r--r--. 1 root root 26520 8月 21 11:57 hadoop-dist-2.7.7.jar
-rw-r--r--. 1 root root 399070206 8月 21 11:57 hadoop-dist-2.7.7-javadoc.jar
-rw-r--r--. 1 root root 24049 8月 21 11:57 hadoop-dist-2.7.7-sources.jar
-rw-r--r--. 1 root root 24049 8月 21 11:57 hadoop-dist-2.7.7-test-sources.jar
drwxr-xr-x. 2 root root 4096 8月 21 11:57 javadoc-bundle-options
drwxr-xr-x. 2 root root 4096 8月 21 11:57 maven-archiver
drwxr-xr-x. 3 root root 4096 8月 21 11:57 maven-shared-archive-resources
drwxr-xr-x. 3 root root 4096 8月 21 11:57 test-classes
drwxr-xr-x. 2 root root 4096 8月 21 11:57 test-dir