Hadoop-6-ソースコードのコンパイル

1.環境を整える

[0] Hadoopソースコード

この記事は、hadoop-2.7.7-srcバージョンに基づいてコンパイルおよびパッケージ化されています

ダウンロードして解凍します

2.7.7のソースコードはコンパイルおよびパッケージ化されているため、apache-tomcat-6.0.48がダウンロードされるため、事前にダウンロードできるため、コンパイルと実行の時間を短縮できます。

hadoop-2.7.7-src/hadoop-common-project/hadoop-kmsとhadoop-2.7.7-src/hadoop-hdfs-project/hadoop-hdfs-httpfsディレクトリにそれぞれサブディレクトリを作成し、downloadsapache-tomcat-6.0.48.tar.gzをディレクトリにコピーします

[1]圧縮パッケージをダウンロードします

jdk-8u211-linux-x64ここではJDK8を選択しますが、JDK7も選択できます
Maven
蟻

1.10.6リリース-最低必要とするJava 8を実行時に
1.9.14のリリースでは、 -の最小必要とするJava 5を実行時に

選択したJDKバージョンに注意してください。JDK7を選択した場合は、バージョン1.9.14を使用してください。
Protobuf-2.5.0
googletest-release-1.5.0

JDK、Maven、Ant、Protobuf、GoogleTestを解凍します。その中で、GoogleTestはProtobufディレクトリに解凍する必要があり、ディレクトリ名が変更されますgtest

JDK、Maven、Antの環境変数を構成します

Protobufディレクトリで、コマンドを個別に実行します。

sh autogen.sh
./configure
make
make install

バージョンを表示

$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

$ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /export/apache-maven-3.6.1
Java version: 1.8.0_211, vendor: Oracle Corporation, runtime: /export/jdk1.8.0_211/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "amd64", family: "unix"

$ ant -version
Apache Ant(TM) version 1.10.6 compiled on May 2 2019

$ protoc --version
libprotoc 2.5.0

jarパッケージをダウンロードするときに高速になるように、Mavenのミラーを構成します

<mirror>
    <id>alimaven</id>
    <mirrorOf>central</mirrorOf>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
</mirror>
<mirror>
    <id>repo1</id>
    <mirrorOf>central</mirrorOf>
    <name>Human Readable Name for this Mirror.</name>
    <url>http://repo1.maven.org/maven2/</url>
</mirror>
<mirror>
    <id>repo2</id>
    <mirrorOf>central</mirrorOf>
    <name>Human Readable Name for this Mirror.</name>
    <url>http://repo2.maven.org/maven2/</url>
</mirror>

[2]依存関係パッケージをダウンロードする

yum install -y autoconf automake libtool glibc-headers gcc-c++ cmake openssl-devel ncurses-devel

2.コンパイルしてパッケージ化します

[1]ツールを追加せずにLinux64ビットシステムでコンパイルするだけです

①動作環境

ラップトップ

プロセッサー：Intel®Core™i5-7200U CPU @ 2.50GHz 2.71 GHz
仮想マシン

プロセッサー：プロセッサーの数は2、プロセッサーあたりのコアの数は2です。

オペレーティングシステム：CentOSリリース6.8（最終版）

②注文を実行する

mvn clean
mvn package -Pdist,native -DskipTests -Dtar

jarパッケージのダウンロード中にシステムがハングした場合は、Ctrl + Cコンパイルを停止してコマンドを再実行してください。

③実施結果

main:
     [exec] $ tar cf hadoop-2.7.7.tar hadoop-2.7.7
     [exec] $ gzip -f hadoop-2.7.7.tar
     [exec] 
     [exec] Hadoop dist tar available at: /hadoop-2.7.7-src/hadoop-dist/target/hadoop-2.7.7.tar.gz
     [exec] 
[INFO] Executed tasks

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.7.7:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  9.443 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  3.441 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  5.180 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  4.112 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.241 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.501 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  7.762 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.587 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  5.265 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.194 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:11 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.243 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  9.756 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.042 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:36 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.569 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.641 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.899 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.039 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.032 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 26.082 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 20.657 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.044 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.394 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 13.435 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.603 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.085 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.511 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  3.605 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.714 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.540 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.026 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.107 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.499 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.026 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.788 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.378 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.100 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.367 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.601 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.966 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  6.613 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.880 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.363 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.570 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.993 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.233 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.210 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.003 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.686 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.086 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.175 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.858 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.420 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.098 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  5.602 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.521 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 10.019 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 26.678 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  6.604 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.603 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.638 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.282 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.021 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 26.823 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:55 min
[INFO] Finished at: 2019-08-21T11:57:53+08:00
[INFO] ------------------------------------------------------------------------

コマンドを再度実行した後、途中で一度偽の死、成功！

hadoop-2.7.7-src/hadoop-dist/target/ディレクトリ構造

drwxr-xr-x. 2 root root      4096 8月  21 11:57 antrun
drwxr-xr-x. 3 root root      4096 8月  21 11:57 classes
-rw-r--r--. 1 root root      1864 8月  21 11:57 dist-layout-stitching.sh
-rw-r--r--. 1 root root       637 8月  21 11:57 dist-tar-stitching.sh
drwxr-xr-x. 9 root root      4096 8月  21 11:57 hadoop-2.7.7
-rw-r--r--. 1 root root 198929775 8月  21 11:57 hadoop-2.7.7.tar.gz
-rw-r--r--. 1 root root     26520 8月  21 11:57 hadoop-dist-2.7.7.jar
-rw-r--r--. 1 root root 399070206 8月  21 11:57 hadoop-dist-2.7.7-javadoc.jar
-rw-r--r--. 1 root root     24049 8月  21 11:57 hadoop-dist-2.7.7-sources.jar
-rw-r--r--. 1 root root     24049 8月  21 11:57 hadoop-dist-2.7.7-test-sources.jar
drwxr-xr-x. 2 root root      4096 8月  21 11:57 javadoc-bundle-options
drwxr-xr-x. 2 root root      4096 8月  21 11:57 maven-archiver
drwxr-xr-x. 3 root root      4096 8月  21 11:57 maven-shared-archive-resources
drwxr-xr-x. 3 root root      4096 8月  21 11:57 test-classes
drwxr-xr-x. 2 root root      4096 8月  21 11:57 test-dir