Hadoop 3.2.1 win10 64位系统 vs2015 编译

Hadoop 3.2.1 win10 64位系统 vs2015 编译

1        环境配置

1.1   JDK下载安装

1.1.1         下载

 JDK 1.8    （jdk1.8.0_102）
   下载地址    ： http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html（按照电脑版本下载安装）

1.1.2         安装

解压到指定文件夹

（1） 安装JDK 
（2）新建系统变量JAVA_HOME=D:\Program Files\Java\jdk1.8.0_102 
（3）编辑系统变量Path，新增%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin

1.2   Marven下载和安装

1.2.1         下载

https://blog.csdn.net/changge458/article/details/53576178

从该网站 http://maven.apache.org/download.cgi 下载

 

1.2.2         安装

解压到文件夹D:\marven\apache-maven-3.6.3

添加系统环境变量：MARVEN_HOME

 

D:\marven\apache-maven-3.6.3\

在系统环境变量path中加入

 

测试mvn是否安装成功，打开cmd，以管理员身份运行，否则容易报错：'mvn' 不是内部或外部命令，也不是可运行的程序

 

查看是否配置成功可在黑窗口中输入 mvn –v 出现如下图所示说明配置成功

 

打开D:\marven\apache-maven-3.6.3\conf\settings.xml 加入阿里云镜像，找到mirrors标签，添加如下阿里云镜像地址，marven下载库从这里下载更快速。

<mirror>

    <id>nexus-aliyun</id>

    <mirrorOf>central</mirrorOf>

    <name>Nexus aliyun</name>

    <url>http://maven.aliyun.com/nexus/content/groups/public</url>

</mirror>

1.3   编译安装protobuff

Protobuff是用于序列化数据的，hadoop编译需要依赖这个库。

1.3.1         下载

ProtocolBuffer 2.5.0    （两个文件protobuf-2.5.0.zip  protoc-2.5.0-win32.zip ）

   下载地址    ： https://github.com/google/protobuf/releases/tag/v2.5.0

   注：除了下载protobuf源码外，还需要下载相应版本的编译过的用于Windows平台的protoc命令（protoc-2.5.0-win32.zip）,该命令用于将.proto文件转化为Java或C++源文件。

1.3.2         安装

① 解压ProtocolBuffer到指定目录

② 解压protoc-2.5.0-win32.zip，将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下

③ 安装ProtocolBuffer，打开CMD命令行

cd C:\WorkSpace\protobuf-2.5.0\java

mvn test

 

mvn install

 

 

protoc –version    这个命令失败

将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src，添加到系统变量Path

 

1.4   Git下载和安装

Git是在window的cmd黑窗口中，可以使用linux的命令。不安装会出现错误

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on project hadoop-project-dist: Command execution failed.: Cannot run program "bash" (in directory "D:\hadoop\hadoop-3.2.1-src\hadoop-project-dist\target"): CreateProcess error=2, 系统找不到指定的文件。 -> [Help 1]

 

1.4.1         下载

 https://git-for-windows.github.io/

1.4.2         安装

（1）
     
      

   （2）、
      

    （3）、
      

    （4）、
      

   （5）、
      

 

1.5   CMake 安装

1.5.1         下载

 https://cmake.org/download/

Windows win64-x64 ZIP  cmake-3.16.0-win64-x64.zip

1.5.2         安装

解压到指定文件夹即可，path环境变量添加bin路径。

D:\hadoop\cmake-3.16.0-win64-x64\bin

 

1.6   Zlib下载安装

1.6.1         下载

http://jaist.dl.sourceforge.net/project/libpng/zlib/1.2.8/

、

1.6.2         安装

解压到指定文件夹，添加zlib系统变量

 

2        编译hadoop

2.1   升级项目Visual Studio的版本

编译hadoop时，自动编译会去根据项目文件找到编译器visual studio去编译，hadoop3.2.1默认是VS2010，现在要用VS2015去编译，需要将项目文件winutils.sln和native.sln升级为vs2015。由于Visual Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，会弹出升级的对话框，升级项目至Visual Studio 2015版本即可：

(1）       Window工具编译

D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\winutils

（2）hadoop.dll编译

D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\native

这个项目编译输出的就是hadoop.dll文件，编辑编译生成按钮，会报很多头文件错误，是因为投啊文件件包含了..\..\..\target\native\javah，这个文件夹内的头文件，编译hadoop时，编译命令自动先复制头文件到此文件夹，然后才能编译。

打开cmd窗口，进入路径D:\hadoop\hadoop-3.2.1-src\，执行下面的命令：

mvn package -Pdist,native-win-DskipTests -Dtar

2.2   Apache Hadoop Common编译错误

2.2.1         convert-ms-winutils错误描述

错误描述

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 12.916 s]

[INFO] Apache Hadoop Common ............................... FAILURE [02:06 min]

[INFO] Apache Hadoop NFS .................................. SKIPPED

[INFO] Apache Hadoop KMS .................................. SKIPPED

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <args> -rf :hadoop-common

 

解决办法：

（1）   确认2.1中两个项目都升级成功，sln文件都是显示Visual studio 14

（2）   VS 2015 x64 native Tools 工具执行命令mvn package -Pdist,native-win -DskipTests -Dtar -e -X

 

2.2.2         Javah错误

错误描述

[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command

 

解决办法

因为D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common路径下的pom.xml文件中，<javahPath>${env.JAVA_HOME}/bin/javah</javahPath>采用env.JAVA_HOME 无法识别，所以讲pom.xml文件中换成javah的绝对路径：D:\Java\jdk1.8.0_181\bin\,一共有两个地方。

 

然后命令加入参数执行–rf :hadoop-common 表示从hadoop-common开始编译：

mvn package -Pdist,native-win-DskipTests –Dtar –rf :hadoop-common

2.3  HDFS Native Client编译错误

2.3.1         错误描述

[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [02:26 min]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec failonerror="true" dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 5:140 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1

around Ant part ...<exec failonerror="true" dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 5:140 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

 

2.3.2         解决方法

D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml文件打开，修改如下部分的true为false；

 

2.4  hadoop-hdfs-native-client :RelWithDebInfo编译错误

2.4.1         错误描述

Finished at: 2019-12-01T18:20:30+08:00

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist.

[ERROR] around Ant part ...<copy todir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 13:101 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

[ERROR] -> [Help 1]

2.4.2         解决方法

D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist.错误是这个目录不存在，则创建这个目录

2.5  执行maven-plugin:1.3.1:exec (pre-dist)失败

2.5.1         错误描述

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on project hadoop-hdfs-native-client: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on project hadoop-hdfs-native-client: Command execution failed.

2.5.2         解决办法

下载安装Cygwin
下载：http://www.cygwin.com/setup-x86_64.exe ,安装
配置D:\cygwin64\bin到PATH

然后打开cygwin64的terminal窗口，执行命令mvn package -Pdist,native-win -DskipTests -Dtar -e -X -rf :hadoop-hdfs-native-client

 

2.6  Msbuild编译错误

2.6.1         错误描述

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "msbuild" (in directory "D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native"): CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒

[ERROR] around Ant part ...<exec failonerror="false" dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="msbuild">... @ 9:143 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

2.6.2         解决办法

下载安装vscode  https://code.visualstudio.com/docs/?dv=win

将路径C:\Program Files (x86)\MSBuild\14.0\Bin添加到path环境变量。

2.7  maven-surefire-plugin编译错误

2.7.1         错误描述

Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test)

2.7.2         解决方法

在D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\pom.xml找到org.apache.maven.plugins，在configuration中添加<testFailureIgnore>true</testFailureIgnore>屏蔽测试错误。

<plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-surefire-plugin</artifactId>

            <configuration>

                       <testFailureIgnore>true</testFailureIgnore>

              <forkCount>${testsThreadCount}</forkCount>

              <reuseForks>false</reuseForks>

              <argLine>${maven-surefire-plugin.argLine} -DminiClusterDedicatedDirs=true</argLine>

              <systemPropertyVariables>

                <testsThreadCount>${testsThreadCount}</testsThreadCount>

                <test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data>

                <test.build.dir>${test.build.dir}/${surefire.forkNumber}</test.build.dir>

                <hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir>

                <!-- Due to a Maven quirk, setting this to just -->

                <!-- surefire.forkNumber won't do the parameter substitution. -->

                <!-- Putting a prefix in front of it like "fork-" makes it -->

                <!-- work. -->

                <test.unique.fork.id>fork-${surefire.forkNumber}</test.unique.fork.id>

              </systemPropertyVariables>

            </configuration>

          </plugin>

2.8  Apache Hadoop Distribution编译错误

2.8.1         错误描述

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (dist) on project hadoop-dist: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

 

2.8.2         解决办法

执行命令mvn package -Pdist,native-win -DskipTests -Dtar -e -X -rf :hadoop-dist，加上-e和-X参数显示详细的错误。会发现是目录结构问题，找不到如下路径。解决方法就是创建缺少的路径，并将生成的jar包成果物复制到路径下

[DEBUG] Executing command line: [bash, D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching, 3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]

cp: cannot stat '/d/hadoop/hadoop-3.2.1-src/hadoop-common-project/hadoop-kms/target/hadoop-kms-3.2.1/*': No such file or directory

2.9  hadoop-dist: exec (toolshooks)编译错误

2.9.1         错误描述

Executing command line: [bash, D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker, 3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target, D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]

找不到文件 - *.tools-builtin.txt

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker: line 137: D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new: No such file or directory

mv: cannot stat 'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new': No such file or directory

Rewriting D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Distribution 3.2.1:

[INFO]

[INFO] Apache Hadoop Distribution ......................... FAILURE [ 25.920 s]

[INFO] Apache Hadoop Client Modules ....................... SKIPPED

[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED

[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  33.530 s

[INFO] Finished at: 2019-12-08T11:56:12+08:00

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (toolshooks) on project hadoop-dist: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (toolshooks) on project hadoop-dist: Command execution failed.

2.9.2         解决方法

（1）错误原因和2.8相似，缺少了文件夹和文件，按照错误提示创建文件夹，找到缺少的文件放入指定文件夹。

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker, 3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target, D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]

找不到文件 - *.tools-builtin.txt

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker: line 137: D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new: No such file or directory

mv: cannot stat 'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new': No such file or directory

Rewriting D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

（2）根据描述是添加了/etc/hadoop文件夹，并且从D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\conf \ 中追到了hadoop-env.sh复制过去，但是发现在执行编译时，总是会自动删除这个文件夹和文件。在执行下面的命令时，删除的：

[DEBUG] Executing command line: [bash, D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching, 3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]

Current directory /d/hadoop/hadoop-3.2.1-src/hadoop-dist/target

$ rm -rf hadoop-3.2.1

$ mkdir hadoop-3.2.1

$ cd hadoop-3.2.1

（3）打开D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching文件，发现dist-layout-stitching中有一个语句run rm -rf "hadoop-${VERSION}"会删除hadoop-3.2.1文件夹，然后再新建，新建之后再复制，所以怎么加etc/hadoop文件夹都会被删除

（4）在文件末尾处添加如下代码，用代码去创建文件夹，并复制hadoop-env.sh文件。

run mkdir "etc"

run cd "etc"

run mkdir "hadoop"

run cd "hadoop"

run copy  "${ROOT}\hadoop-common-project\hadoop-common\src\main\conf\hadoop-env.sh"

 

（5）执行命令>mvn package -Pdist,native-win -DskipTests -Dtar -e -X -rf :hadoop-dist

3       大工告成

[DEBUG]   (f) siteDirectory = D:\hadoop\hadoop-3.2.1-src\hadoop-cloud-storage-project\src\site

[DEBUG]   (f) skip = false

[DEBUG] -- end configuration --

[INFO] No site descriptor found: nothing to attach.

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop Distribution 3.2.1:

[INFO]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 39.961 s]

[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  5.721 s]

[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  10:36 h]

[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  1.471 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  10:37 h

[INFO] Finished at: 2019-12-09T10:05:07+08:00

[INFO] ------------------------------------------------------------------------

自己开发了一个股票智能分析软件，功能很强大，需要的点击下面的链接获取：

https://www.cnblogs.com/bclshuai/p/11380657.html

参考文献

https://blog.csdn.net/qq_37475168/article/details/90746823