Hadoop——源码编译

1.准备源码包

源码下载地址:点击进入官网下载地址
这里写图片描述

博主将hadoop正在维护的几个版本的最新源码包都下载了下来,本篇博客博主将针对2.7.6版本进行编译,其他版本的编译大同小异

2.BUILDING.txt

解压hadoop-2.7.6-src.tar.gz包,第一个文件就是BUILDING.txt文件,它是官方提供的构建Hadoop的说明,下面是该文件的部分截取

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------

翻译可知:

1.针对类Unix系统而言
2.第一次构建时必须保持网络通畅来获取maven和hadoop依赖,当本地已经有所需依赖时则可以不考虑网络问题
3.需要JDK、Maven、ProtocolBuffer、CMake、zlib、openssl
4.Findbugs和FUSE可选

3.安装依赖

[root@centos6-1 ~]# yum -y install zlib-devel
[root@centos6-1 ~]# yum -y install openssl-devel
[root@centos6-1 ~]# yum -y install cmake 
[root@centos6-1 ~]# cmake -version
cmake version 2.8.12.2
[root@centos6-1 ~]# 

4.安装配置JDK

//解压安装
[root@centos6-1 servers]# tar -zvxf jdk-8u65-linux-x64.tar.gz
//文件末尾追加配置环境变量
[root@centos6-1 servers]# vi /etc/profile
//追加内容如下
JAVA_HOME=/export/servers/jdk1.8.0_65
JRE_HOME=$JAVA_HOME/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
//更新环境变量
[root@centos6-1 servers]# source /etc/profile
[root@centos6-1 servers]# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
[root@centos6-1 servers]# 

5.安装配置Maven

//解压安装
[root@centos6-1 servers]# tar -zvxf apache-maven-3.5.4-bin.tar.gz
[root@centos6-1 apache-maven-3.5.4]# pwd  
/export/servers/apache-maven-3.5.4
//环境变量
export MAVEN_HOME=/export/servers/apache-maven-3.5.4
export PATH=.:$PATH:$MAVEN_HOME/bin
//生效
[root@centos6-1 apache-maven-3.5.4]# source /etc/profile
//检查
[root@centos6-1 apache-maven-3.5.4]# mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /export/servers/apache-maven-3.5.4
Java version: 1.8.0_65, vendor: Oracle Corporation, runtime: /export/servers/jdk1.8.0_65/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-573.el6.x86_64", arch: "amd64", family: "unix"
[root@centos6-1 apache-maven-3.5.4]# 

6.安装ProtocolBuffer

先安装gcc、gcc-c++、make

[root@centos6-1 apache-maven-3.5.4]# yum install gcc
[root@centos6-1 apache-maven-3.5.4]# yum install gcc-c++
[root@centos6-1 apache-maven-3.5.4]# yum install make

解压编译安装

[root@centos6-1 apache-maven-3.5.4]# cd ..
[root@centos6-1 servers]# ll
total 2360
drwxr-xr-x.  6 root root     4096 Jul  3 21:39 apache-maven-3.5.4
drwxr-xr-x.  8 uucp   143    4096 Oct  7  2015 jdk1.8.0_65
-rw-r--r--.  1 root root  2401901 Jun 26 08:40 protobuf-2.5.0.tar.gz
drwxr-xr-x. 10  501 games    4096 Nov  5  2012 zookeeper-3.4.5
[root@centos6-1 servers]# [root@centos6-1 servers]# tar -zvxf protobuf-2.5.0.tar.gz
[root@centos6-1 servers]# cd protobuf-2.5.0
[root@centos6-1 protobuf-2.5.0]# ./configure
[root@centos6-1 protobuf-2.5.0]# make
[root@centos6-1 protobuf-2.5.0]# make install
[root@centos6-1 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@centos6-1 protobuf-2.5.0]# 

7.解压编译hadoop

[root@centos6-1 protobuf-2.5.0]# cd ..
[root@centos6-1 servers]# ll
total 45912
drwxr-xr-x.  6 root   root      4096 Jul  3 21:39 apache-maven-3.5.4
-rwxrwxrwx.  1 root   root  46994526 Jul  2 17:02 hadoop-2.7.6-src.tar.gz
drwxr-xr-x.  8 uucp     143     4096 Oct  7  2015 jdk1.8.0_65
drwxr-xr-x. 10 109965  5000     4096 Jul  3 21:54 protobuf-2.5.0
drwxr-xr-x. 10    501 games     4096 Nov  5  2012 zookeeper-3.4.5
[root@centos6-1 servers]# tar -zvxf hadoop-2.7.6-src.tar.gz 
[root@centos6-1 servers]# cd hadoop-2.7.6-src
[root@centos6-1 hadoop-2.7.6-src]# mvn package -Pdist,native -DskipTests -Dtar

编译成功

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.571 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.021 s]
[INFO] Apache Hadoop Distribution 2.7.6 ................... SUCCESS [ 27.038 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28:02 min
[INFO] Finished at: 2018-07-03T22:31:52+08:00
[INFO] ------------------------------------------------------------------------
[root@centos6-1 hadoop-2.7.6-src]# 

这里写图片描述

8.查看编译结果

在hadoop-2.7.6-src文件夹下面有一个hadoop-dist文件夹,进入里面的target文件夹,然后就可以看到编译成功64位的hadoop文件,解压后的在hadoop-2.7.6这个文件夹中,同时还生成了一个压缩包:hadoop-2.7.6-tar.gz 这个压缩包可以拷贝到别的机器上进行安装 。

[root@centos6-1 hadoop-2.7.6-src]# cd hadoop-dist/target/
[root@centos6-1 target]# ll
total 584076
drwxr-xr-x. 2 root root      4096 Jul  3 22:31 antrun
drwxr-xr-x. 3 root root      4096 Jul  3 22:31 classes
-rw-r--r--. 1 root root      1872 Jul  3 22:31 dist-layout-stitching.sh
-rw-r--r--. 1 root root       645 Jul  3 22:31 dist-tar-stitching.sh
drwxr-xr-x. 9 root root      4096 Jul  3 22:31 hadoop-2.7.6
-rw-r--r--. 1 root root 198929351 Jul  3 22:31 hadoop-2.7.6.tar.gz
-rw-r--r--. 1 root root     26521 Jul  3 22:31 hadoop-dist-2.7.6.jar
-rw-r--r--. 1 root root 399032461 Jul  3 22:31 hadoop-dist-2.7.6-javadoc.jar
-rw-r--r--. 1 root root     24049 Jul  3 22:31 hadoop-dist-2.7.6-sources.jar
-rw-r--r--. 1 root root     24049 Jul  3 22:31 hadoop-dist-2.7.6-test-sources.jar
drwxr-xr-x. 2 root root      4096 Jul  3 22:31 javadoc-bundle-options
drwxr-xr-x. 2 root root      4096 Jul  3 22:31 maven-archiver
drwxr-xr-x. 3 root root      4096 Jul  3 22:31 maven-shared-archive-resources
drwxr-xr-x. 3 root root      4096 Jul  3 22:31 test-classes
drwxr-xr-x. 2 root root      4096 Jul  3 22:31 test-dir
[root@centos6-1 target]# 

9.总结

嗯嗯,博主睡觉了,明天会更好。

猜你喜欢

转载自blog.csdn.net/weixin_37490221/article/details/80906578