Small knowledge point: Compilation based on Hadoop-2.6.5 source code under Linux

1. Relevant installation packages

• [hadoop-2.6.5-src.tar.gz] hadoop source package
  • http://archive.apache.org/dist/hadoop/common/hadoop-2.6.5/hadoop-2.6.5-src.tar.gz
• [jdk-8uxx-linux-x64.tar.gz] jdk1.8
  • https://www.oracle.com/java/technologies/downloads/#java8
• [apache-ant-1.9.9-bin.tar.gz] build tool, for packaging
  • http://archive.apache.org/dist/ant/binaries/apache-ant-1.9.9-bin.tar.gz
• [apache-maven-3.3.9-bin.tar.gz] maven environment
  • http://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
• [protobuf-2.5.0.tar.gz] serialization framework
  • https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

2. Environmental preparation

The configuration environment variable template mentioned below is as follows

#xxx
export XXX_HOME=/xxx/soft/xxx
export PATH=$XXX_HOME/bin:$PATH

unzip command

# tar.gz
tar -zxvf xxx.tar.gz

• Download the source package
  • Download the corresponding package and unzip it
• Install jdk1.8
  • Download the corresponding package, unzip the tar.gz package, and configure environment variables
  • java -version view
• Install the ant environment
  • Download the corresponding package, unzip the tar.gz package, and configure environment variables
  • ant -version view
• Install maven environment
  • Download the corresponding package, unzip the tar.gz package, and configure environment variables
  • mvn -version view
  • Modify maven source (conf/settings.xml)

<mirrors>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|-->
<!-- 阿里云仓库 -->
	<mirror>
		<id>alimaven</id>
		<mirrorOf>central</mirrorOf>
		<name>aliyun maven</name>
		<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
	</mirror>
</mirrors>

• Install the protobuf directory
  • Download the corresponding package and unzip the tar.gz package
  • compile protobuf
    • ./configure
    • make
    • make check
    • make install
    • ldconfig
  • Configure environment variables
    • protoc -version view

#protobuf
export LD_LIBRARY_PATH=/xxx/protobuf
export PATH=$LD_LIBRARY_PATH:$PATH

• glibc-headers、g++、make、cmake、openssl、ncurses-devel
  • yum install glibc-headers
  • yum instal gcc-c++
  • yum install make
  • yum install cmake
  • yum install openssl-devel
  • yum install ncurses-devel

3. Compile Hadoop

• Unzip and enter the folder
• compile command
  • mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
    • If the compilation error is reported and executed repeatedly, it may be a network problem. If it fails, you can continue to execute, and then continue to execute the original command after the last download
    • Do not follow the last download to add clean (mvn clean package...)
  • After the compilation is successful, there will be the file hadoop-2.6.5-src/hadoop-dist/target

Four, error resolution

• Hadoop Annotations package com.sun.javadoc not found problem
  • Check whether the local maven already has javadoc
    • ~/.m2/repository/org/apache/maven/plugins/maven-javadoc-plugin already has 2.8.1
  • Modify the corresponding version number in the pom.xml file under the hadoop source package

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-javadoc-plugin</artifactId>
	<version>2.8.1</version>     //修改这行，没有就添加
	<inherited>false</inherited>
	<executions>
		<execution>
			<!-- build aggregate javadoc in parent only -->
			<id>default-cli</id>
			<goals>
			<goal>aggregate</goal>
			</goals>
			<configuration>
			<overview>hadoop-common-project/hadoop-common/src/main/java/overview.html</overview>
			</configuration>
		</execution>
	</executions>
</plugin>

• If you still can't find the com.sum.javadoc package, check the pom file in the hadoop-common-project/haddop-annotations directory
  • The default is 1.7, modify jdk to the version installed on this machine