CentOS-based Hadoop source code compilation supports Snappy compression

initial preparation work

jar package preparation (hadoop source code, JDK7 , maven , ant , protobuf)

1. hadoop-2.7.2-src.tar.gz
   
2. jdk-8u171-linux-x64.tar.gz
   
3. snappy-1.1.3.tar.gz
   
4. apache-maven-3.0.5-bin.tar.gz
   
5. protobuf-2.5.0.tar.gz
   

Jar package installation

Please execute the program as root user

1. Unzip the JDK , configure the environment variables JAVA_HOME and PATH , and verify the java -version ( you need to verify whether the configuration is successful as follows )
   [root@hadoop101 software] # tar -zxf jdk-7u79-linux-x64.gz-C /opt/module/
   [ root@hadoop101 software]# vi /etc/profile
   
   

   #JAVA_HOME export JAVA_HOME=/opt/module/jdk1.8.0_161 export PATH=$PATH:$JAVA_HOME/bin

   [root@hadoop101 software]#source /etc/profile
   Verification command: java -version
   
2. Maven unzips, configures  MAVEN_HOME and PATH .
   [root@hadoop101 software]# tar -zxvfapache-maven-3.0.5-bin.tar.gz -C /opt/module/
   [root@hadoop101 apache-maven-3.0.5]# vi/etc/profile
   
   

   #MAVEN_HOME exportMAVEN_HOME=/opt/module/apache-maven-3.0.5 export PATH=$PATH:$MAVEN_HOME/bin

   [root@hadoop101 software]#source /etc/profile
   Verification command: mvn -version

Compile the source code

1. Prepare the build environment
   

   [root@hadoop101software]# yuminstall svn
   [root@hadoop101software]# yuminstall autoconf automake libtool cmake
   [root@hadoop101software]# yuminstall ncurses-devel
   [root@hadoop101software]# yuminstall openssl-devel
   [root@hadoop101software]# yum install gcc*

2. Compile and install snappy
   

   [root@hadoop101software]# tar -zxvfsnappy-1.1.3.tar.gz -C /opt/module/
   [root@hadoop101 module]# cd snappy-1.1.3/
   [root@hadoop101 snappy-1.1.3]# ./configure
   [root@hadoop101 snappy-1.1.3]# make
   [root@hadoop101 snappy-1.1.3]# make install
   # 查看snappy库文件
   [root@hadoop101 snappy-1.1.3]# ls -lh /usr/local/lib |grep snappy

Compile and install protobuf

[root@hadoop101 software]# tar -zxvfprotobuf-2.5.0.tar.gz -C /opt/module/
[root@hadoop101 module]# cd protobuf-2.5.0/
[root@hadoop101 protobuf-2.5.0]# ./configure 
[root@hadoop101 protobuf-2.5.0]# make  [root@hadoop101 protobuf-2.5.0]
# make install #Check
the protobuf version to test if the installation is successful [root@hadoop101 protobuf-2.5.0]# protoc - -version

Compile hadoop native

[root@hadoop101 software]# tar -zxvfhadoop-2.7.2-src.tar.gz
[root@hadoop101 software]# cd hadoop-2.7.2-src/
[root@hadoop101 software]# mvn clean package -DskipTests -Pdist , native -Dtar -Dsnappy.lib=/usr/local/lib 
                                                -Dbundle.snappy
After successful execution, /opt/software/hadoop-2.7.2-src/hadoop-dist/target/hadoop -2.7.2.tar . gz is the newly generated binary installation package that supports snappy compression.