Hive often uses snappy compression format, but Hadoop itself does not support snappy compression (you can view it through the hadoop checknative command). If you want Hadoop to support snappy compression, you need to compile the source code of the corresponding version of Hadoop. Copy the native file to the directory of hadoop and restart hadoop. I think it’s pretty good, but the big pit is in the compilation stage. It took two days to get it right. Here are some of my conclusions:
First, let’s talk about some pits:
the installation of the compilation environment, otherwise it cannot be compiled
yum install -y gcc gcc-c++ make cmake
protobuf must be version 2.5.0, snappy must also compile my snappy is the 1.1.3 version downloaded from the official website
protobuf, snappy must be compiled and installed, the order of instructions is as follows:
./configure
make
make install
compiling hadoop requires some dependencies Software:
protocolbuf and ant are
not installed. These will report an error when compiling: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.4.0:protoc (compile-protoc) on project hadoop-common: org. apache.maven.plugin.MojoExecutionException:'protoc --version' did not return a version -> [Help 1]
Ant installation is very simple. After downloading the tar package, you can configure the environment variables and
finally compile the hadoop source code. The compilation instructions use this. There are many online versions. This is correct:
cd hadoop-2.4.0-src/
mvn clean package -Pdist ,native -DskipTests -Dtar -Drequire.snappy -Dbundle.snappy -Dsnappy.lib=/usr/local/lib -Dmaven.javadoc.skip=true During the
compilation process, if the network is not good, there will be a download suspended animation (stuck there and not moving ), or there may be a file download timeout, then ctrl+c will do it again! The entire compilation process is relatively slow, about 1 hour (maybe faster if the network is good).
This article is well written, I just combined the problem solved by this article, you can refer to it if you need it