- Install LZO with HomeBrew, LZO is a wonderful compression scheme to use with Hadoop. HomeBrew is prerequisite, visit https://github.com/mxcl/homebrew to see how to install HomeBrew.
brew install lzo
- Clone the hadoop-lzo source code to local:
git clone https://github.com/twitter/hadoop-lzo.git ~/code/hadoop-lzo && cd ~/code/hadoop-lzo
- Build hadoop-lzo to generate hadoop-lzo-lib as well as hadoop lzo native library, find the built artifacts in the target directory.
C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.06/include \ LIBRARY_PATH=/usr/local/Cellar/lzo/2.06/lib \ mvn clean test packageWaiting for build complete. If something wrong, refer to https://github.com/twitter/hadoop-lzo for troubleshoting.
- Copy the built artifacts into hadoop lib directory or native library directory respectively.
- Once the libs are built and installed, you may want to add them to the class paths and library paths. That is, in hadoop-env.sh, set
export HADOOP_CLASSPATH=$HADOOP_HOME/lib/hadoop-lzo-xxxx.jar export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native-libs:/path/to/standard-hadoop-native-libs
- Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; (refer to https://issues.apache.org/jira/browse/HADOOP-6453) . Comment out the line
JAVA_LIBRARY_PATH=''
See https://github.com/twitter/hadoop-lzo for further instruction if you have any question or leave a comment.