Hadoop3.1.2 + Hbase2.2.0 set lzo compression algorithm:
EDITORIAL when configuring hbase use lzo algorithm, the Internet found a lot of articles, the articles are generally older, one low version, the second is generally used hadoop-gpl-compression, hadoop-gpl-compression is when an older dependencies, has been replaced by hadoop-lzo, and hope to meet hadoop hbase configuration lzo algorithm, can help
Installation lzo library
1. Download the latest lzo library Download: http://www.oberhumer.com/opensource/lzo/download/
2. Extract lzo library
tar -zxvf lzo-2.10.tar.gz
3. After the files are decompressed lzo directory, execute ./configure --enable-shared
cd lzo-2.10 ./configure --enable-shared -prefix=/usr/local/hadoop/lzo
4. Run make to compile, the compiled make install process to install
make && make install
If you do not lzo library, specify compression is lzo will complain when you create a table in hbase in:
ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:2314) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2156) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2048) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:651) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103) at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384) at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154) ... 7 more Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98) ... 10 more
5, the library file is installed by default to the / usr / local / lib, the / usr / local / lib copied to the / usr / lib, or establish a flexible connection in the / usr / lib
cd /usr/lib ln -s /usr/local/lib/* .
6. Installation lzop wget http://www.lzop.org/download/lzop-1.04.tar.gz
tar -zxvf lzop-1.04.tar.gz ./configure -enable-shared -prefix=/usr/local/hadoop/lzop make && make install
7. lzop to copy / usr / bin / or establish a flexible connection
ln -s /usr/local/hadoop/lzop/bin/lzop /usr/bin/lzop
Second, the installation hadoop-lzo
1. Download hadoop-lzo, Download: wget https://github.com/twitter/hadoop-lzo/archive/master.zip This is a zip archive, if you want to use git to download, you can use this link: HTTPS: //github.com/twitter/hadoop-lzo
2. Compile hadoop-lzo source code before compiling If you do not need to configure maven maven environment, decompress master.zip, is: hadoop-lzo-master, enter hadoop-lzo-master, a modified version of hadoop configuration in pom.xml , a maven compile
master.zip the unzip
cd hadoop-lzo-Master
vim pom.xml
modifications hadoop.current.version for their corresponding hadoop version, I have here is 3.1.2
<the Properties>
<Project .build .sourceEncoding> UTF- 8 </ Project . Build .sourceEncoding>
<Hadoop .current .version> 3.1.2 </ Hadoop .current .version>
<Hadoop .old .version> 1.0 .4 </ Hadoop .old .version>
</ Properties>
3. Run hadoop-lzo-master directory the following command to compile hadoop-lzo:
the CFLAGS = -m64 Export Export CXXFLAGS = -m64 Export C_INCLUDE_PATH = / usr / local / Hadoop / lzo / # corresponding to the include lzo installation directory export LIBRARY_PATH = / usr / local / hadoop / lzo / lib # directory corresponding to the mounted lzo
mvn clean package -Dmaven.test.skip=true
4. After the completion of packing, into the target / native / Linux-amd64-64, the libgplcompression * Native hadoop copied to the copy hadoop-lzo.xxx.jar common to each of the bag hadoop
cd target/native/Linux-amd64-64 tar -cBf - -C lib . | tar -xBvf - -C ~ cp ~/libgplcompression* $HADOOP_HOME/lib/native/ cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/
libgplcompression * file:
Which libgplcompression .so and libgplcompression .so .0 is linked files, point libgplcompression .so .0 .0 .0,
Generated above libgplcompression * and target / Hadoop-XXX-LZO -SNAPSHOT .jar directory corresponding to a synchronous machine (all clusters to $ HADOOP_HOME is / lib / Native /, $ HADOOP_HOME is / Share / Hadoop / Common / ).
Hadoop configuration environment variable
1. $ HADOOP_HOME / etc / hadoop / hadoop-env.sh configuration file:
export LD_LIBRARY_PATH=/usr/local/lib/lzo/lib
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native
2. $ HADOOP_HOME / etc / hadoop / core-site.xml add the following configuration:
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,
org.apache.hadoop.io.compress.BZip2Codec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
If no, in the hbase will complain compression lzo specified when creating the table:
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103) at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384) at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154) ... 7 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:128) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:114) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:353) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98) ... 10 more Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:124) ... 13 more
3. $ HADOOP_HOME / etc / hadoop / mapred-site.xml add the following configuration:
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>
The above modified configuration file to the cluster all synchronized on all machines, and restart Hadoop cluster, so that in use in Hadoop lzo.
Lzo in the configuration Hbase
1. Copy the hadoop-lzo-xxx.jar / hbase / lib in
cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HBASE_HOME/lib
2. Create a folder in the native hbase / lib, create Linux-amd64-64 in / hbase / lib / native -> / opt / hadoop / lib / native soft connection
ln -s /opt/hadoop/lib/native Linux-amd64-64
Figure:
3. Add follows $ HBASE_HOME / conf / hbase-env.sh of:
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
4. Add follows $ HBASE_HOME / conf / hbase-site.xml in:
<property>
<name>hbase.regionserver.codecs</name>
<value>lzo</value>
</property>
5. Start all normal hbase
Note: For an explanation of hadoop-gpl-compression:
hadoop-lzo-xxx, formerly known as hadoop-gpl-compression-xxx, before management on google code, the address: http: //code.google.com/p/hadoop-gpl-compression/ but due to protocol issues. later ported to github, which is now hadoop-lzo-xxx, github, link address: https://github.com/kevinweil/hadoop-lzo online presentation hadoop lzo compression mostly based on hadoop-gpl-. compression of introduction. the hadoop-gpl-compression was developed in 2009, with now no longer fully compatible hadoop version, some problems will occur. So some times also pit. I hope to give some friends a little help.
When using hadoop-gpl-compression-xxx.jar, hbase start will be reported the following error:
2019-09-03 11:36:22,771 INFO [main] lzo.GPLNativeCodeLoader: Loaded native gpl library 2019-09-03 11:36:22,866 WARN [main] lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc 2019-09-03 11:36:22,866 ERROR [main] lzo.LzoCodec: Failed to load/initialize native-lzo library 2019-09-03 11:36:23,169 WARN [main] util.CompressionTest: Can't instantiate codec: lzo org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:69) at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:834) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198) Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98) ... 14 more 2019-09-03 11:36:23,183 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer java.io.IOException: Compression codec lzo not supported, aborting RS construction at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198) 2019-09-03 11:36:23,184 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster. at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3187) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198) Caused by: java.io.IOException: Compression codec lzo not supported, aborting RS construction at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180) ... 5 more
When you delete hadoop-gpl-compression-xxx.jar, replaced after hadoop-lzo.xxx.jar, then start hbase, everything is normal:
2019-09-03 14:57:43,755 INFO [main] lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries 2019-09-03 14:57:43,758 INFO [main] lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5dbdddb8cfb544e58b4e0b9664b9d1b66657faf5] 2019-09-03 14:57:43,983 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 2019-09-03 14:57:44,088 INFO [main] compress.CodecPool: Got brand-new compressor [.lzo_deflate]