hbase reads the snapshot data compression problems encountered -lzo

1. Read the snapshot data hbase Times UnsatisfiedLinkError: no gplcompression in java.library.path wrong:

2019-09-04 17:36:07,441 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] GPLNativeCodeLoader:35 | Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
	at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:127)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:328)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-04 17:36:07,460 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] LzoCodec:70 | Cannot load native-lzo without native-hadoop

  On the java command execution dissecting -Djava.library.path = / opt / hadoop / lib / native, this error would not

java  -Djava.library.path=/opt/hadoop/lib/native -cp lib/*:conf/*:stream-0.0.1.jar com.hbasereader.HbaseReader

2. After solve the above exceptions, has emerged native-lzo library not available abnormality

 org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	... 21 more

	at com.suning.data.integration.engine.stream.core.exception.DiException.asDiException(DiException.java:51)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:131)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	... 12 more
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	... 21 more

 This error is reported out:

 

 

The reason after investigation finally found a week, really pit father, as quoted hadoop-gpl-compression-xxx.jar, and hadoop-gpl-compression is an older version,

 

 The hadoop-gpl-compression commented, using hadoop-lzo jar package, problem solving

Twitter currently in use to maintain the version on GitHub, download the source code clone git  https://github.com/twitter/hadoop-lzo.git  , use the maven compiler package, the hadoop-lzo-xxx.jar to join the project perfect to solve the native-lzo library not available issues.

As for how to use maven to compile hadoop-lzo, you can refer to my blog post:  Hadoop3.1.2 + Hbase2.2.0 set lzo compression algorithm 

 

Note: hadoop-gpl-compression outdated, please use the hadoop-lzo

     hadoop-gpl-compression outdated, please use the hadoop-lzo

     hadoop-gpl-compression outdated, please use the hadoop-lzo

   The important thing to say three times! ! !

Guess you like

Origin www.cnblogs.com/qixing/p/11514823.html