1. Read the snapshot data hbase Times UnsatisfiedLinkError: no gplcompression in java.library.path wrong:
2019-09-04 17:36:07,441 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] GPLNativeCodeLoader:35 | Could not load native gpl library java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:127) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:328) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-09-04 17:36:07,460 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] LzoCodec:70 | Cannot load native-lzo without native-hadoop
On the java command execution dissecting -Djava.library.path = / opt / hadoop / lib / native, this error would not
java -Djava.library.path=/opt/hadoop/lib/native -cp lib/*:conf/*:stream-0.0.1.jar com.hbasereader.HbaseReader
2. After solve the above exceptions, has emerged native-lzo library not available abnormality
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) ... 21 more at com.suning.data.integration.engine.stream.core.exception.DiException.asDiException(DiException.java:51) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:131) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) ... 12 more Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) ... 21 more
This error is reported out:
The reason after investigation finally found a week, really pit father, as quoted hadoop-gpl-compression-xxx.jar, and hadoop-gpl-compression is an older version,
The hadoop-gpl-compression commented, using hadoop-lzo jar package, problem solving
Twitter currently in use to maintain the version on GitHub, download the source code clone git https://github.com/twitter/hadoop-lzo.git , use the maven compiler package, the hadoop-lzo-xxx.jar to join the project perfect to solve the native-lzo library not available issues.
As for how to use maven to compile hadoop-lzo, you can refer to my blog post: Hadoop3.1.2 + Hbase2.2.0 set lzo compression algorithm
Note: hadoop-gpl-compression outdated, please use the hadoop-lzo
hadoop-gpl-compression outdated, please use the hadoop-lzo
hadoop-gpl-compression outdated, please use the hadoop-lzo
The important thing to say three times! ! !