Article Directory
Introduction
To customize the HBase Comparator, you need to use protoc, which can be downloaded from the following link:
protoc new version download
protoc2.5 version download
You can download the complete project at the following link:
HBase custom Comparator instance
proto file format
enum Sex{
MALE = 1;
FEMALE = 2;
}
message Bean {
required int32 id = 1;
required string name = 2;
optional Sex sex = 3[defalut = MALE];
}
The enum keyword defines an enumerated type, and the message defines a class.
- required: The field must be included once
- optional: include this field zero or one time
- repeated: The field can be repeated any number of times
Field definition description:
optional Sex sex = 3 [defalut = MALE];
字段规则 字段类型 字段名称 唯一标识符 可选的选项：默认值
Data default value:
- int: 0
- string: empty character
- bool: false
- enum: the first item of the enumeration
Define a proto file
syntax = "proto2";
option java_package = "org.curitis.comparator";
option java_outer_classname = "HBaseFilterNumberComparatorProto";
option java_generic_services = true;
option optimize_for = SPEED;
message HBaseFilterNumberComparator {
required bytes value = 1;
required string fieldType = 2;
}
The first line is to specify the version of protoc, which can be proto2 or proto3, according to my needs, because I use HBase2.0.0 version and rely on protoc2.5, so fill in proto2 here.
To see which version of protoc HBase depends on, you can execute the following command in the corresponding project:
mvn dependency:tree >> tree.txt
Then search for protoc in the corresponding output file tree.txt.
java_package specifies the name of the generated package, java_outer_classname specifies the name of the output class, if not specified, the name of the proto scheme file is used by default.
message defines the entity class to be serialized.
protoc command
protoc.exe ./HBaseFilterNumberComparatorProto.proto --java_out=./
Compilation is very simple, just specify the proto scheme file and output directory.
Custom comparator
The HBaseFilterNumberComparatorProto class generated by protoc compilation is relatively long, we put it at the end, here we directly look at the custom comparator.
HBase custom comparators generally inherit the ByteArrayComparable class. For specific implementation, we can refer to the LongComparator and BigDecimalComparator that have been implemented in the hBase-common package.
import com.google.protobuf.ByteString;
import com.google.protobuf.InvalidProtocolBufferException;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.filter.ByteArrayComparable;
import org.apache.hadoop.hbase.util.Bytes;
/**
* 自定义比较器
*/
public class FilterNumberComparator extends ByteArrayComparable {
private String fieldType;
private byte[] data;
/**
* @param value
* @param fieldType
*/
public FilterNumberComparator(byte[] value, String fieldType) {
super(value);
this.fieldType = fieldType.toLowerCase();
this.data = value;
}
@Override
public byte[] toByteArray() {
HBaseFilterNumberComparatorProto.HBaseFilterNumberComparator.Builder builder =
HBaseFilterNumberComparatorProto.HBaseFilterNumberComparator.newBuilder();
builder.setValue(ByteString.copyFrom(this.data));
builder.setFieldType(this.fieldType);
return builder.build().toByteArray();
}
// 定义该方法，用于对象反序列化操作
public static FilterNumberComparator parseFrom(final byte[] bytes) throws DeserializationException {
HBaseFilterNumberComparatorProto.HBaseFilterNumberComparator proto;
try {
proto = HBaseFilterNumberComparatorProto.HBaseFilterNumberComparator.parseFrom(bytes);
} catch (InvalidProtocolBufferException e) {
throw new DeserializationException(e);
}
return new FilterNumberComparator(proto.getValue().toByteArray(), proto.getFieldType());
}
@Override
public int compareTo(byte[] bytes, int offset, int length) {
if (fieldType.equalsIgnoreCase("int") || fieldType.equalsIgnoreCase("integer")) {
int that = Bytes.toInt(bytes, offset, length);
return Integer.compare(Bytes.toInt(data), that);
} else if (fieldType.equalsIgnoreCase("float")) {
float that = Bytes.toFloat(bytes, offset);
return Float.compare(Bytes.toFloat(data), that);
} else if (fieldType.equalsIgnoreCase("double")) {
double that = Bytes.toDouble(bytes, offset);
return Double.compare(Bytes.toDouble(data),that);
}
return 1;
}
}
The specific business logic is in the compareTo method. Above we have implemented the comparison of integer, float, and double types.
How to use this Comparator specifically?
First, you need to introduce the jar package of the dependent FilterNumberComparator
@Test
public void singleColumnValueFilter() throws IOException {
Scan scan = new Scan();
FilterNumberComparator comparator = new FilterNumberComparator(Bytes.toBytes(1100.0),"double");
byte[] family = "consume".getBytes();
byte[] column = "total".getBytes();
SingleColumnValueFilter singleColumnValueFilter = new SingleColumnValueFilter(
family,
column,
CompareOperator.GREATER,
comparator);
singleColumnValueFilter.setFilterIfMissing(true);
scan.setFilter(singleColumnValueFilter);
ResultScanner resultScanner = table.getScanner(scan);
printResultScanner(resultScanner);
}
Above we are filtering the rows where the total value of the column in consume is greater than 1100. We must pay attention to 1100.0 instead of 1100, because it is of double type. If it is 1100, Bytes.toBytes will recognize it as an integer. .
Running problem
The custom FilterNumberComparator needs to be packaged separately into a jar package and placed in the lib directory of HBase, such as: /usr/local/hbase-2.0.0/lib. If it is a cluster, pay attention to each node.
Then you need to restart HBase:
sudo ./stop-hbase.sh
sudo ./start-hbase.sh
HBaseFilterNumberComparatorProto
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: HBaseFilterNumberComparatorProto.proto
package org.curitis.comparator;
public final class HBaseFilterNumberComparatorProto {
private HBaseFilterNumberComparatorProto() {}
public static void registerAllExtensions(
com.google.protobuf.ExtensionRegistry registry) {
}
public interface HBaseFilterNumberComparatorOrBuilder
extends com.google.protobuf.MessageOrBuilder {
// required bytes value = 1;
/**
* <code>required bytes value = 1;</code>
*/
boolean hasValue();
/**
* <code>required bytes value = 1;</code>
*/
com.google.protobuf.ByteString getValue();
// required string fieldType = 2;
/**
* <code>required string fieldType = 2;</code>
*/
boolean hasFieldType();
/**
* <code>required string fieldType = 2;</code>
*/
String getFieldType();
/**
* <code>required string fieldType = 2;</code>
*/
com.google.protobuf.ByteString
getFieldTypeBytes();
}
/**
* Protobuf type {@code HBaseFilterNumberComparator}
*/
public static final class HBaseFilterNumberComparator extends
com.google.protobuf.GeneratedMessage
implements HBaseFilterNumberComparatorOrBuilder {
// Use HBaseFilterNumberComparator.newBuilder() to construct.
private HBaseFilterNumberComparator(com.google.protobuf.GeneratedMessage.Builder<?> builder) {
super(builder);
this.unknownFields = builder.getUnknownFields();
}
private HBaseFilterNumberComparator(boolean noInit) { this.unknownFields = com.google.protobuf.UnknownFieldSet.getDefaultInstance(); }
private static final HBaseFilterNumberComparator defaultInstance;
public static HBaseFilterNumberComparator getDefaultInstance() {
return defaultInstance;
}
public HBaseFilterNumberComparator getDefaultInstanceForType() {
return defaultInstance;
}
private final com.google.protobuf.UnknownFieldSet unknownFields;
@Override
public final com.google.protobuf.UnknownFieldSet
getUnknownFields() {
return this.unknownFields;
}
private HBaseFilterNumberComparator(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
initFields();
int mutable_bitField0_ = 0;
com.google.protobuf.UnknownFieldSet.Builder unknownFields =
com.google.protobuf.UnknownFieldSet.newBuilder();
try {
boolean done = false;
while (!done) {
int tag = input.readTag();
switch (tag) {
case 0:
done = true;
break;
default: {
if (!parseUnknownField(input, unknownFields,
extensionRegistry, tag)) {
done = true;
}
break;
}
case 10: {
bitField0_ |= 0x00000001;
value_ = input.readBytes();
break;
}
case 18: {
bitField0_ |= 0x00000002;
fieldType_ = input.readBytes();
break;
}
}
}
} catch (com.google.protobuf.InvalidProtocolBufferException e) {
throw e.setUnfinishedMessage(this);
} catch (java.io.IOException e) {
throw new com.google.protobuf.InvalidProtocolBufferException(
e.getMessage()).setUnfinishedMessage(this);
} finally {
this.unknownFields = unknownFields.build();
makeExtensionsImmutable();
}
}
public static final com.google.protobuf.Descriptors.Descriptor
getDescriptor() {
return HBaseFilterNumberComparatorProto.internal_static_HBaseFilterNumberComparator_descriptor;
}
protected FieldAccessorTable
internalGetFieldAccessorTable() {
return HBaseFilterNumberComparatorProto.internal_static_HBaseFilterNumberComparator_fieldAccessorTable
.ensureFieldAccessorsInitialized(
HBaseFilterNumberComparator.class, Builder.class);
}
public static com.google.protobuf.Parser<HBaseFilterNumberComparator> PARSER =
new com.google.protobuf.AbstractParser<HBaseFilterNumberComparator>() {
public HBaseFilterNumberComparator parsePartialFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return new HBaseFilterNumberComparator(input, extensionRegistry);
}
};
@Override
public com.google.protobuf.Parser<HBaseFilterNumberComparator> getParserForType() {
return PARSER;
}
private int bitField0_;
// required bytes value = 1;
public static final int VALUE_FIELD_NUMBER = 1;
private com.google.protobuf.ByteString value_;
/**
* <code>required bytes value = 1;</code>
*/
public boolean hasValue() {
return ((bitField0_ & 0x00000001) == 0x00000001);
}
/**
* <code>required bytes value = 1;</code>
*/
public com.google.protobuf.ByteString getValue() {
return value_;
}
// required string fieldType = 2;
public static final int FIELDTYPE_FIELD_NUMBER = 2;
private Object fieldType_;
/**
* <code>required string fieldType = 2;</code>
*/
public boolean hasFieldType() {
return ((bitField0_ & 0x00000002) == 0x00000002);
}
/**
* <code>required string fieldType = 2;</code>
*/
public String getFieldType() {
Object ref = fieldType_;
if (ref instanceof String) {
return (String) ref;
} else {
com.google.protobuf.ByteString bs =
(com.google.protobuf.ByteString) ref;
String s = bs.toStringUtf8();
if (bs.isValidUtf8()) {
fieldType_ = s;
}
return s;
}
}
/**
* <code>required string fieldType = 2;</code>
*/
public com.google.protobuf.ByteString
getFieldTypeBytes() {
Object ref = fieldType_;
if (ref instanceof String) {
com.google.protobuf.ByteString b =
com.google.protobuf.ByteString.copyFromUtf8(
(String) ref);
fieldType_ = b;
return b;
} else {
return (com.google.protobuf.ByteString) ref;
}
}
private void initFields() {
value_ = com.google.protobuf.ByteString.EMPTY;
fieldType_ = "";
}
private byte memoizedIsInitialized = -1;
public final boolean isInitialized() {
byte isInitialized = memoizedIsInitialized;
if (isInitialized != -1) return isInitialized == 1;
if (!hasValue()) {
memoizedIsInitialized = 0;
return false;
}
if (!hasFieldType()) {
memoizedIsInitialized = 0;
return false;
}
memoizedIsInitialized = 1;
return true;
}
public void writeTo(com.google.protobuf.CodedOutputStream output)
throws java.io.IOException {
getSerializedSize();
if (((bitField0_ & 0x00000001) == 0x00000001)) {
output.writeBytes(1, value_);
}
if (((bitField0_ & 0x00000002) == 0x00000002)) {
output.writeBytes(2, getFieldTypeBytes());
}
getUnknownFields().writeTo(output);
}
private int memoizedSerializedSize = -1;
public int getSerializedSize() {
int size = memoizedSerializedSize;
if (size != -1) return size;
size = 0;
if (((bitField0_ & 0x00000001) == 0x00000001)) {
size += com.google.protobuf.CodedOutputStream
.computeBytesSize(1, value_);
}
if (((bitField0_ & 0x00000002) == 0x00000002)) {
size += com.google.protobuf.CodedOutputStream
.computeBytesSize(2, getFieldTypeBytes());
}
size += getUnknownFields().getSerializedSize();
memoizedSerializedSize = size;
return size;
}
private static final long serialVersionUID = 0L;
@Override
protected Object writeReplace()
throws java.io.ObjectStreamException {
return super.writeReplace();
}
public static HBaseFilterNumberComparator parseFrom(
com.google.protobuf.ByteString data)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data);
}
public static HBaseFilterNumberComparator parseFrom(
com.google.protobuf.ByteString data,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data, extensionRegistry);
}
public static HBaseFilterNumberComparator parseFrom(byte[] data)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data);
}
public static HBaseFilterNumberComparator parseFrom(
byte[] data,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data, extensionRegistry);
}
public static HBaseFilterNumberComparator parseFrom(java.io.InputStream input)
throws java.io.IOException {
return PARSER.parseFrom(input);
}
public static HBaseFilterNumberComparator parseFrom(
java.io.InputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return PARSER.parseFrom(input, extensionRegistry);
}
public static HBaseFilterNumberComparator parseDelimitedFrom(java.io.InputStream input)
throws java.io.IOException {
return PARSER.parseDelimitedFrom(input);
}
public static HBaseFilterNumberComparator parseDelimitedFrom(
java.io.InputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return PARSER.parseDelimitedFrom(input, extensionRegistry);
}
public static HBaseFilterNumberComparator parseFrom(
com.google.protobuf.CodedInputStream input)
throws java.io.IOException {
return PARSER.parseFrom(input);
}
public static HBaseFilterNumberComparator parseFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return PARSER.parseFrom(input, extensionRegistry);
}
public static Builder newBuilder() { return Builder.create(); }
public Builder newBuilderForType() { return newBuilder(); }
public static Builder newBuilder(HBaseFilterNumberComparator prototype) {
return newBuilder().mergeFrom(prototype);
}
public Builder toBuilder() { return newBuilder(this); }
@Override
protected Builder newBuilderForType(
BuilderParent parent) {
Builder builder = new Builder(parent);
return builder;
}
/**
* Protobuf type {@code HBaseFilterNumberComparator}
*/
public static final class Builder extends
com.google.protobuf.GeneratedMessage.Builder<Builder>
implements HBaseFilterNumberComparatorOrBuilder {
public static final com.google.protobuf.Descriptors.Descriptor
getDescriptor() {
return HBaseFilterNumberComparatorProto.internal_static_HBaseFilterNumberComparator_descriptor;
}
protected FieldAccessorTable
internalGetFieldAccessorTable() {
return HBaseFilterNumberComparatorProto.internal_static_HBaseFilterNumberComparator_fieldAccessorTable
.ensureFieldAccessorsInitialized(
HBaseFilterNumberComparator.class, Builder.class);
}
// Construct using org.curitis.comparator.HBaseFilterNumberComparatorProto.HBaseFilterNumberComparator.newBuilder()
private Builder() {
maybeForceBuilderInitialization();
}
private Builder(
BuilderParent parent) {
super(parent);
maybeForceBuilderInitialization();
}
private void maybeForceBuilderInitialization() {
if (com.google.protobuf.GeneratedMessage.alwaysUseFieldBuilders) {
}
}
private static Builder create() {
return new Builder();
}
public Builder clear() {
super.clear();
value_ = com.google.protobuf.ByteString.EMPTY;
bitField0_ = (bitField0_ & ~0x00000001);
fieldType_ = "";
bitField0_ = (bitField0_ & ~0x00000002);
return this;
}
public Builder clone() {
return create().mergeFrom(buildPartial());
}
public com.google.protobuf.Descriptors.Descriptor
getDescriptorForType() {
return HBaseFilterNumberComparatorProto.internal_static_HBaseFilterNumberComparator_descriptor;
}
public HBaseFilterNumberComparator getDefaultInstanceForType() {
return HBaseFilterNumberComparator.getDefaultInstance();
}
public HBaseFilterNumberComparator build() {
HBaseFilterNumberComparator result = buildPartial();
if (!result.isInitialized()) {
throw newUninitializedMessageException(result);
}
return result;
}
public HBaseFilterNumberComparator buildPartial() {
HBaseFilterNumberComparator result = new HBaseFilterNumberComparator(this);
int from_bitField0_ = bitField0_;
int to_bitField0_ = 0;
if (((from_bitField0_ & 0x00000001) == 0x00000001)) {
to_bitField0_ |= 0x00000001;
}
result.value_ = value_;
if (((from_bitField0_ & 0x00000002) == 0x00000002)) {
to_bitField0_ |= 0x00000002;
}
result.fieldType_ = fieldType_;
result.bitField0_ = to_bitField0_;
onBuilt();
return result;
}
public Builder mergeFrom(com.google.protobuf.Message other) {
if (other instanceof HBaseFilterNumberComparator) {
return mergeFrom((HBaseFilterNumberComparator)other);
} else {
super.mergeFrom(other);
return this;
}
}
public Builder mergeFrom(HBaseFilterNumberComparator other) {
if (other == HBaseFilterNumberComparator.getDefaultInstance()) return this;
if (other.hasValue()) {
setValue(other.getValue());
}
if (other.hasFieldType()) {
bitField0_ |= 0x00000002;
fieldType_ = other.fieldType_;
onChanged();
}
this.mergeUnknownFields(other.getUnknownFields());
return this;
}
public final boolean isInitialized() {
if (!hasValue()) {
return false;
}
if (!hasFieldType()) {
return false;
}
return true;
}
public Builder mergeFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
HBaseFilterNumberComparator parsedMessage = null;
try {
parsedMessage = PARSER.parsePartialFrom(input, extensionRegistry);
} catch (com.google.protobuf.InvalidProtocolBufferException e) {
parsedMessage = (HBaseFilterNumberComparator) e.getUnfinishedMessage();
throw e;
} finally {
if (parsedMessage != null) {
mergeFrom(parsedMessage);
}
}
return this;
}
private int bitField0_;
// required bytes value = 1;
private com.google.protobuf.ByteString value_ = com.google.protobuf.ByteString.EMPTY;
/**
* <code>required bytes value = 1;</code>
*/
public boolean hasValue() {
return ((bitField0_ & 0x00000001) == 0x00000001);
}
/**
* <code>required bytes value = 1;</code>
*/
public com.google.protobuf.ByteString getValue() {
return value_;
}
/**
* <code>required bytes value = 1;</code>
*/
public Builder setValue(com.google.protobuf.ByteString value) {
if (value == null) {
throw new NullPointerException();
}
bitField0_ |= 0x00000001;
value_ = value;
onChanged();
return this;
}
/**
* <code>required bytes value = 1;</code>
*/
public Builder clearValue() {
bitField0_ = (bitField0_ & ~0x00000001);
value_ = getDefaultInstance().getValue();
onChanged();
return this;
}
// required string fieldType = 2;
private Object fieldType_ = "";
/**
* <code>required string fieldType = 2;</code>
*/
public boolean hasFieldType() {
return ((bitField0_ & 0x00000002) == 0x00000002);
}
/**
* <code>required string fieldType = 2;</code>
*/
public String getFieldType() {
Object ref = fieldType_;
if (!(ref instanceof String)) {
String s = ((com.google.protobuf.ByteString) ref)
.toStringUtf8();
fieldType_ = s;
return s;
} else {
return (String) ref;
}
}
/**
* <code>required string fieldType = 2;</code>
*/
public com.google.protobuf.ByteString
getFieldTypeBytes() {
Object ref = fieldType_;
if (ref instanceof String) {
com.google.protobuf.ByteString b =
com.google.protobuf.ByteString.copyFromUtf8(
(String) ref);
fieldType_ = b;
return b;
} else {
return (com.google.protobuf.ByteString) ref;
}
}
/**
* <code>required string fieldType = 2;</code>
*/
public Builder setFieldType(
String value) {
if (value == null) {
throw new NullPointerException();
}
bitField0_ |= 0x00000002;
fieldType_ = value;
onChanged();
return this;
}
/**
* <code>required string fieldType = 2;</code>
*/
public Builder clearFieldType() {
bitField0_ = (bitField0_ & ~0x00000002);
fieldType_ = getDefaultInstance().getFieldType();
onChanged();
return this;
}
/**
* <code>required string fieldType = 2;</code>
*/
public Builder setFieldTypeBytes(
com.google.protobuf.ByteString value) {
if (value == null) {
throw new NullPointerException();
}
bitField0_ |= 0x00000002;
fieldType_ = value;
onChanged();
return this;
}
// @@protoc_insertion_point(builder_scope:HBaseFilterNumberComparator)
}
static {
defaultInstance = new HBaseFilterNumberComparator(true);
defaultInstance.initFields();
}
// @@protoc_insertion_point(class_scope:HBaseFilterNumberComparator)
}
private static com.google.protobuf.Descriptors.Descriptor
internal_static_HBaseFilterNumberComparator_descriptor;
private static
com.google.protobuf.GeneratedMessage.FieldAccessorTable
internal_static_HBaseFilterNumberComparator_fieldAccessorTable;
public static com.google.protobuf.Descriptors.FileDescriptor
getDescriptor() {
return descriptor;
}
private static com.google.protobuf.Descriptors.FileDescriptor
descriptor;
static {
String[] descriptorData = {
"\n&HBaseFilterNumberComparatorProto.proto" +
"\"?\n\033HBaseFilterNumberComparator\022\r\n\005value" +
"\030\001 \002(\014\022\021\n\tfieldType\030\002 \002(\tB\035\n\026org.curitis" +
".comparatorH\001\210\001\001"
};
com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner assigner =
new com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner() {
public com.google.protobuf.ExtensionRegistry assignDescriptors(
com.google.protobuf.Descriptors.FileDescriptor root) {
descriptor = root;
internal_static_HBaseFilterNumberComparator_descriptor =
getDescriptor().getMessageTypes().get(0);
internal_static_HBaseFilterNumberComparator_fieldAccessorTable = new
com.google.protobuf.GeneratedMessage.FieldAccessorTable(
internal_static_HBaseFilterNumberComparator_descriptor,
new String[] { "Value", "FieldType", });
return null;
}
};
com.google.protobuf.Descriptors.FileDescriptor
.internalBuildGeneratedFileFrom(descriptorData,
new com.google.protobuf.Descriptors.FileDescriptor[] {
}, assigner);
}
// @@protoc_insertion_point(outer_class_scope)
}