How to create custom serializer for kafka-mirror-maker?

Nikita :

I want to replicate data from kafka cluster A to kafka cluster B using kafka-mirror-maker. Also, i need some special serialization logic for it. As i know, to create custom serializator i have to implement Serializer<> interface. So, the problem is that kafka-mirror-maker does not use custom serializer which i specify in producer.properties:

bootstrap.servers=kafkaB:9092
client.id=producerB
value.serializer=mypackage.CustomSerializer

Instead, i see that default ByteArraySerializer is used.

To build my serializer i use gradle:

dependencies {
    constraints {
        compileOnly(group: 'org.apache.kafka', name: 'kafka-clients', version: '2.1.0')
    }

    compileOnly(group: 'org.apache.kafka', name: 'kafka-clients')
}

and then just put compiled jar into directory with other jars (confluent-kafka-docker): /usr/share/java/kafka/customSerializer.jar

Logic of my serializer currently is trivial:

public class CustomSerializer implements Serializer<Object> {
    public CustomSerializer() {
    }

    @Override
    public void configure(Map<String, ?> configs, boolean isKey) {

    }

    @Override
    public byte[] serialize(String topic, Object data) {
        return (new String((byte[]) data)).toUpperCase().getBytes(StandardCharsets.UTF_8);
    }

    @Override
    public void close() {
    }
}

In which place should i put my jar to be able to use it?

Probably, i found a reason (https://github.com/apache/kafka/blob/252e7e958a2fabfa680d23a00e7f0ce7c74f6ad8/core/src/main/scala/kafka/tools/MirrorMaker.scala#L233):

  // Always set producer key and value serializer to ByteArraySerializer.
      producerProps.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, classOf[ByteArraySerializer].getName)
      producerProps.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, classOf[ByteArraySerializer].getName)
      producer = new MirrorMakerProducer(sync, producerProps)

As i see, there is no option to use custom serializer for key/value

cricket_007 :

You still have to deserialize bytes, so I'm not sure I understand the purpose of overriding only the serializer

If you want to manipulate the message, look at the MessageHandler interface, and then the --handler argument. In there, you would need to wrap both a deserailizer and serializer

Example here of renaming the topic - https://github.com/gwenshap/kafka-examples/tree/master/MirrorMakerHandler

How to create custom serializer for kafka-mirror-maker?

Guess you like