Creating a Custom Processor in Nifi and send to PublishKafka

DQMan :

Can someone provide me with a simple example for setting up a flowfile in a custom Nifi processor so the payload can be sent out thru the PublishKafka processor?

I have a legacy messaging protocol that I wrote a custom processor for. Pretty simple structure, just a MessageID (String) and the MessageBody (byte[]). My custom processor handles the input with the messages being received fine. I'm now attempting to put this data into a flowfile so it can be sent on to the publishKafka processor but I've had trouble finding any resources online with how to do this. Here's my current code snippet of the relevant portion:

    try {
         this.getLogger().info("[INFO - ListenMW] - Message Received: " +
                            data.getMsgID().toString() + " Size: " +
                            data.getMsgData().length);
         this.currentSession.adjustCounter("MW Counter", 1, true);

         // Setup the flowfile to transfer
         FlowFile flowfile = this.currentSession.create();
         flowfile = this.currentSession.putAttribute(flowfile, "key",data.getMsgID().toString());
         flowfile = this.currentSession.putAttribute(flowfile, "value", new String(data.getMsgData(),StandardCharsets.UTF_8));

         this.currentSession.transfer(flowfile, SUCCESS);


     }catch(Exception e) {
          this.getLogger().error("[INFO - ListenMW] - "+e.getMessage());
          this.currentSession.adjustCounter("MW Failure", 1, true);
     }

I've been unable to determine what attribute(s) to use for the msgID and msgData so I created my own for now. I saw one post where someone recommended building your own json structure and sending that through as your payload, but again which attribute would you send that thru so it will get mapped properly to the kafka message? I'm pretty new to Kafka and have only experimented with rudimentary test cases to this point so forgive my ignorance for any wrong assumptions.

Thanks for any guidance! I'm using Kafka2.0.1 and the PublishKafka_2.0 processor.

Mike Thomsen :

Based on what you've shared, it looks like the main reason you're not getting anything published into Kafka is you're not actually writing anything to the flowfile contents. For a reference point, here is a copy of the javadocs for NiFi (also, here are the processor docs). What you should be doing is something like this:

flowFile = session.write(flowFile, outStream -> {
  outStream.write("some string here".getBytes());
});

I use PublishKafkaRecord, but the PublishKafka processors are pretty similar conceptually. You can set the key for the message the way you're doing it there, but you need to set the value by writing it to the flowfile body.

Without knowing your broader use case here, it looks like you can do what you need to do with ExecuteScript. See this as a starting point for ExecuteScript with multiple scripting language references.

If you need further help, we have multiple options here for you.