Kettle and Kafka stream processing integration

Kettle8.0 version automatically integrates the kafka plugin. If the version is lower, you need to install the plugin yourself.

1. Environmental preparation

Set up Kafka stand-alone test environment in Docker of virtual machine.

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka

docker run -d --name zookeeper -p 2181:2181 -t  wurstmeister/zookeeper

docker run -d --name kafka --publish 9092:9092 --link zookeeper --
env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 --env
KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --env
KAFKA_ADVERTISED_PORT=9092 wurstmeister/kafka

Note: make sure the Docker host port is free

2. Connect Kafka in Kettle

2.1 Producer introduction

2.2 Consumer introduction

These two steps are to establish a sub-conversion, we can specify in the consumer to return data from any step of the sub-conversion, and then perform subsequent operations

3. Test results

I performed a log printing operation on the data obtained from the stream in the test