Kettle integrates Rabbitmq message queue
For information about Rabbitmq message queue, please refer to my previous blog
1. Introduction
Kettle does not have a direct rabbitMQ step, but there are MQTT steps under the streaming category. rabbitMQ uses the amqp protocol by default, but you can also enable the MQTT plugin to use the MQTT protocol.
Therefore, we use Kettle to produce and consume rabbitMQ through MQTT protocol steps.
2. Environmental preparation
My test environment is a virtual machine Docker
docker pull cyrilix/rabbitmq-mqtt
docker run -p 1883:1883 -p 4369:4369 -p 5671:5671 -p 5672:5672 -p 8883:8883 -p 15672:15672 -p 25672:25672 -p 15675:15675 -d cyrilix/rabbitmq-mqtt
The default port of the abbitmq management console is 15672, the default port of mqtt is 1883, and the default port of websocket is 15675, so port mapping is required when starting the container, otherwise it cannot be accessed through the host machine
Three. Rabbitmq configuration
- Enter the rabbitMQ management interface:
http://你的IP:15672
- Default login user guest/guest
- Create a "myTopic" topic under exchange
- Add new myTopic permissions to the guest user
Four. Kettle connects to the message queue
4.1 Configure MQTT producer
- Create a transformation
- Add conntion, topic, user/pass and other information
Note: The port must be noted, it is no longer 5672 but 1883, which is the MQTT protocol port.
4.2 Configure Consumer
- Create a conversion, select MQTT consumers, configure topics, and process data conversion
- Create a message to process the conversion, use the Get records from stream node to get the message
5. Testing
- Start consumers, kettle will continue to monitor
- Run the producer and generate a message
- At this time, the consumer receives the data and prints it out
