MQTT and Kafka｜Internet of Things Message and Flow Data Integration Practice

How does MQTT work with Kafka?

MQTT (Message Queuing Telemetry Transport) is a lightweight message transport protocol designed for device communication in a restricted network environment. Apache Kafka is a distributed stream processing platform designed to process large-scale real-time data streams.

Kafka and MQTT are complementary technologies for end-to-end integration of IoT data. By combining Kafka and MQTT, enterprises can build a powerful IoT architecture to achieve stable connection and efficient data transmission between devices and IoT platforms. At the same time, it can also support real-time processing and analysis of high-throughput data for the entire IoT system.

The integration of MQTT and Kafka can bring significant value to many IoT scenarios, such as connected cars and vehicles, smart city infrastructure, industrial IoT monitoring, logistics management, etc. In this article, we will introduce how to realize the seamless integration of MQTT data and Kafka in IoT applications.

What IoT challenges can Kafka and MQTT solve?

When designing an IoT platform architecture, several challenges need to be addressed:

• Connectivity and network resilience : In some critical IoT scenarios, such as connected cars, data needs to be sent to the platform through a network connection. The architecture should be able to cope with various network conditions such as unstable network connections and network delays.
• Scalability : In order to cope with the increasing number of devices, the architecture should have good scalability and be able to handle the large amount of data generated by the increasing number of IoT devices.
• Message throughput : IoT devices generate massive amounts of data in real-time, such as sensor readings, location information, and more. The platform architecture must support high message throughput to ensure that all data is efficiently collected, processed and distributed to the appropriate components.
• Data storage : IoT devices continuously generate data streams, requiring efficient data storage and management solutions.

Why do you need to integrate MQTT and Kafka in the IoT architecture?

As a reliable stream data processing platform, Kafka can effectively promote data sharing between enterprise systems, but in the IoT scenario, it has some shortcomings:

• Unreliable connections : Kafka clients require stable IP connections, which is a challenge for IoT devices running on unstable mobile networks. These networks have very unstable connections, causing interruptions in the constant communication that Kafka requires.
• Client complexity and resource intensity : Kafka clients are known for their complexity and resource consumption. This is a challenge for small, resource-constrained IoT devices, where it might not be practical or efficient to run a Kafka client.
• Topic scalability : Kafka has some limitations in handling a large number of topics. This can be a problem for IoT applications, as they may involve many different topics, and Kafka's architecture may not be able to accommodate this effectively, especially if a large number of devices are involved, each with multiple topics case.

Through the integration of MQTT and Kafka, many limitations of Kafka in IoT device connection can be overcome:

• Reliable Connections : MQTT is designed to operate in unstable network environments, thus becoming a reliable message transfer protocol between IoT devices.
• Lightweight client : The MQTT client is designed to be lightweight, making it ideal for resource-constrained IoT devices.
• Massive Topic Expansion : MQTT excels at handling a large number of business topics, making it ideal for IoT platforms with a large number of topics. Massive topics can be aggregated and mapped to Kakfa topics through MQTT to realize the aggregation and processing of IoT data.

Comparison of several feasible MQTT-Kafka integration solutions

There are several options for integrating MQTT and Kafka in IoT platforms. Each option has its own pros and cons and factors to consider. Let's take a look at some commonly used MQTT+Kafka integration solutions.

EMQX Kafka data integration

EMQX is a popular MQTT Broker, which can realize seamless integration with Kafka through its built-in Kafka data integration function. As a bridge between MQTT and Kafka, EMQX realizes smooth communication between the two.

This integration makes it possible to create data bridges in both the roles of producers (sending messages to Kafka) and consumers (receiving messages from Kafka). EMQX allows users to establish data bridges in any of these two roles. EMQX has bidirectional data transmission capability, which provides great flexibility for architecture design. In addition, it features low latency and high throughput, ensuring efficient and reliable data bridging operations.

Confluent MQTT Broker

Confluent is the commercial operating company of Kafka. It provides an MQTT protocol proxy module for connecting MQTT clients and Kafka Broker, enabling clients to publish and subscribe to Kafka topics. This solution abstracts the complexity of direct communication with Kafka Broker, simplifies the integration process, and avoids redundant replication and delay.

Currently, this solution only supports MQTT version 3.1.1, and the connection performance of the MQTT client may affect the data throughput.

Custom development of open source MQTT Broker and Kafka

Users can use the open source MQTT Broker to develop their own bridging service to realize the connection between MQTT and Kafka. This bridge service subscribes data from MQTT Broker through MQTT client, and utilizes Kafka Producer API to send data to Kafka.

This solution requires users to develop and maintain bridging services themselves, and to consider reliability and scalability issues.

Use EMQX to integrate MQTT data into Kafka

As a highly scalable MQTT Broker, EMQX provides powerful functions for IoT platforms. Its data integration capabilities enable easy and efficient bidirectional transmission of MQTT data with Apache Kafka.

EMQX supports massive device connections, combined with Kafka's powerful high throughput and persistent data processing capabilities, it builds a perfect data infrastructure for the Internet of Things.

EMQX provides the following functions from MQTT to Kafka :

• Two-way connection : EMQX can not only forward the MQTT messages of devices to Kafka in batches, but also subscribe to Kafka messages from the back-end system and send them to connected IoT clients.
• Flexible MQTT-to-Kafka topic mapping : EMQX supports multiple topic mapping methods, such as one-to-one, one-to-many, many-to-many, etc., and also supports MQTT topic filters (wildcards).
• EMQX Kafka producer supports synchronous/asynchronous writing mode, which can flexibly balance delay and throughput according to different scenarios.
• Real-time metrics , such as total number of messages, number of successful/failed deliveries, message rate, etc., can be used in conjunction with SQL rules to extract, filter, enrich, and transform data before pushing messages to Kafka or devices.

Application Scenario Example: MQTT and Kafka Enabling Connected Cars and the Internet of Vehicles

The architecture of MQTT + Kafka is suitable for various IoT platforms in different industries, especially in the field of connected cars and the Internet of Vehicles.

The following are the main application scenarios of this architecture:

• In-vehicle information system and vehicle data analysis : MQTT + Kafka architecture can realize cloud access, stream processing and analysis of massive real-time vehicle data, such as sensor readings, GPS location, fuel consumption and driving behavior data, etc. This data can be used for vehicle performance monitoring, predictive maintenance, fleet management and improving overall operational efficiency.
• Intelligent traffic management : By integrating MQTT and Kafka, data from various traffic sources such as connected cars, traffic sensors, and infrastructure can be acquired and processed. This facilitates the development of intelligent traffic management systems that enable real-time traffic monitoring, congestion detection, route optimization, and intelligent traffic signal control.
• Remote diagnosis : MQTT + Kafka architecture supports high-throughput data transmission for connected cars. It can be used for remote diagnostics and troubleshooting, enabling proactive maintenance and rapid problem resolution.
• Energy efficiency and environmental impact : The MQTT + Kafka architecture enables connected cars to have two-way data interaction with smart grid systems and energy management platforms. This application scenario includes monitoring energy consumption in real time, implementing demand response mechanisms, and optimizing electric vehicle charging strategies.
• Predictive Maintenance : MQTT + Kafka architecture enables continuous tracking of vehicle health and performance data. This application scenario involves high-throughput real-time in-vehicle data collection, anomaly detection and predictive maintenance algorithms. Car owners can discover potential problems in time and schedule maintenance tasks.

epilogue

The MQTT + Kafka architecture is very suitable for application scenarios that require real-time data collection, scalability, reliability, and IoT integration capabilities. It enables smooth data transmission, efficient communication, and innovative applications, such as various functions and services in the connected car ecosystem. Therefore, the combination of MQTT and Kafka is an ideal IoT architecture solution, which enables seamless end-to-end integration between IoT devices and the cloud, and ensures the reliability of bidirectional communication.

Copyright statement: This article is original by EMQ, please indicate the source for reprinting.
Original link: https://www.emqx.com/zh/blog/mqtt-and-kafka