Architecture design of MQTT messaging platform for tens of millions of vehicles

In this series of articles, we will share with you how to build a reliable, efficient vehicle that meets the needs of industry scenarios based on EMQ's practical experience in the field of IoV, from theoretical knowledge such as protocol selection to practical operations such as platform architecture design. Internet platform.

foreword

With the advancement of the new four modernizations (electrification, intelligence, networking and sharing) in the entire automotive travel field, various car manufacturers are gradually building a car networking system with intelligent driving and intelligent networking as the core. The new generation of Internet of Vehicles system puts forward higher requirements for the platform architecture of underlying message collection, transmission and processing.

In the previous article of this series, "MQTT Protocol in the Internet of Vehicles Scenario", we have mentioned that the MQTT protocol is currently the most suitable communication protocol for building a data platform in the Internet of Vehicles scenario. Based on this, in this article, we will continue to discuss the collection and delivery of MQTT messages in the Internet of Vehicles scenario, and how to build a MQTT messaging platform of tens of millions of Internet of Vehicles, in order to provide platform architecture design reference for enterprise users who are engaged in the Internet of Vehicles business.

The basis of the Internet of Vehicles: message collection and transmission

Evolution of Internet of Vehicles Transmission Protocols

As we all know, vehicle-to-everything (V2X) refers to the interaction between vehicle and cloud, vehicle and network, vehicle and vehicle, vehicle and road, vehicle and human, vehicle and sensor equipment, etc., to realize the communication between vehicle and public network. The dynamic mobile communication system is a heterogeneous communication network established to meet the elements of efficiency, safety and management in every link related to the vehicle. The communication protocol running in it has become the key and core of the construction of the Internet of Vehicles system.

In the course of the development of the Internet of Vehicles, there are mainly two mainstream communication technologies, which have played a role in promoting the overall development of the Internet of Vehicles:

DSRC (DeDICated Short Range Communication) : It was developed by the American Society for Testing and Materials (ASTM) in 1992 for the business scenario of ETC. After years of improvement and iteration, it evolved into the IEEE (802.1X) vehicle networking communication technology standard. For quite a long time, DSRC technology has been the mainstream car networking communication protocol used in the mainstream production and consumer markets of the international automobile.

**C-V2X (Cellular Vehicle to Everything, cellular vehicle networking communication): **C-V2X relies on the existing cellular base station, in addition to supporting the direct communication of PC5, RSU and vehicles can pass the 4/5G channel (using Uu interface) is connected to the V2X platform to realize vehicle-road collaborative communication. Compared with DSRC, C-V2X is technically better, it enhances the security and confidentiality of communication, supports high network capacity, and can support high bandwidth and large data volume requirements.

The competition between DSRC and C-V2X technologies is fierce, and both are expected to become mainstream vehicle networking communication standards. At present, my country has the most complete 5G communication network infrastructure, so it is more inclined to adopt C-V2X (LTE-V, 5G-V2X) communication technology. Next-generation car networking architecture for autonomous driving.

The significance of message platform construction for the Internet of Vehicles

With the rapid development of the Internet of Vehicles construction today, all OEMs have formed a consensus: the purpose of the Internet of Vehicles construction is not networking for networking, nor networking for in-vehicle entertainment, networking is for data. With the Internet of Vehicles, there is data. With data, supplemented by a complete data governance and application system, everything is available.

And the target data of this business is not limited to the relevant data on the car side. In the V2X framework, it is necessary to solve the interconnection between vehicles and vehicles (V2V), vehicles and roads (V2R), vehicles and the Internet (V2I), vehicles and cloud (V2C), vehicles and people (V2H), etc. Comprehensive data collection and analysis of roads, clouds, networks, and people. The 5G-based C-V2X protocol and communication method provide basic capability guarantee for the construction of the entire system.

From traditional OTA applications to many new intelligent application scenarios such as smart cockpit, high-precision map adaptation, centimeter-level positioning, long-distance connection between vehicle and mobile terminal, mobile terminal message collection, vehicle-road cloud map, vehicle-road collaboration and many other new intelligent application scenarios, the Internet of Vehicles business is very important to the message platform. The demand for and data processing system has been expanded from the original vehicle cloud to the overall architecture construction of people-vehicle-road-network-cloud, which puts forward higher requirements for the construction of the entire message platform.

How to build a message communication and transmission system architecture with massive connections, high concurrent throughput, and low latency to ensure the ubiquity, convenience, high availability, reliability, security and high concurrency of the entire system has become a The key to the construction of a new generation of Internet of Vehicles systems in autonomous driving and vehicle-road collaboration scenarios.

Architecture design of tens of millions of car networking message platform

Next, we will take EMQ 's Internet of Vehicles message platform and overall data processing solution as an example to introduce how to build a tens of millions of Internet of Vehicles message platform.

business challenges

Secure access to on-board machines, road test units and mobile phone systems

The car side needs to cover new car-connected services such as car-machine data reporting, POI distribution, file push, configuration distribution, push message, operation care, etc. The mass message topics generated require more secure and stable access and transmission to realize message subscription and publishing . The road end needs to realize the secure access of the roadside RSU, the collection and transmission of messages, and the transmission of map data.

Real-time and Reliability for Large Concurrent Messaging

Application scenarios such as high-precision maps, centimeter-level positioning, and vehicle-road coordination all need to solve the millisecond-level low-latency and high-reliability transmission capability guarantee of massive vehicle-road map messages. The message processing platform requires high performance, low latency, and high reliability. The ability to support tens of millions of connections and millions of concurrent business scenarios.

Rich application scenario integration

In the Internet of Vehicles system with autonomous driving as the core, it is necessary to use the message platform to connect various applications based on people, roads, maps, and clouds. Connect the vehicle-end data with application scenarios such as high-precision maps, centimeter-level positioning, vehicle-road coordination, and mobile phone connection through the message platform, ensure the consumption supply of applications through the message platform, and provide high-performance, low-latency and high-reliability. data schema.

Mass data storage, processing and distribution

After the massive IoT data from people, vehicles, roads, clouds, graphs, and networks is collected, it is necessary to conduct full life cycle management for the access, storage, processing, and distribution of these large-scale real-time data streams to provide applications. Database support for dynamic and continuous data flow supports applications to deeply use car networking data to serve consumers and make business decisions.

Overall solution

In the solution, we mainly use EMQ's cloud-native distributed IoT access platform - EMQX , to realize the data connection, movement and processing of the vehicle end, the human and the road end in the Internet of Vehicles system. EMQX's integrated distributed MQTT message service and powerful IoT rule engine can provide a basic capability base for highly reliable, high-performance IoT real-time data movement, processing and integration, helping enterprises to quickly build critical business IoT platforms and applications.

Message processing for the car side

EMQX uses the MQTT protocol to access the connected car system. The vehicle terminal is connected to the EMQX distributed cluster through load balancing. The horizontal expansion capability of EMQX can realize the data communication capability of tens of millions of vehicle connections and millions of concurrent responses. Through the rule engine, one-stop capabilities such as massive message bridging message queues, persistent storage, offline message storage, etc. can be realized, while providing rich API atomic capabilities northbound integration.

In terms of security, EMQX not only supports TLS/DTLS or national secret GMSSL security protocol to ensure the reliability and stability of the system; it also provides multiple guarantee mechanisms such as heartbeat monitoring, testament message, QoS level, etc., through offline message storage to achieve real-time performance in complex network environments , Safe and reliable vehicle-machine message communication.

Message processing for people and roadside

EMQX provides a message collection and processing platform for mobile APP, RSU and other terminals for people and road ends. Based on the 5G network slicing capability, through the nearby access of personal terminals and roadside units, ultra-low latency traffic information services are realized. Through MQTT and other protocols, the road condition information perceived by the human terminal and roadside facilities is pushed to the cloud control platform, and the cloud control platform integrates V2X algorithm to realize intelligent traffic scenarios such as road collaborative perception, safety reminder, and remote collaborative control.

In terms of security, it supports the international standard TLS/DTLS encryption or the national encryption algorithm GMSSL encryption, and guarantees the secure communication of the coordination of the human-vehicle-road information system by extending the PKI/CA certificate authentication system.

10 million message access framework model

For the new generation of Internet of Vehicles scenarios, the overall message access and data processing platform reference architecture of EMQ with tens of millions of connections and millions of concurrency is as follows:

Business scenario : Vehicles, mobile APPs, roadside RSUs and other equipment in the Internet of Vehicles system are connected through MQTT, enabling concurrent access to more than ten million terminals.
**System architecture:** Terminal devices are accessed through protocols such as MQTT and HTTP, and are connected to the distributed messaging platform EMQ X through load balancing components. Through distributed multi-cluster deployment to meet the requirements of tens of millions of concurrent connections, according to the throughput capacity of millions of messages, the rules engine is connected to the Kafka cluster to realize data forwarding. The Internet of Vehicles service platform, high-precision map service, V2X cloud control service, positioning service and other applications related to the vehicle network can be directly consumed by subscribing to Kafka data. Meanwhile, EMQ provides three southbound interface services of REST, MQTT and MQ message queue. Two-way communication of vehicle control (remote control) messages.

Through the above reference framework, EMQ can achieve the business requirements of tens of millions of connections and millions of concurrent throughputs in the Internet of Vehicles scenario through the EMQX cloud-native distributed IoT access platform.

Tens of millions of messages access test

Test environment and purpose

A car company plans to verify the following capabilities of the EMQX cluster based on the test environment in the Internet of Vehicles scenario, and prepare the corresponding technical architecture and capability support for subsequent business growth:

It can support 10 million concurrent connections, and support QoS 0 messages of 100,000 to 150,000 per second, with a payload of 100 bytes, bridged to Kafka through the rule engine;
10 million concurrent connections to subscribe and consume OTA broadcast topics;
3 million users connecting at the same time will not cause cluster avalanche and test the time it takes to connect.

In addition, after completing all the above tests, continue to explore the highest throughput of message sending and bridge forwarding to Kafka that can be supported under the current configuration with 10 million concurrency (increase the client message sending frequency according to the EMQX cluster resource usage), and Tested to meet the highest message throughput with a QoS of 2 and an average response time of 50 ms under 10 million connections.

test preparation

The client connects to the load balancing ELB through TLS encryption, then TLS terminates the client at HAProxy, and finally connects to the EMQX cluster through TCP. By terminating TLS on HAProxy, the support capability of the EMQX cluster can be improved. In this deployment mode, the processing capability of EMQX is exactly the same as the client's direct connection through MQTT TCP. On the other hand, compared with the MQTT TCP connection, the client needs to consume more resources through the TLS connection. The scale of this test is tens of millions, and the number of test machines required is large. In order to reduce the required test resources, it does not Affects the test target of EMQX cluster, this test will use TCP connection directly.

Serve	quantity	Version	YOU	Instance Specifications	CPU	RAM	network card	port
Load Balancing Cloud Service	1			Network/Large II				18083/1883/8081
EMQX node	10	V4.3.4	Centos7.8	c6.16xlarge.2 ordinary block storage	64C	128G	1	18083/1883/8081/8883
Kafka cloud service	4	2.3.0	Centos7.8	c6.4xlarge.2 ultra-high IO block storage	16	32G	1	9092
XMeter Stress Test Control Node	2	3.0	Centos7.8	c6.4xlarge.2	16	32G	1	443/80/3000/8086 Intranet release
XMeter Stress Test Node	43	3.0	Centos7.8	c6.4xlarge.2	16	32G	5	Intranet release

testing scenarios

serial number	scene name	describe	Desired result
1	Ten million connections + message throughput	10 million MQTT TCP concurrent connections with a heartbeat interval of 200s. Among them, 7 million are background connections (only connect without sending messages), 3 million active users, each user reports a QOS 0 message every 15S, and the payload is 100B. Messages are bridged to Kafka via the rules engine. 1 hour test first, 24 hours stability test after passing	The success rate of the intranet test is 100%, there is no message backlog, the CPU and memory are stable during the test, and there is no large jitter.
2	message broadcast	10 million MQTT TCP concurrent connections, all connections subscribe to the same OTA broadcast topic (QoS 0, payload is 100B). Simulate an MQTT client to broadcast a message to the topic every 10 minutes, test for 30 minutes	The success rate of the intranet test is 100%, and all subscribing clients successfully consume 3 messages
3	3 million concurrent transient connections	3 million MQTT clients initiate connections at the same time, and test the time required for all connections to complete	3 million clients are successfully connected, and the cluster will not avalanche
4	Exploration of the highest message throughput under 10 million connections	The highest message throughput rate (Qos 0, payload 100B/1kB) that can be achieved under the existing configuration and 10 million connections and bridging kafka	After the highest message throughput rate is reached, the test is performed for 2 hours. The success rate of the intranet test is 100%. There is no message backlog. The CPU and memory are stable during the test, and there is no large jitter.
5	TPS at Average Response Time	Under 10 million connections, the message is QoS2, the payload is 100B, and the average response time is within 50 mm. The highest message TPS supported	Able to achieve a throughput capacity of not less than 200,000 TPS

Test Results

The following are the results of this test:

serial number	Scenes	average response time	EMQX node CPU usage	EMQX node CPU IDLE	EMQX node memory usage (G)	Bandwidth required for LB (MB)
1	10 million connections + 200,000 message throughput, QoS 0, payload 100B	1.5ms	31%~48% Average 47%	37%~54% Average 47%	Used: 27.7~42 Free: 78.2~92.5	45
2	Message broadcast under 10 million connections	100ms	up to 21%	Minimum 69%	Used Max 32.3 Free Min 87.9	200
3	3 million client instant connections	3 minutes to complete the connection	up to 25%	Minimum 63%	Used Max 14.7 Free Min 108.2	400
4	Explore the highest throughput: 10 million connections + 1.2 million message throughput, QoS 0, payload 1kB	164.3ms	23%~64% Average 46%	20%~64% Average 43%	Used: 33~38 Free: 81.3~87.1	1350
5	10 million connections + QoS2 200,000 message throughput, payload 100B	51.4ms	3%~51% Average 41%	31%~53% Average 43%	Used: 22.2~29 Free:91~98	95

summary

如以上结果所示，在目前的部署架构下，可以满足该车企对于千万并发连接 +20 万消息桥接至 Kafka、消息广播及 300 万瞬时并发连接的验证需求。在探索测试中，1000 万连接下测试到最高 120 万消息 TPS(QoS 0、payload 1kB)，测试持续 10 小时 EMQX 集群稳定，CPU idle 最低至 20%，内存使用平稳。

由以上可知，EMQX 在车联网场景下支持千万连接性能表现突出，架构稳定可靠。

压力测试工具简介和使用

本次测试由于所需测试机数量多，管理复杂，故使用 EMQ 旗下商业版测试软件 XMeter 性能测试平台和 JMeter-MQTT 插件进行。

XMeter 是基于开源测试工具 JMeter 扩展的性能测试平台。针对物联网具有的接入规模大、弹性扩展要求、多种接入协议、混合场景等特点，XMeter 对 JMeter 进行了改造，可以支持大规模、高并发的性能测试，比如实现千万级别的 MQTT 并发连接和消息吞吐测试。除了测试 MQTT 协议之外，还可以支持 HTTP/HTTPS 等主流的应用的测试。

JMeter-MQTT 插件是由 XMeter 实现的开源 MQTT 性能测试插件，在众多的项目中得到了使用，目前是 JMeter 社区中流行度最高的 MQTT 插件。

写在最后

通过本文，我们介绍了基于云原生分布式物联网接入平台 EMQX 的千万级车联网 MQTT 消息平台架构设计，并验证了该架构在千万级并发连接场景环境下的性能表现，为车联网系统的消息数据平台建设提供了一种可能的设计参考。

As the world's leading provider of IoT data infrastructure software, EMQ is committed to building high-performance, low-latency, high-availability, and high-reliability products, and providing overall solutions for message collection, movement, processing, and analysis for the next-generation Internet of Vehicles system The solution provides infrastructure service guarantee for the autonomous driving and intelligent networked vehicle business of vehicle manufacturers, T1 suppliers, after-market service providers, travel service companies and government management agencies, and realizes the intelligent connection of people, vehicles, roads and clouds .

About the author

Fan Rongjie, Director of EMQ Solution Architecture. With rich experience in the field of IT infrastructure, he has been deeply involved in enterprise digital transformation and IoT architecture construction, data platform construction, agile development, lean operation and maintenance, and DevOps concepts for many years. service to market.