CKafka builds a one-stop data transfer link to help the Great Wall Internet of Vehicles platform reduce operation and maintenance costs.

12.23 Yuan Chuanghui·Shanghai Station, let’s talk about LLM infrastructure

About Great Wall Intelligent New Energy

Great Wall Motors is a global intelligent technology company whose business includes automobile and parts design, research and development, production, sales and service. It owns Wei brand, Haval, Tank, Euler and Great Wall pickup trucks. In 2022, Great Wall Motors sold 1,067,523 vehicles throughout the year, with sales exceeding 1 million vehicles for seven consecutive years. Great Wall Motors provides smart and green travel services to users around the world, accelerating its advancement into a global smart technology company. The penetration rate of smart models reaches 86.17%. The Internet of Vehicles, one of the two major application directions of smartness, is developing rapidly in this process.

The Great Wall Internet of Vehicles platform covers in-car bus data reporting, remote control, vehicle configuration distribution, file push, message push, operation care and other new Internet of Vehicles services, realizing the decoupling of the vehicle terminal and the business platform and efficiently completing business docking and integration.

Main scenarios include:

● Vehicle-side data reporting - motor, position, engine, vehicle data, battery, alarm, etc., reported to the Internet of Vehicles platform through tbox, and real-time data processing, calculation, and reasoning are performed on the reported data to provide intelligent vehicle status inquiries, alarms, etc. Serve.

● Remote control - mobile phone APP/smart devices integrate the capabilities of the Internet of Vehicles platform to achieve remote control and diagnosis.

The following is the architecture diagram (schematic) of the Internet of Vehicles platform.

Internet of Vehicles platform architecture diagram (schematic)

The challenges of explosive growth

The Internet of Vehicles platform has currently connected millions of vehicles, with a peak number of one million vehicles online. The amount of signal data reported by the vehicle terminal is large and the frequency of uploading is high. The data is growing explosively. Real-time processing and analysis of massive data face serious challenges.

The following requirements are placed on the system:

1. High processing time requirements

Query timeliness, analysis and decision-making, monitoring and alarming

2. Large amount of data and stable

Distributed, parallel expansion, low coupling, high availability, data security

IoT devices usually have weak performance, making it difficult to use popular traditional messaging middleware. Basically, MQTT is used for message transmission in IOT devices. However, MQTT has the following disadvantages:

1. Just queuing, not stream processing

2. Unable to handle usage surges (no buffering)

3. Most MQTT brokers do not support high scalability

4. Asynchronous processing (usually offline for a long time)

5. Lack of good integration with other parts of the enterprise

6. Single infrastructure (usually at the edge)

7. Events cannot be reprocessed

Only MQTT data may be discarded before it can be processed, and it cannot meet the challenges brought by real-time processing and analysis of massive data.

solution

As a distributed message queue, Kafka can achieve high-throughput data processing due to its design and features such as multi-partition, zero-copy, batch processing, and sequential reading and writing. At the same time, as an event streaming platform, it combines messaging, storage and data processing to build a highly scalable, reliable, secure and real-time infrastructure. From the perspective of Internet of Vehicles, it has the following advantages:

1. Stream processing, not just queuing

2. High throughput

3. Large-scale

4. High availability

5. Long-term storage and buffering

6. Events to be processed again

7. Well integrated with other parts of the enterprise

The combination of Kafka and MQTT is a natural choice for building a scalable, reliable and secure Internet of Vehicles infrastructure, so Great Wall Internet of Vehicles Platform chooses Kafka as the core component of data processing.

MQTT's Broker cluster is then connected to the Kafka cluster. Data is first collected from the device through MQTT, and then dumped to Kafka for subsequent engine analysis and processing. Even if the processing speed is not as fast as the collection speed, the data will not be lost because it has been dumped to Kafka. Great Wall uses this solution to achieve continuous monitoring and analysis of the status of Internet of Vehicles equipment.

However, building your own Kafka brings increasing R&D and operation and maintenance costs:

First of all, R&D and operation and maintenance personnel who solve problems need to have solid computer skills (familiar with computer networks, IO, etc.), have a deep understanding of Kafka's underlying principles, various configuration parameters, etc., and can perform Kafka cluster parameter tuning and rapid Handle sudden failures, restore cluster jitter, and dynamically perform cluster expansion and contraction, etc.

Secondly, on the one hand, more manpower and material costs need to be invested. On the other hand, the health of the cluster needs to be monitored at all times and problems can be eliminated in a timely manner to ensure the stable operation of the business.

Finally, the self-built message queue has deficiencies in scalability and maintainability. When the volume of business message data reaches a certain level, the self-built message queue cluster will cause various problems, and the solution of the problem will bring A big challenge.

Here are a few simple examples:

● When an abnormality occurs in the cluster, it is difficult to troubleshoot and locate the problem due to incomplete monitoring indicators and unreasonable log output. The problem can only be solved by suspending the business and restarting the Kafka cluster, which will have a greater impact on the business.

● Kafka’s cluster expansion is highly complex. When migrating during business peaks, partition migration may become stuck.

● It is difficult to operate and maintain ZK in self-built clusters, and ZK has a high load, resulting in frequent disconnection of ZK.

After communicating with the Tencent Cloud technical team, CKafka (Cloud Kafka), as a version of Kafka on the cloud, has a complete monitoring and alarm system and operation and maintenance work order system, and has strong advantages in performance, scalability, business security, operation and maintenance, etc. You can enjoy low cost, high performance, and rich functions while eliminating tedious operation and maintenance work.

The Internet of Vehicles platform uses CKafka, a high-performance, high-throughput, scalable distributed message queue engine, to achieve business decoupling, peak load shaving and valley filling, and asynchronous data processing to achieve high business reliability.

Data reporting scenario

Real-time data generated by vehicles (such as GPS location, speed, fuel consumption, etc.) are collected, transmitted, and distributed through CKafka, realizing multiple flows of one data to meet the needs of multiple scenarios.

Real-time calculation part

Through the Kafka connector provided by Flink, the streaming data is processed by Flink operators and falls into the high-performance columnar database Clickhouse for real-time update data analysis. This process provides exact-once processing semantics, while CKafka multi-partitioning provides higher throughput and reduces data skew and hot spots.

Vehicle status data such as vehicle failures and abnormal behaviors can be quickly discovered and dealt with through real-time analysis.

Offline analysis part

Through log collection systems such as Flume, massive log data in CKafka can be efficiently collected, aggregated, moved, and finally stored in HDFS or Hbase. In the production and processing link, when production and processing speeds are inconsistent, CKafka can act as a cache. Having a Partition structure and using Append to append data make CKafka have excellent throughput capabilities; at the same time, it has a Replication structure, making CKafka highly fault-tolerant.

Vehicle data is analyzed and mined offline, and the analysis results can be used to optimize vehicle performance, improve driving safety, reduce energy consumption, etc.

Instruction issuing scenario

In the command issuance scenario, CKafka accepts remote commands and response results, providing asynchronous coupling, peak-shaving and valley-filling capabilities for multiple upstream and downstream systems. At the same time, message persistence and traceable product features can ensure the final consistency of the command status. sex.

Business benefits after using CKafka

Compared with self-built Kafka, CKafka has a complete monitoring and alarm system and operation and maintenance work order system. CKafka R&D experts are ready to answer questions and solve customer problems quickly, saving worry and effort.

CKafka has strong advantages in performance, scalability, business security, operation and maintenance, etc., allowing customers to enjoy low cost and super functions while eliminating tedious operation and maintenance work. When the traffic and disk capacity of the CKafka cluster exceeds the alarm threshold, the backend will promptly expand the equipment without being aware of the client, solving the long-standing pain points of data migration in open source Kafka, making configuration upgrades invisible, and easily coping with business peaks.

In addition to scalability, Ckafka supports customized multi-availability zone deployment in the same region, cross-regional disaster recovery, and improved business disaster recovery capabilities.

future outlook

In response to the two core requirements of reducing storage costs and quickly responding to sudden traffic peaks, CKafka will evolve into a pay-as-you-go storage model and launch elastic bandwidth capabilities.

● Storage by volume

Flexible billing is based on the actual use of storage space, without the need to consider reserved storage space, which is more flexible, easier to operate and maintain, and lower in cost.

● Flexible bandwidth

Provide a certain range of floating space (i.e. elasticity) based on the given bandwidth specification.

If there is a sudden traffic glitch, the cluster will not trigger current limiting. Instead, it will elastically expand and contract within the specified range. Traffic exceeding the original bandwidth will be billed on a per-meter basis.

Through reasonable architectural design and flexible product capabilities, CKafka helps users host high-throughput, high-availability, easy-to-use and operation-free message queue Kafka services on the cloud at a lower cost, and build a one-stop data flow link. We also look forward to more cooperation with customers in the travel industry in the future and sharing more best practices in the cloud.