Introduction to the CAN Bus protocol of the Internet of Vehicles and real-time data stream processing

What is CAN Bus?

CAN (Control Area Network) Bus is a serial communication protocol that enables reliable and efficient data transmission between devices. It is widely used in the field of vehicles, connecting various electronic control units inside the vehicle like a nervous system.

The CAN Bus was originally designed by Bosch in the 1980s for automotive applications. It is a multi-master, multi-slave, half-duplex, fault-tolerant protocol that is very suitable for the needs of the automotive field. It is simple, low-cost, reliable and able to work in harsh environments. The CAN Bus provides a unified access point for all electronic control units in the vehicle for easy connection and diagnostics.

CAN Bus data can reflect the performance and status of connected devices. However, collecting and processing CAN Bus data can be challenging due to factors such as large data volumes, limited bandwidth, and network instability.

To address these challenges, the MQTT protocol can be utilized to ensure timely transmission of vehicle data to the cloud in the event of poor network conditions. EMQX is an open source MQTT Broker, which can build a reliable and scalable MQTT infrastructure to collect CAN Bus data.

A Brief History of CAN Bus Development

The CAN Bus was developed by Bosch, a German multinational engineering giant, in the early 1980s to provide an efficient communication system for automotive applications, with the main purpose of simplifying the complexity of the vehicle's internal wiring harness.

In 1986, Bosch released the first CAN protocol, which was quickly adopted by car manufacturers due to its reliability and robustness. In 1993, it became the ISO-11898 international standard. The protocol evolution process is roughly as follows:

• 1991: Mercedes-Benz adopted CAN Bus in its W140 S-Class, becoming one of the first automakers to adopt CAN Bus.
• 2004: CAN FD is introduced, offering higher data rates and larger payloads than conventional CAN networks.
• 2015: Use ISO-16845:2015 as a test solution for conformance testing for conformance testing of devices implementing the classic CAN and CAN FD protocols.

In addition to the automotive field, the CAN Bus protocol is gradually being used in other industries, such as industrial automation systems (CANopen) and marine electronics (NMEA 2000). Its wide application is mainly due to its stable operation under harsh conditions and low cost of implementation.

How the CAN Bus works

CAN Bus is a distributed communication protocol. Its distributed nature makes it ideal for applications that require high reliability and real-time performance, such as automotive and industrial systems.

In CAN network, all nodes are connected by twisted pair or optical fiber. Each node has its own microcontroller and is responsible for processing incoming and outgoing messages. Data is broadcast by nodes on the shared bus and can be received by all other nodes. Several key stages of the communication process include:

1. Arbitration: In order to avoid conflicts caused by multiple nodes sending data at the same time, CAN adopts an arbitration process based on message priority. The smaller the identifier value of the message, the higher the priority.
2. Error detection: The built-in error detection mechanism ensures the integrity of data in the CAN network. These mechanisms include cyclic redundancy check (CRC), frame check sequence (FCS), and receiving node acknowledgment bits.
3. Fault Confinement: If a node detects an error or failure during transmission, it enters an "error passive" state until the problem is resolved. This mechanism prevents failures from interfering with the overall system functionality.

These features work together to allow the CAN Bus to operate efficiently, ensuring reliable communication between components in complex systems such as vehicles or factory automation equipment.

Message structure of the CAN protocol

In a CAN Bus system, the message structure is very important for efficient communication between devices. The data frame format of the protocol consists of the following fields: identifier, control field, data field and error detection mechanism.

• Identifier (CAN ID): This is a unique value used to determine the priority of each message on the network. The standard 11-bit identifier (CAN 2.0A) provides up to 2048 different priorities. The extended 29-bit identifier (CAN 2.0B) provides more options with more than 500 million priorities.
• Data Length Code (DLC): Located in the control field, it is used to specify the number of bytes contained in the data field, and the value ranges from 0 to 8 bytes.
• Data field: Contains the actual information to be transmitted between nodes, in bytes.
• Cyclic Redundancy Check (CRC): A built-in error detection mechanism that ensures the reliability of communications by detecting transmission errors and requesting retransmissions if necessary.
• Acknowledgment Slot: A single bit used by the receiving node to send an acknowledgment to the sending node, indicating that the message was received successfully or that a retransmission was required on error.
• Error Frame: An optional part of a CAN message used by a node to signal when it detects a problem with a message it has sent or received.

Types of CAN

CAN mainly includes three types:

low-speed CAN

Low-speed CAN, also known as fault-tolerant CAN or ISO 11898-3, has a maximum transmission speed of 125 kbps. It is suitable for less critical systems like body control modules, door locks, window controls, etc., which do not require high data transmission speed. Its main feature is that even if a wire in the bus is broken, it can continue to work normally.

high-speed CAN

High-Speed ​​CAN, or ISO 11898-2, transfers at speeds up to 1 Mbps. Because it has a faster data transfer speed than a low-speed network, it is suitable for applications that require timely response, such as engine management systems and electronic braking systems. However, it does not have the fault tolerance of low-speed networks.

CAN FD

Introduced by Bosch in 2012, CAN FD is an extended version of the high-speed network with higher data transfer speeds of up to 5 Mbps while being backward compatible with existing high-speed devices. The key advantage of this technology is that it can transmit larger loads more efficiently than conventional CAN, making it ideally suited for the increasingly complex electronic systems of modern vehicles.

CAN Bus Advantages and Challenges

What are the main advantages of CAN Bus?

CAN Bus data can reflect vehicle performance, health and behavioral characteristics. Collecting CAN Bus data to the cloud is an effective way to utilize the potential of vehicle data, and the value of data can be discovered through big data analysis. By applying machine learning, artificial intelligence or other analytical tools to the vast amounts of data collected from vehicles, vehicle manufacturers can obtain valuable information and use them to optimize vehicle performance.

• Detect, Eliminate, Predict Faults: By analyzing CAN Bus data, anomalies or erroneous signals from devices and sensors can be identified. This helps diagnose the root cause of the problem and fix it before it causes more damage or creates a security issue. Manufacturers can also use the collected data to train machine learning models to predict failures.
• Visualization of vehicle data: Users can use the collected data to build a system that displays comprehensive data on a dashboard, enabling filtering, sorting and comparison of different vehicles and indicators. The dashboard also provides alerts and recommendations based on data analysis, giving users a holistic view of performance.
• Vehicle-road coordination: By combining the collected data with road infrastructure data, a vehicle-road coordination system can be established.

In the age of artificial intelligence, data is the most valuable asset. By collecting vehicle data into the cloud and then distributing it to various data infrastructures such as databases and data lakes, users can utilize the data to implement various types of applications.

What are the challenges of real-time data collection?

Although collecting CAN Bus data locally in the vehicle is quite mature, due to high data rate, low bandwidth and unstable network environment, collecting, processing and transmitting CAN Bus data to the cloud in real time is facing great challenges. It is impractical to transmit all CAN Bus data to the cloud for processing. Therefore, CAN Bus data can be collected and processed locally at the edge to reduce the amount of data, and then the processing results can be transmitted to the cloud in real time.

We need at least two components to build such a solution:

1. Edge computing engine: The edge computing engine can only collect the required CAN Bus signals, process them flexibly, and trigger MQTT transmission actions in real time. LF Edge eKuiper is an open source edge computing engine that can help you process and analyze CAN Bus data in real time.
2. MQTT Broker in the cloud: Using MQTT Broker, the processed CAN Bus data can be transmitted to the cloud in real time. EMQX is an open source MQTT Broker, which can build a reliable and scalable MQTT infrastructure to collect CAN Bus data.

Next, we will introduce in detail the overall solution combining EMQX and eKuiper.

Addressing CAN Bus data local processing challenges

eKuiper is an open source edge computing engine that can help you process and analyze CAN Bus data in real time. eKuiper is designed for edge stream processing, suitable for real-time processing of typical stream data generated by CAN Bus. eKuiper addresses the following challenges:

• Possess the ability to efficiently process CAN Bus large-capacity and high-speed data in real time. The ability to flexibly filter, process, and select signals of interest reduces the bandwidth required to transmit data.
• Ability to parse binary CAN frames into meaningful signals for rule processing and action triggering. It supports dynamic loading of DBC files, enabling users to change DBC files flexibly and adapt to different CAN Bus devices without restarting the engine. At the same time, it also ensures the privacy and security of the DBC files without being exposed to the development team.
• With the ability to flexibly combine signals from different CAN frames, complete messages can be constructed by rules for use by applications. Users can also freely modify the rules to adapt to different user scenarios or changes in requirements, and support hot loading.

Tutorial: Use eKuiper to realize local processing of CAN Bus data

Step 1: Connect to CAN Bus

eKuiper uses a CAN source plug-in to connect to the CAN Bus and receive CAN frames. It supports two modes of connecting CAN Bus, as shown in the figure below.

1. Connect directly to CAN Bus via socketCan. SocketCAN uses the Berkeley socket API, the Linux network stack, and implements CAN device drivers as a network interface. Once the CAN Bus is connected to the Linux system, the user can get the CAN network interface. In eKuiper, users can create CAN streams by specifying CAN network interface and DBC file path . Then, any rules can be applied to the CAN stream to process the CAN Bus data.
2. Utilize TCP/UDP to connect to CAN Bus through gateway. The gateway can be a CAN adapter that combines multiple CAN frames into one data packet and sends it in batches via TCP or UDP. Note that the packet format sent by the gateway is not standardized, so the plugin may need to be modified to accommodate it. In eKuiper, users can create CAN streams by specifying TCP/UDP address and DBC file path . Then, any rules can be applied to the CAN stream to process the CAN Bus data.

Step 2: Decode CAN Bus Data

CAN Bus data is binary data and in units of frames, and CAN frames consist of multiple fields. There are several versions of the CAN protocol, including CAN 2.0A, CAN 2.0B and CAN FD. Different versions of the CAN frame format are slightly different. The figure below shows the CAN 2.0A frame format.

Among them, two fields are important for decoding CAN Bus data:

1. ID field: The ID field is used to identify the CAN frame. The ID field of CAN 2.0A is 11 bits, and that of CAN 2.0B and CANFD is 29 bits.
2. Data field: The data field is the payload and is used to carry the actual data. CAN 2.0A and CAN 2.0B are 0-8 bytes, and CAN FD is 0-64 bytes.

In the payload, the data consists of a series of signals, each with its own name, length and location. A DBC file is a text file that contains information to decode raw CAN Bus data into "physical values". This file defines the signal's name, length, location, and conversion formula used to convert raw data into physical values.

In eKuiper, users can specify the DBC file path to use when parsing CAN Bus data. Configuring DBC in eKuiper is very flexible and secure.

• Multiple DBC files: The user can select a directory as the DBC path, and the DBC files in the directory will be loaded one by one in alphabetical order and take effect. This approach allows the user to incrementally add or restore signals through separate DBC files.
• Dynamic DBC file loading: DBC files are loaded at runtime and do not need to be deployed at development time. This helps users keep their DBC files private and secure.
• Hot Loading: Users can change the DBC file at any time to suit different CAN Bus devices without restarting the engine.

After configuring the eKuiper CAN source, users can create a stream to receive and parse CAN Bus data to extract meaningful physical signals from it. For example, a CAN payload 0x0000000000000000can be parsed into the following signals:

{
  "signal1": 0,
  "signal2": 0,
  "signal3": 0,
  "signal4": 0,
  "signal5": 0,
  "signal6": 0,
  "signal7": 0,
  "signal8": 0
}

Next, users can use eKuiper's powerful stream processing capabilities to flexibly process the parsed data just like receiving data from MQTT.

Step 3: Process CAN Bus Data

After getting the parsed data, we can use eKuiper to perform various operations on it. To save bandwidth for transmitting data, we can pick only the signals of interest. For example, you can select only the two signals signal1and .signal2

{
  "signal1": 0,
  "signal2": 0
}

This can be easily achieved with the eKuiper SQL statement:

SELECT signal1, signal2 FROM canStream

Due to the limited size of a CAN frame, we may need to extract the desired signal from multiple CAN frames. Therefore, we can flexibly combine the signals in different CAN frames using algorithms as needed to build a complete message for the application. You can learn more details in the example of data merging . Here, we use signal1 as the main condition to filter the data.

SELECT signal1, latest(signal2) as signal2 FROM canStream WHERE isNull(signal1) = false

Another example is to only collect data when certain events occur, which can significantly reduce bandwidth consumption. For example, we can collect data only when signal1 exceeds 100.

SELECT signal1, signal2 FROM canStream WHERE signal1 > 100

Furthermore, these processing rules are very flexible and can be modified at any time . Even if the required signals are not identified at the initial stage, there is no need to worry, you can adjust the rules at any time as your needs change, and implement hot reloading.

In addition to the common use of data collection, eKuiper can also be applied to other scenarios, such as:

1. Real-time and flexible rule engine on the vehicle side: corresponding actions can be triggered according to specific conditions. For example, automatically close the windows when the speed exceeds 70 mph.
2. Flexible intelligent analysis: Data and AI models (currently TF Lite) can be combined without coding or connecting to the cloud. This feature enables real-time data analysis, such as predicting and suggesting driving patterns based on data such as speed and tire pressure (even without a network connection).
3. Edge computing: It can reduce transmission bandwidth, reduce cloud computing pressure, and analyze, reorganize, and convert data, such as calculating and saving the average speed over a period of time.
4. Heterogeneous data integration: Data from different protocols (TCP, UDP, HTTP, MQTT) and different formats (CAN, JSON, CSV, etc.) can be parsed and merged through flexible rules.
5. Message routing: It can decide which data is sent to the cloud and which data is stored locally for use by other in-vehicle applications. For example, routing based on GDPR or some whitelist.

Use MQTT to collect CAN Bus data

Using MQTT Broker such as EMQX to collect CAN Bus data has the following advantages:

• Reduce network overhead: MQTT uses binary format and minimal header to encode messages, which can save network bandwidth and improve data transmission efficiency.
• Improve scalability: A single MQTT Broker can support thousands of concurrent connections and millions of messages per second. This enables large-scale data collection from multiple CAN Bus devices without compromising performance or reliability.
• Enhanced security: MQTT supports multiple security mechanisms, such as TLS/SSL encryption, username/password authentication, and access control list (ACL) to prevent unauthorized data access or tampering.
• Improved interoperability: MQTT is based on open standards and is widely supported by various platforms and languages. This facilitates the integration of CAN Bus data with other systems or applications.

In addition, EMQX provides many other functions and can work with eKuiper to help users save bandwidth, reduce delay and improve reliability when transmitting CAN Bus data.

save bandwidth

When transmitting CAN Bus data through MQTT, we usually need to transmit in a weak network environment with limited bandwidth. Therefore, we need to minimize the size of the data.

In eKuiper sink, we can use formatoptions to specify the data format. The default format is JSON. We can change this protobufto serialize the data into a binary format to significantly reduce the data size. In addition, we can use compressoptions to compress data via gzipor other compression methods. In these ways we can make the size of the data much smaller than the original JSON data. In one of our test cases, the data size can be reduced by more than 90% when sending data in batches.

Real-time data

For cloud applications, some data is time-sensitive, such as data used for vehicle accident diagnosis. In this case, reducing latency is very important. In eKuiper rules, we can use MQTT sink to quickly send data to EMQX to meet the needs of efficient data transmission.

In order to save bandwidth in real-time scenarios, we can set the serialization format and compression method in the eKuiper MQTT sink as described above. EMQX provides a rule engine that can decompress and deserialize data, so that cloud applications can process data in real time without writing code.

Bulk save data to file

For data that is not urgently processed, we can save it in a file or a local database, and then send it to the cloud in batches. This can achieve a higher compression ratio, which in turn saves more bandwidth. In eKuiper rules, we can use file sinks to store and compress data locally. The file sink supports setting the file rolling policy, such as rolling every 10 minutes, so that we can save the data in batches in the file. EMQX is developing a new function to transfer files with MQTT. After this function is completed, the saved file can be transmitted by MQTT. Currently, users can use other tools to transfer files to the cloud.

epilogue

This article introduces how to use eKuiper and EMQX to collect, process, and transmit CAN Bus data from vehicles to the cloud. In the following articles, we will explain each step in more detail.

Copyright statement: This article is original by EMQ, please indicate the source for reprinting.

Original link: https://www.emqx.com/zh/blog/can-bus-how-it-works-pros-and-cons