Design and Analysis of Remote Update of Embedded Platform Based on Compression and Differential Algorithm

The traditional embedded remote update solution generally adopts the whole package update method, which has a large amount of update data, takes up a long time of network bandwidth, and also increases the power consumption of the device.

Aiming at these problems, two remote update schemes with the core of reducing the amount of update data are proposed. These two schemes respectively use LZ77 compression and BSDiff differential algorithm to process update packets, reducing the amount of data to be transmitted; at the same time, in terms of data transmission, a breakpoint resume function is added to enable the terminal to continue to obtain data from the disconnected place, so as to achieve The purpose of saving device traffic and power consumption. In addition, a FLASH partition method is designed, which optimizes the flow of local self-update operations and removes redundant copy operations. The two schemes are verified and tested on the embedded platform built by STM32F407ZGT6 and NB-IoT. Experimental results show that the average update efficiency of these two update schemes is increased by 26.44% and 72.17% respectively compared with the package update method.

introduction

With the development of wireless communication technologies such as NB-IoT and GPRS, more and more embedded wireless terminal devices are used in daily life. The improvement of the functions of these devices and the repair of bugs are inseparable from the update of the system firmware. To better maintain these terminal devices, the function of remotely updating firmware online is particularly important.

However, in the traditional remote firmware update solution, the entire update package is usually used. Although there is little difference in code functions between the old and new update packages, it is still necessary to download the entire update package to the hardware memory wirelessly, and then Start the update. This update method takes up a long time in the network bandwidth, and it is prone to packet loss and disconnection when the network is congested, which in turn affects the power consumption of the device and the efficiency of the upgrade.

LZ77 compression algorithm

LZ77 is a dictionary-based lossless compression algorithm. First, a forward buffer and a sliding window of different lengths are respectively defined.
The data to be compressed will enter the forward buffer in order. Suppose there are n data in the forward buffer, which are X1, X2,..., Xn respectively. At this time, these data will form a dictionary Sj={(X1), (X1,X2),...,(X1,X2,...,Xn)}.
Afterwards, these data will enter the sliding window in order. Assuming that the sliding window has m data, they are divided into X1, X2,...,Xm. At this time, these data form a dictionary Sw={(X1,X2,...,Xm),( X2,...,Xm),...,(Xm)}

With the movement of the forward buffer and the sliding window, the contents in the dictionaries Sf and Sw will also change continuously.
Each time you slide one bit, look for the longest string in the dictionary Sf that can match the dictionary Sw. If it can match, mark it as an encoding consisting of three parameters: offset, matching data length, and new character, otherwise the output is not matched. character.
The LZ77 algorithm replaces redundant data with these marks, and then achieves the purpose of compressing data.

BSDiff difference algorithm

In an embedded system, even if a small amount of source code is added or removed, the compiled binary executable file will still have a large difference, because the modified source code, after compilation, will make the original code block and The address pointed to by the data block has undergone a large number of changes, but this change is regular and can be summarized by two rules:

1. In areas not affected by new code, changes are sparse and the differences between changed bytes are small, typically a few bytes apart.
2. Although the affected area varies greatly, most of them belong to the overall offset of the code block and data block pointer, and the offset value is a fixed value.

The figure enumerates the difference in bytes of the contents of the old and new executable files after a single line of code is inserted.
Many bytes of difference are 0, so the difference file is highly compressible.

The steps are:

1. Sort the data of the old file by suffix.
2. Use binary search to find the length of the longest string that can match, and calculate the differences and additions.
3. Compress the difference part and the new part to generate a difference patch package.

Remote update system solution optimization

Scheme 1 uses the LZ77 compression algorithm as the core, and its overall block diagram is shown in Figure 3.
First, put the executable file of the new version on the server, run the LZ77 compression algorithm, and compress the data in the executable file to generate a compressed package. Secondly, the compressed package will be sent to the terminal device through the NB-IoT wireless network. After the embedded terminal device receives the data, it will allow the LZ77 to decompress the program and restore the data to a complete update package. Next, the terminal device will verify the updated data. ;Finally perform a reboot update.

Solution 2 uses the BSDiff differential algorithm as the core. First, the server needs to save the upgrade package of the previous version. Whenever there is a terminal device that needs to be upgraded, the new version upgrade package is first placed on the server, and the server runs the BSDiff program to replace the old and new update packages. The data of the patch is compared and calculated to obtain the Patch patch package between the two versions, and then the data of the patch package is sent to the terminal device through the NB-IoT wireless network.
After the device terminal receives the patch package, it will allow the BSPatch program to merge the data of the old binary update package with the data of the patch package to generate a complete new update package, and the terminal device will verify the updated data; finally execute the restart update program to complete renew.

Communication Protocol Design

The system uses Quectel BC26 module as the NB-IoT wireless communication module.
The BC26 module has rich external interfaces (UART, SPI, ADC, etc.) and various network protocol stacks (TCP, CoAP, MQTT, etc.), and mainly uses AT commands to perform tasks.

Related AT commands:

When STM32 sends AT commands to the BC26 module through the serial port, the BC26 module will return OK, NULL and other information through the serial port to inform whether the AT command is executed successfully.

After STM32 is powered on, send corresponding AT commands through the serial port to let BC26 establish a TCP connection with the server, and actively query whether there is a new version. After finding a new version, continue to request the data of the update package from the server.

This paper designs the communication protocol between the terminal and the server. Table 2 shows the structure of the terminal querying the server for the upgrade protocol package, including the protocol header, protocol length, and operation code.
Wherein, the protocol length is the total length of a frame data packet not including the check code, and the check code is the sum of the total data in a frame data packet not including the check code itself.

In the data protocol returned by the server, when the value of the total size of the upgrade package is 0x0000, it means that there is no new version on the server side that does not need to be upgraded.
When the value of the total size of the upgrade package is not 0x0000, it means an upgrade is required and its actual value is the total number of bytes of the upgrade package.
At this time, the terminal will continue to request the data of the upgrade package from the server according to the format of the protocol package for requesting the upgrade data in Table 3.

Since the BC26 module has a limit on the amount of data sent and received in each frame, the length of the single packet data requested by the terminal to the server in each packet is set in the code to be 0x0100 (256 byte), that is, each request to the server for an upgrade of 256 bytes package data, while the last upgrade package is transmitted according to the actual length.
After each frame of data is transmitted, the terminal performs character string processing on this frame of data, and takes out the upgrade packet data until all the data packets are transmitted.

IAP process optimization

In application programming (IAP) supports users to use the bootloader to erase and program the on-chip FLASH of the MCU, thereby updating the local application program.

Traditional update steps:

1. Save all update package data to off-chip FLASH.
2. Perform a reboot and enter the Bootloader bootloader.
3. Copy the update package data saved in the off-chip FLASH to the on-chip User main program area.
4. Jump to the new program entry.

This system optimizes the IAP function and divides the on-chip FLASH of STM32 into three areas.
It is divided into a Bootloader guide area and two User main program areas, and defines the starting addresses of User1 and User2 FLASH respectively.

The optimized IAP adopts the method of updating the main program area of ​​User1 and User2 alternately.

1. Save the data of all update packages directly in User2, and set the flag bit.
2. Execute a reboot to enter the BootLoader bootloader.
3. Jump to the main program entry of User2.
4. For the next update, save the data of the update package in User1, set the flag bit, execute step 2, and then jump to the entrance of the main program area of ​​User1.

Reason: In the data exchange between FLASH, most of the time will be spent on erasing and copying data. The optimized IAP not only saves the use of off-chip FLASH space, but also removes the link of updating package data copy , further reducing the time required for local self-updates.

The overall implementation steps of remote update of embedded platform

First, the size of the update package is reduced by the LZ77 compression algorithm or the BSDiff differential algorithm.
Secondly, in the link of wireless network data transmission, the breakpoint resume function is added. When there are abnormal situations such as power failure and network disconnection, the serial number of the previous upgrade package will be written into the off-chip FLASH, and the power will be turned on again or the network status is good. In the case of the upgrade package, continue to request the upgrade package data from the recorded upgrade package serial number, without requesting the upgrade package data from the beginning; finally, by optimizing the local execution of the IAP function, the data copy time between FLASH is saved.

1. After the system is powered on, initialize the GPIO, clock, serial port and other functions to ensure that the system can run normally, and enter the BootLoader boot program.
2. According to the flag bit of the main program, jump to the User1 (User2) program area to start.
3. STM32 sends corresponding AT commands to the BC26 module through the serial port, allowing it to create a TCP Socket and connect to the server.
4. After the connection is completed, send the information of the current software version number of the terminal to the server according to the communication protocol in Table 2. In the response packet returned by the server, if the value of the upgrade packet size is 0x0000, there is no need to upgrade, and then exit the upgrade; if When the value is not 0x0000, it indicates that an upgrade is required and step 5 is performed.
5. According to the communication protocol in Table 3 and the upgrade package serial number recorded in the current off-chip FLASH, request the upgrade package data from the server and add 1 to the upgrade package serial number, and then return the data of the upgrade package data bit of the data frame by analyzing the server. Write the parsed data into the off-chip FLASH through the Flash_Write(wBuf,WriteAddr,NumByteToWrite) function, where wBuf is the upgrade package data to be stored, the start address to be stored in the WriteAddr bit, and NumBByteToWrite is the total number of words to be stored Section number.
6. Divide the total amount of upgrade package data by 256byte to calculate the total number of data packages that need to be requested. If all upgrade packages have been requested, go to step 8, otherwise go to step 7.
7. If the request is not completed, further judgment will be made. If the communication is interrupted or timed out due to abnormalities such as power failure and poor signal, the serial number of the previous upgrade package will be recorded, written into FLASH and disconnected from the TCP socket, and skip to step 3 ; If the transmission is not interrupted and timed out, skip to step 5 to request the next upgrade package.
8. After all the upgrade package requests are completed, the decompression program will be executed according to the corresponding experimental plan or the difference will be restored to restore the update package.
9. Use the Iap_WriteBin(BinAddr, BinBuf, BinSize) function to save all the restored update package data to the User2 main program.
10. After saving, set the flag bit of the main program and reset the upgrade package serial number to 1.
11. Restart the system, then enter the Bootloader boot program, execute the Iap_LoadBin(BinAddr) function to initialize the stack pointer and jump to the user program area whose value is BinAddr according to the main program flag bit just set, that is, complete the upgrade and record the Software version number.