CRC check (2): CRC32 table lookup method detailed explanation, code implementation and CRC reversal

For current CPUs, CRC checks are basically implemented on hardware. But we still want to use software to implement the CRC code, so that we can have a deeper understanding of the principles inside. So this section will explain in detail how to use the look-up table method to realize the CRC-32 verification from the software. In addition, CRC also has a reversal situation. In fact, there is no big difference between reversal and non-reversal, mainly due to the difference in requirements and standards.

1 find the pattern

Let's take a look at the example in the previous section, the calculation process of the dividend 100100and the divisor is 1101:

Let's find the law and think about how it should be implemented in the computer. It is not difficult to find the following rules:

• The highest bit of the divisor must be 1, because we use the highest order number of the polynomial it represents
• For the dividend, if its highest bit is not 1, it will "divide", do nothing, shift one bit to the left, and continue to judge.
• Since the XOR operation is done, the current result does not necessarily have to "subtract" the dividend. It only needs to satisfy that the highest bit of the divisor is 1, then perform XOR operation with the dividend, and then shift one bit to the left

2 Principle of look-up table method

If we want to implement CRC in software, it is very inefficient to do these calculations one by one, so we wonder if we can calculate every 8 bits, because each specific number and the dividend pass through 8 times The XOR results obtained by the shift operation are all the same.
(1) Why not count every 16 bits?
This will save $2^{16}$ pieces of uint16_t type data, a table has 128KB, which is larger than the Flash of many single-chip microcomputers
(2) What is the theoretical basis for calculating every 8 bits?
We know that$CRC(0)=0$ , and then we can derive this formula from the characteristics analyzed above:$CRC(A+B)=CRC(A)+CRC(B)$。

Now for example, if we want to find $CRC(0\times abcdef12)$ , he can be equivalent to:
$CRC(0\times abcdef12)=CRC(0\times ab000000)+CRC(0\times 00cdef12)=CRC(0\times ab)+CRC(0\times cd)+CRC(0\times ef)+CRC(0\times 12)$

If we define an 8-bit look-up table, we can shorten the original 32 shift and XOR operations to 4 times.

3 CRC32 code C language implementation

3.1 CRC32 table generation

After the previous analysis, we know that we just need to do a modulo two division for every number in the range of uint8_t (0~255), and then save the result to a table of type uint8_t and size 256. We only need to judge whether the highest bit of CRC is 1, if it is 0, then shift it to the left by one bit, if it is 1, first shift the CRC to the left by 1 bit, and then do XOR operation with the polynomial. The CRC32 table generation code is as follows:

void GenerateTable(uint32_t polynomial)
{
	for (int byte = 0; byte < 256; ++byte)
	 {
		uint32_t crc = byte;
		
		for (int bit = 32; bit > 0; --bit)
		{
			if (crc & 0x80000000)
			{
				  crc = (crc << 1) ^ polynomial;
			}
			else
			{
				  crc <<= 1;
			}
		}
		crcTable[byte] = crc;
	 }
}

If you compare the above code with the picture in the previous example, I believe many people will wonder why it is not executed when the highest bit of the CRC is 1 crc = (crc ^ polynomial) << 1?

Don't forget, for CRC32, the binary corresponding to its polynomial has 33 bits. In fact, we cannot use a single uint32_tvariable to save this polynomial. But we know that the highest bit of the CRC32 polynomial must be 1, so here we save the lower 32 bits as polynomial. When the highest bit in the code CRCis 1, the result of its XOR with the highest bit of CRC32 must also be 0. So here, first CRCshift right by one bit, and then XOR with the polynomial, and cycle 32 times in this way to get the corresponding one polynomialunder the polynomial , and store it in .byteCRCcrcTable

3.2 CRC table lookup code implementation

With the previous table, we can use the following function to calculate the CRC32 of a string with a length of len. msgThe result of shifting eight bits to the right is XORed, and finally the CRC result of this string is obtained.

unsigned int calcMsgCRC(char *msg, unsigned int len)
{
    unsigned long crc = 0;
    for (int n = 0; n < len; n++)
    {
    	uint8_t c = msg[n] & 0xff;
        crc = crcTable[(crc >> 24) ^ c] ^ (crc << 8);
    }
    return crc;
}

• In the CRC algorithm of some protocols, the CRC has a fixed initial value, and after the CRC result is obtained, a fixed number is also required to be XORed.

In addition, in some occasions, such as obtaining the latest firmware from the serial port to upgrade the system, the integrity of the firmware is also checked by CRC. In some devices, there is not so much memory to save the entire buffer, and then calculate the CRC check of the entire buffer. For this case, we can calculate the CRC segment by segment, and then assign the result of the previous CRC to the unsigned long crcinitial value in the function. That is, calcMsgCRCone more parameter is added to the function lastCRC.

• I extracted a segmented CRC code from the NXP SDK before, and you can also use this framework: CRC32 segmented calculation implementation in C language

4 CRC Inversion

Sometimes you may find that the CRC table of some standard protocols is different from the one you generated. This is probably because CRC has two implementations:

• Non-inverted CRC: each bit of data is processed sequentially from the highest bit to the lowest bit
• Reverse CRC: The processing order of each byte or bit of data can be reversed as required.

4.1 Standard inversion function of CRC32

I haven't found the definition of this inversion, but I have found the code, so let's understand from the code how the so-called inversion function is inverted.

• Of course, these inversions can be defined by yourself, for example, if you convert the little-endian CRC to big-endian, each bit is inverted, etc. The inversion method introduced here is used in the CRC algorithm used in some well-known protocols.

uint32_t Reverse(uint32_t value)
{
  value = ((value & 0xAAAAAAAA) >> 1) | ((value & 0x55555555) << 1);
  value = ((value & 0xCCCCCCCC) >> 2) | ((value & 0x33333333) << 2);
  value = ((value & 0xF0F0F0F0) >> 4) | ((value & 0x0F0F0F0F) << 4);
  value = ((value & 0xFF00FF00) >> 8) | ((value & 0x00FF00FF) << 8);
  value = (value >> 16) | (value << 16);
  return value;
}

(1) Take out the odd and even bits of the value respectively, and shift the odd bit to the right by 1 bit, and the even bit to the left by 1 bit, and then perform a bitwise OR operation on them. This swaps the values ​​of the odd and even bits.
(2) Group every 2 bits of the value, and reverse the bit order of the high and low bits of each group. Similarly, use bitwise AND and bitwise OR operations to exchange high and low bits.
(3) Group every 4 bits of the value, and reverse the bit order of the high and low bits of each group.
(4) Group every 8 bits of the value, and reverse the bit order of the high and low bits of each group.
(5) Reverse the bit order of the upper 16 bits and lower 16 bits of the value

4.2 CRC input/output inversion

In addition, we can choose to reverse the CRC when inputting, or choose to reverse the CRC when outputting, of course, we can also choose to reverse both. So at this time, the reverse CRC32 table generation function needs to be slightly modified:

void GenerateTable(uint32_t polynomial, bool reflectIn, bool reflectOut)
{
    for (int byte = 0; byte < 256; ++byte)
    {
        uint32_t crc = (reflectIn ? (Reverse(uint32_t(byte)) >> 24) : byte);

        for (int bit = 32; bit > 0; --bit)
        {
            if (crc & 0x80000000)
            {
                crc = (crc << 1) ^ polynomial;
            }
            else
            {
                crc <<= 1;
            }
        }
        crcTable[byte] = (reflectOut ? Reverse(crc) : crc);
    }
}

For the calculation function of reversing CRC (assuming both input and output are reversed), it should be modified as follows:

unsigned int calcMsgCRCReverse(char *msg, unsigned int len)
{
    unsigned long crc = 0;
    for (int n = 0; n < len; n++)
    {
        uint8_t c = msg[n] & 0xff;
        c = Reverse(c);
        crc = crcTable[(crc ^ c) & 0xFF] ^ (crc >> 8);
    }
    return Reverse(crc);
}

When using inverted CRC or non-inverted CRC, the method of generating and looking up the CRC table will also be different. There is no obvious difference in performance or accuracy between these two methods in practical applications, and which method to choose mainly depends on the requirements and standards of the application.