About 1 LZW algorithm
of data compression is divided into lossy and lossless compression. Lossy compression is a data compression reconstruct (decompress), data reconstructed with the original data is different, in exchange for a large compression ratio. Lossless compression refers to compressed data is reconstructed, the reconstructed data identical with the original data. Here only LZW lossless compression and decompression algorithms.
The LZW compression algorithm is a common method of lossless compression, the compression algorithm is named after its inventors Lempel, Zif, Welch initials. LZW compression algorithm can also be implemented in software hardware; software compression is generally used in applications where time critical, while the running speed of the case of special requirements, real-time data compression generally required to implement in hardware. We mainly study to achieve the LZW algorithm (including compression and decompression), which uses hardware to achieve the LZW algorithm based on FPGA.
Principle 2 LZW compression
LZW algorithm uses an advanced string table (String Table). The first occurrence of each string in the string table, and with one code (Code Word) are represented. Only storing digital archive. So that the string can not store a file compression efficiency is greatly improved. Whether the compression or decompression process, the same sequence will be correctly established table, compression or decompression is completed, the string table will be discarded.
String table like a dictionary, the input string (word) into all the page number and the page number in the dictionary. It is decoded page number and page number found in the string (word). Use a dictionary basis with the right to restore. (LZW algorithm does not describe basic principles, you can refer to network resources).
3 LZW compression to achieve the difficulties
FPGA implementation of the LZW algorithm to solve the following five questions:
1) Select the dictionary capacity
When implementing the LZW compression and decompression algorithms need to build a dictionary, used to store the string table is formed during compression. LZW algorithm is adaptive dictionary generated, in practical applications, if the unlimited increase the capacity of the dictionary, although it may achieve better compression ratio, but when the time to find matching character string becomes long, and as codeword digit level of increase may sometimes result in reduced compression efficiency, impact compression rate, the dictionary capacity subject to certain restrictions.
2) a compression speed
by using the dictionary RAM constructed, since RAM lookup is accumulated according to the serial address lookup, to find a method which will affect the data output speed, eventually leading to the overflow of the input data. For example, a depth of the dictionary 512, the time required to find the range of l ~ 512 th cycle (if the N-th address is found, then requires N clock cycles).
3) the compression ratio
of the first set of dictionaries of different sizes may affect the compression ratio. When the dictionary is set relatively small when the string table will be filled in a very short period of time, resulting in the dictionary matching is not strong, so that an impact on the data compression, the compression rate is not high.
Second, the Dictionary Update Policy may affect the compression ratio. Due to the limited capacity of the dictionary, along with the process of compression, the dictionary will be filled, if not simple additions to the dictionary, then the compression ratio will be reduced back, and if the re-establishment dictionary dictionary clear, in the initial establishment of the dictionary compression ratio is very low.
4) resource utilization
establish dictionary using the FPGA chip resources to complete, although large-capacity dictionary will improve the compression ratio, but must take into account the resources (resource FPGA internal RAM) inside the FPGA.
5) The program robustness
FPGA module partition design process is very important, good module division can greatly reduce the area consumed by the logic and timing critical path optimization function. For more than logic design system clock of 200MHz, if only to consider the logical function, regardless of the final code to achieve a comprehensive and timing of the results of a large number of timing violations, we can say that the design is not reliable.
4 LZW compression scheme
1) the capacity of the dictionary should choose how much?
Usually string table 1024,2048,4096 bytes in length, which means that a string table can hold up to 1024,2048,4096 basic string.
2) How to improve the compression rate?
First, to increase the frequency of the system clock; a second, a byte clock period shorter compression required. To solve this problem, we will propose a query-based CAM implementation. With improvements in the structure and increase the number of FPGA devices doors and constantly enrich IP library, CAM FPGA-based implementation has become possible.
3) how to consider the compression ratio?
First, the capacity of the dictionary size settings. Second, the Dictionary Update Policy.
4) Consider how resource utilization?
First, the capacity of the dictionary size settings. Second, the dictionary maintenance algorithm design as simple as possible. Dictionary of maintenance problems, due to the dictionary maintenance must use the internal logic of the FPGA to complete, it is necessary to take into account, although overly complex dictionary management will improve the compression ratio, for the occupation of FPGA logic resources and real-time compression of time spent is worth the candle ; and
5) how to design robust program
first, consider the hardware architecture design; second, to view the timing of the report to address the critical path.
Considering the LZW compression rate, compression rate issues, resource utilization, using Xilinx FPGA to achieve LZW compression algorithm. Is constructed using the FPGA internal BRAM dictionary, the dictionary size is set to 2048, a dictionary lookup technique is a critical part of the LZW algorithm, the relationship between the efficiency of data once for each matching process. To improve the calculation speed dictionary address proposed (based on content addressable CAM, fast matching by a hardware circuit) implementation based on the CAM query.
5 LZW compression test and performance
LZW algorithm testing is divided into software testing, hardware testing, resource utilization, compression rate and compression rate. Software testing is functional simulation, logic functions to verify the correctness; hardware testing is code that runs in real FPGA chip, the logic analyzer to debug through online LZW algorithm validation logic correctness; resource utilization test the code refers to the occupation of FPGA synthesis or implementation internal resource usage.
1) software testing
using simulation software modelsim simulation logic functions. Write a test stimulus input to the image LZW data compression module, after the compressed data is then output through the LZW decompression module. The raw input data into a file 1.txt, LZW data into decompressed file 2.txt, compared, both files are files 171,520 bytes by comparing these two tools, differences in bytes 0, exact .
2) Testing of hardware
and software testing method also uses a consistent loopback test, the test is hardware original data is stored into the FPGA BRAM, after solution LZW LZW compression module and a compression module, the decompressed data to the original data by comparing the test code one by one , if the data is consistent, the LZW algorithm prove correct logic function; if inconsistent data demonstrate LZW algorithm logic function errors; hardware data consistent test results show, the LZW algorithm prove correct logic functions are 171520 bytes, 0 byte difference, exact .
3) resource utilization
on FPGA resource consumption LZW compression module as shown below.
A resource utilization FIG.
4) Compression
software testing, the file data is stored into the LZW compression 3.txt, statistics to 136,633.75 byte, compression rate of 79.68%.
5) The rate of compression of
the compressed single byte requires 15 clock cycles (T = 5ns), the work rate 12.72MB / s.
Conclusion 6 LZW algorithm
to achieve LZW compression algorithm based on xilinx FPGA. After the hardware simulation and verification results show FPGA implementation of this algorithm to obtain a compression ratio of 20%, the work rate 12.72MB / s (if required higher operating speed, can only improve the system clock, 250MHz clock, work rate 15.89MB / s), FPGA resource utilization is very low, fully meet the needs of real-time processing. Technical advice please email communication: [email protected].
Currently the LZW algorithm improvements, a single byte compression requires 10 clock cycles, if the period T = 5ns, the work rate 19.07MB / s; if the period T = 4ns, the work rate 23.84MB / s. Description: FPGA resource utilization relative increase in number.
