FPGA is used in image processing
FPGA (Field-Programmable Gate Array) literally translates as field programmable gate array. It is a programmable logic device with a high degree of flexibility and can be programmed to implement different functions according to specific needs.
FPGA devices are a kind of semi-customized circuits in dedicated integrated circuits. They are programmable logic arrays with basic logic units, configurable logic blocks, digital clock management modules, embedded RAM, and rich wiring resources. Some FPGAs also integrate SDRAM and Cortex-ARM hard cores. Use Verilog or VHDL for programming.
Since everyone can use FPGA, it means that everyone has taken digital circuit experimental classes in school. Generally speaking, you can intuitively think that the various logic blocks inside the FPGA are like various chip modules in the laboratory. Using Verilog programming to route logic resources is equivalent to facing a lot of electronic components. Use a breadboard and Dupont wire to connect the functions you need. How to connect and the way to connect the lines is defined by you using verilog. Is not it simple?
Advantages and Disadvantages:
First, let’s talk about the advantages and disadvantages of FPGA.
advantage:
FPGA is composed of hardware resources such as logic units, RAM, and multipliers. Through the combination of these resources, hardware circuits such as multiplication, registers, and address generators can be implemented;
FPGA can be designed by using block diagrams or Verilog HDL, and everything from simple gate circuits to complex FIR circuits can be implemented;
FPGA can be infinitely reprogrammed, loading a new design only takes a few hundred milliseconds, and the use of reconfiguration circuits can reduce hardware overhead;
The operating frequency of FPGA is determined by the FPGA chip and design. Certain demanding working conditions can be achieved by modifying the design or replacing a faster FPGA chip.
shortcoming:
All functions of FPGA are implemented in hardware, and operations such as branch conditional jumps cannot be implemented;
FPGA can only implement fixed-point operations;
summary:
FPGA relies on hardware to implement all functions, and its speed can be compared with special-purpose chips, but its design flexibility is still somewhat insufficient compared with general-purpose processors.
Industries where FPGA can be applied:
video processing;
Image segmentation;
Data delay and storage design;
Communications industry;
security;
Industrial control industry;
AI: (also the hottest industry at the moment);
####The following is a brief introduction to the image processing industry that I am relatively familiar with, and a brief talk about my own views;
The first thing I learned about was the 51 microcontroller, but my actual work was in the application development of image recognition. Due to the congenital nature of the 51 microcontroller, it was rarely used in actual work. At first, I used ADI's Blackfin series of DSPs. As image resolution increases and the complexity of processing algorithms increases, the Blackfin processor industry is gradually unable to meet actual needs; so it turns to FPGA, and it has been used on FPGA for more than ten years.
One of the advantages of FPGA is pipeline operation. Although the main frequency is much lower than that of CPU, GPU, NPU, etc., the pipeline method can be used for simultaneous processing, and good real-time performance can be achieved. Therefore, in some application fields with very high real-time requirements, only FPGA can be used for image processing. In the past few years, I have been engaged in image processing of sorting equipment. Currently, in the field of sorting, FPGA is basically used as the main processor. Whether it is a linear array sensor or an area array sensor, the FPGA starts from collecting image data to The next time new image data is collected, this delay time is fixed at the interval of the line rate or frame rate. This advantage cannot be satisfied by the CPU. Only the real-time pipeline structure of FPGA can meet the real-time processing of large amounts of data.
The traditional method of using CPU, DSP, etc. to perform image operations is that first the CPU or DSP transfers the collected data to the memory through DMA or other bus methods, and then the CPU or DSP computing part reads the data from the memory for processing. Processing, after the processing is completed, the processing results are saved to the memory for access by other functions; if the processing is performed according to the line frequency format of the line array sensor, and calculated according to the line frequency of 10K, the actual interval of each line of data is 0.1ms, This means that the data processing must be completed within 0.1ms and the processing results must be saved to the memory. If the main frequency of the CPU or DSP is high enough, it is theoretically feasible, but the main frequency cannot be increased infinitely.
If you use FPGA for processing, you can perfectly avoid frequent memory access; because FPGA processes images in real-time pipeline operations, the internal logic resources of FPGA can build several processing units, and each processing unit works at the same time. Yes, when there are enough processing units, although the main frequency of FPGA is low, the ability to process data at the same time will be very large.
The above picture is a simple pipeline diagram. In the first line, data A comes in, and the first unit processes data A. By the second line, data A has entered the next processing unit, and data B comes in at the same time. This Data A and data B belong to different processing units, so data A and data B can be processed at the same time; when there are enough processing units, a lot of data can be processed at the same time, and the output of data A is only a fixed delay Number of clocks for several processing units. As long as the number of processing units is fixed, the delay interval of this output data is also fixed.
For example, in practical applications, the FPGA is directly connected to the Sensor, and then drives the Sensor to directly obtain image data. The first unit can convert the obtained image data, such as Bayer format to RGB, RAW to RGB, etc.; the second processing unit can Perform difference operations and sub-pixel correction on RGB data, the third unit performs erosion/expansion operations, and the fourth unit performs canny operator detection. . . . . . .
This kind of pipeline processing method is processed sequentially according to the flow direction of the data stream output by the Sensor. It cannot cache the entire image for out-of-order operations like the CPU or DSP. The FPGA processes the image data equivalent to the completion of each unit's processing. The processing results are handed over to the next processing unit. Each processing unit is only responsible for the currently incoming data and no longer saves the intermediate result data;
As the demand for computing power for massive data becomes higher and higher, FPGA chips will continue to develop towards higher density and higher communication bandwidth. In addition, heterogeneous computing fusion and other forms will become increasingly popular. And from the perspective of software development, it also pays more attention to supporting tool capabilities to provide high-performance AI acceleration capabilities.
The key value of FPGA lies in programmability and flexibility. In the early stage of AI development, many emerging applications are still in the pipeline, various standards are still evolving, and AI algorithms are constantly being introduced, which requires high computing power and flexibility of hardware. FPGA just meets these needs. In addition, in the small-batch data processing in the inference process, FPGA can achieve high parallelism and low latency by relying on pipeline parallelism. Moreover, driven by factors such as low latency, data privacy, and bandwidth constraints, FPGAs can meet the needs for low power consumption, flexibility, and fast response.