Table of contents
1 Introduction
Even if I have never played image stitching, I am embarrassed to say that I have played with FPGA. This is a sentence said by a CSDN boss, and I firmly believe it. . .
Image stitching is widely used in actual projects, especially in the medical and military industries. Currently, the image stitching solutions on the market mainly include the Video Mixer solution officially launched by Xilinx and the custom solution of hand-shredding the code; the Video Mixer solution officially launched by Xilinx Directly calling IP can be achieved through SDK configuration, but it is difficult to enable and has high requirements on FPGA resources. It is not suitable for small-scale FPGA. It is very used on zynq and K7 and above platforms. If you are interested in Video Mixer If you are interested in the solution, you can refer to my previous blog. Blog address:
Click to go directly
This article uses Xilinx's Kintex7 FPGA pure verilog code to implement 8-channel video image splicing. There are two video sources, which correspond to whether the developer has a camera. One is to use the cheap OV5640 camera module; if you don't have one, Camera, or if your development board does not have a camera interface, you can use the static color bar generated inside the code to simulate the camera video; the video source is selected through the `define macro definition at the top level of the code, and ov5640 is used as the video source by default; due to my hand There is only one camera in it, so after the FPGA collects the camera data, it directly copies multiple copies to simulate multiple camera inputs; I use my commonly used FDMA solution to achieve three-frame buffering of images. Different videos are cached at different addresses in DDR3, and read When playing a video, the video cache area is read out at one time to realize the video splicing function; the output video resolution is 1920x1080, which realizes 8-channel video splicing, so the resolution of each video channel is 480x540, so that the 8-channel video occupies the entire output. The screen looks more beautiful; after reading the video, use the pure verilog display HDMI output module to send it to the monitor for display;
This blog describes in detail the design scheme of FPGA pure verilog to realize video splicing. The engineering code can be comprehensively compiled and debugged on the board, and can be directly transplanted to the project. It is suitable for school students and graduate project development, and is also suitable for on-the-job engineers to learn and improve. It can be applied It is suitable for high-speed interfaces or image processing fields in medical, military and other industries;
it provides complete, run-through engineering source code and technical support;
the method of obtaining engineering source code and technical support is placed at the end of the article, please be patient until the end;
Version update instructions
This version is the second version. According to readers’ suggestions, the following improvements and updates have been made to the first version of the project:
1: Added the option of inputting video static color bars. Some readers said that they do not have the OV5640 camera or camera schematic diagram. It is inconsistent with mine, which makes the transplantation process very difficult. Based on this, a static color bar is added, which is generated internally by the FPGA and can be used without an external camera. The usage method is explained later; 2: Optimized
FDMA , the previous data read and write burst length of AXI4 in FDMA was 256, which resulted in insufficient bandwidth on low-end FPGA, resulting in poor image quality. Based on this, the data read and write burst length of AXI4 in FDMA was changed to 128;
3 : Optimized the HDMI output module. Some readers said that the custom IP used before could not be updated. Although it can be used normally, it is inconvenient to view the source code. Based on this, the HDMI output module was changed to a pure verilog implementation, which is straightforward;
Disclaimer
This project and its source code include both parts written by myself and parts obtained from public channels on the Internet (including CSDN, Xilinx official website, Altera official website, etc.). If you feel offended, please send a private message to criticize and educate; based on this, this project The project and its source code are limited to readers or fans for personal study and research, and are prohibited from being used for commercial purposes. If legal issues arise due to commercial use by readers or fans themselves, this blog and the blogger have nothing to do with it, so please use it with caution. . .
2. My existing FPGA video splicing and overlay fusion solution
My homepage currently has a FPGA video splicing, overlay and fusion column. The new column includes the FPGA video splicing, overlay and fusion solutions that I currently have. From the implementation method, there are video splicing based on HSL and video splicing based on pure verilog code. ; From the application point of view, it can be divided into single-channel, 2-channel, 3-channel, 4-channel, 8-channel, 16-channel video splicing; video scaling + splicing; video fusion overlay; from the input video classification, it can be divided into OV5640 camera video splicing and SDI video splicing , CameraLink video splicing, etc.; the following is the column address:
click to go directly
3. Design idea framework
This blog provides a set of vivado project source code. The project design block diagram is as follows:
Video source selection
There are two types of video sources, which correspond to whether the developer has a camera in his hand. If you have a camera in your hand, or your development board has a camera interface, use the camera as the video input source. The one I use here is the cheap one. OV5640 camera module; if you don’t have a camera, or your development board does not have a camera interface, you can use the static color bar generated inside the code to simulate camera video. The dynamic color bar is a moving picture, which can completely simulate video; the default Use ov5640 as the video source; the selection of the video source is carried out through the `define macro definition at the top level of the code; as follows: The
selection logic code part is as follows:
The selection logic is as follows:
When (note) define USE_SENSOR, the input source video is a static color bar;
when ( (No comment) When defining USE_SENSOR, the input source video is the ov5640 camera;
OV5640 camera configuration and collection
The OV5640 camera requires i2c configuration before it can be used. The video data from the DVP interface needs to be collected into RGB565 or RGB888 format video data. Both parts are implemented using the verilog code module. The code location is as follows: The camera is configured with a resolution of 480x540, as follows
:
Camera The acquisition module supports video output in RGB565 and RGB888 formats, which can be configured by parameters, as follows:
RGB_TYPE=0 outputs the RGB565 format;
RGB_TYPE=1 outputs the RGB888 format;
select the RGB565 format for design;
static color bar
The static color bar can be configured for videos of different resolutions. The border width of the video, the size of the dynamic moving block, the moving speed, etc. can all be parameterized. Here I configure the resolution as 480x540, the code location of the dynamic color bar module and the top-level interface. An example is as follows:
Video splicing algorithm
The video splicing scheme is as follows:
the output screen resolution is 1920X1080;
the input camera resolution is 480X540;
8 inputs can just fill the entire screen; the
multi-channel video splicing display principle is as follows:
output the two cameras CAM0 and CAM1 to the same Columns are represented on the monitor. In order to display two images on one monitor, you must first understand the following relationship:
hsize: The effective space actually occupied by each row of images in the memory. When one pixel is expressed in 32bit, the memory size occupied is hsize. 4;
hstride: used to set the address of the first pixel of each row of images. When a pixel is expressed in 32 bits, v_cnt hstride 4;
vsize: a valid row;
therefore it is easy to conclude that the address of the first pixel of each row of cam0 is also v_cnt hstride 4;
Similarly, if we need to display cam1 anywhere in the hsize and vsize spaces, we only need to care about the address of the first pixel of each row of cam1 images, and we can use the following formula v_cnt hstride*4+offset; uifdma_dbuf supports the stride
parameter Setting, the stride parameter can set the interval address from the first pixel to the next starting pixel of each row of data in the X (hsize) direction of the input data. The stride parameter can be used to easily arrange the arrangement of the input video into the memory.
Regarding uifdma_dbuf, you can refer to the article I wrote before and click to view: FDMA implements three-frame buffering of video data
Based on the above preparation, the base address of each camera cache is as follows:
CAM0: ADDR_BASE=0x80000000;
CAM1: ADDR_BASE=0x80000000+(1920-480X1)X4;
CAM2: ADDR_BASE=0x80000000+(1920-480X2)X4;
CAM3: ADDR_BASE=0x80 000000+(1920 -480X3)X4;
CAM4: ADDR_BASE=0x80000000+(1080-540)X1920X4;
CAM5: ADDR_BASE=0x80000000+(1080-540)X1920X4+(1920-480X1)X4;
CAM6: ADDR_BASE=0x8000000 0+(1080-540)X1920X4+(1920-480X2 )X4; CAM7: ADDR_BASE
=0x80000000+ (1080-540)
image cache
Old fans who often read my blog should know that my routine for image caching is FDMA. Its function is to send the image to DDR for 3-frame buffering and then read it out for display. The purpose is to match the input and output clock difference and improve the output. Video quality, regarding FDMA, please refer to my previous blog, blog address: Click to go directly
here. When 8-channel video is spliced, 8-channel FDMA is called for caching. Specifically, each channel of video calls 1 channel of FDMA;
8 channels of FDMA are called, among which 7 channels are configured in write mode, because these 7 channels of video only need to be written to DDR3 here, and the reading is completed by another FDMA. The configuration is as follows: The other 1
channel of FDMA is configured in read and write mode, because 8 channels of video need to be written at the same time. Read out, the configuration is as follows:
The key point of video splicing is the difference in the cache address of the 8-channel video in DDR3. The write address of the 8-channel FDMA is as follows: the
first video cache write base address: 0x80000000;
the second video cache write address Base address: 0x80000780;
No. 3 video cache write base address: 0x80000f00;
No. 4 video cache write base address: 0x80001680; No.
5 video cache write base address: 0x803f4800;
No. 6 video cache write base address: 0x803f4f80
; The write base address of the 7-channel video cache: 0x803f5700;
the write base address of the 8-channel video cache: 0x803f5e80;
the read base address of the video cache: 0x80000000;
video output
After the video is read from FDMA, it is output to the display after passing through the VGA timing module and HDMI sending module. The code location is as follows: The VGA timing
configuration is 1920X1080. The HDMI sending module is handwritten with verilog code and can be used for HDMI sending applications in FPGA. About this module, Please refer to my previous blog, blog address: click to go directly
4. Detailed explanation of vivado project
Development board FPGA model: Xilinx–Kintex7–xc7k325tffg676-2;
Development environment: Vivado2019.1;
Input: OV5640 camera or dynamic color bar, resolution 480x540;
Output: HDMI, 8 480x540 effective area display at 1080P resolution;
Engineering Function: FPGA pure verilog realizes 8-channel video splicing display;
the project BD is as follows:
Because 8-channel FDMA is used here, 7 channels are configured as write-only mode, and the other channel is configured as read-write mode; the
project code structure is as follows:
The resource consumption of the project and The power consumption is as follows:
5. Project transplantation instructions
Vivado version inconsistency handling
1: If your vivado version is consistent with the vivado version of this project, open the project directly;
2: If your vivado version is lower than the vivado version of this project, you need to open the project and click File –> Save As; but this method does not It is not safe. The safest way is to upgrade your vivado version to the vivado version of this project or a higher version;
3: If your vivado version is higher than the vivado version of this project, the solution is as follows:
After opening the project, you will find that the IP has been It is locked, as follows:
At this time, the IP needs to be upgraded, and the operation is as follows:
FPGA model inconsistency handling
If your FPGA model is inconsistent with mine, you need to change the FPGA model. The operation is as follows:
After changing the FPGA model, you also need to upgrade the IP. The method of upgrading the IP has been described previously;
Other things to note
1: Since the DDR of each board is not necessarily exactly the same, the MIG IP needs to be configured according to your own schematic. You can even directly delete the MIG of my original project and re-add the IP and reconfigure it; 2: According to your
own To modify the pin constraints of the schematic diagram, just modify it in the xdc file;
3: When transplanting pure FPGA to Zynq, you need to add the zynq soft core to the project;
6. Board debugging, verification and demonstration
static presentation
The output effect of 8-channel ov5640 camera 480x540 without splicing is as follows:
Dynamic presentation
The dynamic video demonstration is as follows:
FPGA-8-channel video splicing
7. Benefits: Obtain project source code
Bonus: Acquisition of engineering code.
The code is too large and cannot be sent by email. It will be sent via a certain network disk link.
The information acquisition method is: private, or the V business card at the end of the article.
The network disk information is as follows:
