Table of contents
1 Introduction
The most complex and technically difficult protocol in the field of FPGA image acquisition is probably the MIPI protocol. MIPI decoding is so difficult that it has defeated countless heroes, so much so that Xilinx officially had to launch a dedicated IP core for developers to use, otherwise it would be too high-end. This operation directly scares away a large number of FPGA developers, and no one will play with it.
This design is based on Xilinx's Zynq7020 development board. It collects 2Line MIPI videos from 4-channel OV5640 cameras. The OV5640 camera pins are connected to the BANK34 of Zynq7020. The D-PHY and CSI2 RX modules implemented in pure VHDL output Bayer videos, and then the Bayer converts them to RGB modules. Output the RGB video, then enhance the image quality through the gamma correction module, and then call 4 Xilinx official Video Scalers for image scaling, reducing the input 1280x720 video to 960x540; then call 4 Xilinx official VDMAs to send the image to the PS side. Three frames are buffered in DDR3 and read out; the Xilinx official VTC module is called to generate the output video timing. zynq dynamically configures the VTC output timing through the SDK software, supporting three timing outputs of 1080P, 720P and 640P, and calls the Xilinx official AXI4-Stream to Video Out performs data stream conversion, outputs VGA timing RGB data, and finally outputs the video to the display through the RGB to HDMI module implemented in pure VHDL;
Zynq7020 pure VHDL decoding 4-channel MIPI video, 4-channel image splicing output design scheme, the engineering code is compiled and debugged and verified on the board, and the project can be directly transplanted. It is suitable for school students to do graduation projects and graduate project development, and is also suitable for in-service applications Engineers do project development, which can be used in the fields of digital imaging and image transmission in medical, military and other industries;
provide complete, run-through engineering source code and technical support;
the method of obtaining engineering source code and technical support is at the end of the article, please be patient Read to the end;
about the MIPI protocol, please search it yourself, there are many big guys in csdn who talk about it in detail, so I won’t write more about it;
Disclaimer
This project and its source code include both parts written by myself and parts obtained from public channels on the Internet (including CSDN, Xilinx official website, Altera official website, etc.). If you feel offended, please send a private message to criticize and educate; based on this, this project The project and its source code are limited to readers or fans for personal study and research, and are prohibited from being used for commercial purposes. If legal issues arise due to commercial use by readers or fans themselves, this blog and the blogger have nothing to do with it, so please use it with caution. . .
2. The MIPI codec solution I already have here
I currently have a wealth of MIPI encoding and decoding solutions based on FPGA, mainly MIPI decoding. There are both MIPI decoding implemented by pure vhdl, and MIPI decoding implemented by calling Xilinx official IP. There are both 2line MIPI decoding and 4line. MIPI decoding, including MIPI decoding with 4K resolution and MIPI decoding with resolution as small as 720P, MIPI decoding based on Xilinx platform FPGA, MIPI decoding based on Altera platform FPGA, and MIPI decoding based on Lattice platform FPGA. In the future, we will continue to launch MIPI decoding solutions that are better than domestic FPGAs. After all, domestic solutions are currently the mainstream in the future. We will also launch more MIPI encoding DSI solutions in the future, and strive to make MIPI encoding and decoding solutions for FPGAs affordable. . .
Based on this, I created a special column for MIPI encoding and decoding, and put all MIPI encoding and decoding blogs into the column. Brothers who have project needs or learning interests in FPGA encoding and decoding MIPI can go to my column to have a look. The address is as follows:
Click to go directly to the column
3. Performance and advantages of this MIPI CSI2 module
In one word: awesome, the performance is as follows:
1: Pure VHDL code implementation, the learning and readability have reached the ceiling;
2: The portability is okay, as long as the FPGA is compatible with the Xilinx deserialization source language, it can be transplanted;
3: The algorithm has reached the ceiling, The standard CSI2 receiving protocol implements decoding;
4: The practicality reaches the ceiling, using the OV5640 camera as the input (mainly because it is cheap). Different from the verification and experimental projects on the market, this design is directly oriented to practical projects, close to real projects, and Brothers with similar projects can use it directly and get one month’s salary directly. . . 5: Supports MIPI video decoding with resolutions up to
1920
It will be lower, but users only need to know the user interface and do not need to look at the complex internal code;
7: Easy to use, although it is implemented in VHDL code, it has been encapsulated as a custom IP, and users do not need to care about the code implementation For complex logic, you only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still open it directly to watch; 8
: Collect 4 MIPI cameras with non-source clocks at the same time, and decode 4 channels Video splicing display is highly consistent with realistic high-end projects;
4. Detailed design plan
Design schematic diagram
The design principle block diagram is as follows:
OV5640 camera and its configuration
The OV5640 camera I use outputs 2 Line MIPI format. The output resolutions are 720p@60Hz and 1080p@30Hz. You can choose which one to configure through the SDK software; OV5640 outputs RAW10 data; zynq is configured through i2c on-chip peripherals. OV5640, this operation is completed in the SDK software, and the PL side hardware does not need to be involved;
D-PHY module
Regarding MIPI D-PHY, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI D-PHY function. In order to take care of everyone who is not used to reading VHDL code According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch; this MIPI D-PHY can only support 1 line or 2 line MIPI video; the
D-PHY custom IP call in this project is as follows:
The MIPI D-PHY pure VHDL source code is as follows:
CSI-2-RX module
Regarding MIPI CSI-2-RX, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI CSI-2-RX function. In order to take care of everyone To get used to the habit of reading VHDL code, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see it The source code can still be opened directly for viewing; this MIPI CSI-2-RX can only support 1 line or 2 line MIPI video, and the data format supports RAW10; the
CSI-2-RX custom IP call in this project is as follows:
MIPI CSI-2 -RX pure VHDL source code is as follows:
Bayer to RGB module
Regarding MIPI Bayer to RGB conversion, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here we focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI Bayer to RGB conversion function, in order to take care of everyone who is not used to reading VHDL code. According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch;
the Bayer to RGB custom IP call in this project is as follows:
MIPI Bayer to RGB pure VHDL source code is as follows:
Gamma correction module
Regarding MIPI gamma correction, there are a lot of articles on the Internet introducing the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI gamma correction function, in order to take care of everyone who is not used to reading VHDL code. According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch;
the gamma correction custom IP call in this project is as follows:
MIPI gamma correction pure VHDL source code is as follows:
VDMA image cache
Call 4-channel VDMA, 3 of which are only configured in write mode, and only need to write video to DDE3, and the other 1 is configured for read and write operations. The purpose of this design is to do 4-channel video splicing, and 4-channel VDMA writes the DDR of the image Different addresses can ensure that the image address in the memory does not conflict, but when reading the image, only one VDMA channel can be used to perform the read operation. This is not only a conventional design idea, but also a requirement to reduce the load of the HP interface AXI4 bus; it is configured as write-
only The VDMA in mode is as follows:
The VDMA configured in read and write mode is as follows:
Video Scaler image cache
Call 4 official Xilinx Video Scalers for image scaling and reduce the input 1280x720 video to 960x540; the purpose of this is to splice the 4-channel video to a display with an output resolution of 1920x1080; the Video Scaler scaling operation is completed in the SDK , the IP is as follows:
RGB to HDMI module
Regarding RGB to HDMI, there are a lot of articles on the Internet introducing the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the RGB to HDMI function, in order to take care of everyone who is not used to reading VHDL code. , we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still open it directly and watch it. ;
The RGB to HDMI pure VHDL source code in this project is as follows:
MIPI D-PHY hardware solution
Use a special MC20901 chip to convert to LVDS level and then send it to the FPGA;
this solution does not need to be introduced, a dedicated ASIC (the MC series solution can support a rate of more than 2Gbps/Lane, as long as the IO rate of the FPGA is sufficient), the solution It is not as good as the solution that comes with DPHY, but it is enough. Considering the cost of the solution, this solution is also the solution currently chosen by many companies. The block diagram is as follows: please contact us for detailed schematic diagram;
5. Detailed explanation of vivado project
PL side FPGA hardware design
Development board FPGA model: Xilinx–zynq7020–xc7z020clg484-1;
Development environment: Vivado2019.1;
Input: 4-channel OV5640 MIPI 2 Line RAW10;
Output: HDMI display, 1080P;
Application: Zynq7020 pure VHDL decoding 4-channel MIPI video, 4-channel Image stitching output;
the project Block Design is as follows:
the project code structure is as follows:
FPGA resource consumption and power consumption estimates after comprehensive compilation are as follows:
PS SDK software design
The SDK C++ language software code structure is as follows:
6. Project transplantation instructions
Vivado version inconsistency handling
1: If your vivado version is consistent with the vivado version of this project, open the project directly;
2: If your vivado version is lower than the vivado version of this project, you need to open the project and click File –> Save As; but this method does not It is not safe. The safest way is to upgrade your vivado version to the vivado version of this project or a higher version;
3: If your vivado version is higher than the vivado version of this project, the solution is as follows:
After opening the project, you will find that the IP has been It is locked, as follows:
At this time, the IP needs to be upgraded, and the operation is as follows:
FPGA model inconsistency handling
If your FPGA model is inconsistent with mine, you need to change the FPGA model. The operation is as follows:
After changing the FPGA model, you also need to upgrade the IP. The method of upgrading the IP has been described previously;
Other things to note
1: Since the DDR of each board is not necessarily exactly the same, the MIG IP needs to be configured according to your own schematic. You can even directly delete the MIG of my original project and re-add the IP and reconfigure it; 2: According to your
own To modify the pin constraints of the schematic diagram, just modify it in the xdc file;
3: When transplanting pure FPGA to Zynq, you need to add the zynq soft core to the project;
7. Board debugging and verification
The output is as follows:
8. Benefits: Obtaining engineering codes
Bonus: Acquisition of engineering code.
The code is too large to be sent by email. It will be sent via a certain network disk link.
The information acquisition method is: private, or the V business card at the end of the article.
The network disk information is as follows:
