Xilinx Artix7-100T low-end FPGA decodes MIPI video, implemented based on MIPI CSI-2 RX Subsystem architecture, providing engineering source code and technical support

Xilinx Artix7-100T low-end FPGA decodes MIPI video, implemented based on MIPI CSI-2 RX Subsystem architecture, providing engineering source code and technical support

1 Introduction

The most complex and technically difficult protocol in the field of FPGA image acquisition is probably the MIPI protocol. MIPI decoding is so difficult that it has defeated countless heroes, so much so that Xilinx officially had to launch a dedicated IP core for developers to use, otherwise it would be too high-end. This operation directly scares away a large number of FPGA developers, and no one will play with it.

This design is based on Xilinx's Artix7-100T low-end FPGA development board. It collects 2Line MIPI video from the OV5640 camera. The OV5640 camera is configured in MIPI mode. The pins are connected to the LVDS differential IO of the HS BANK of the FPGA after passing through the weight resistor scheme; calling Xilinx's MIPI CSI-2 RX Subsystem IP implements MIPI's D-PHY function. This IP is provided free of charge by Xilinx. It decodes the MIPI video and outputs it in AXIS video stream format; then calls Xilinx's Sensor Demosaic IP to implement the RAM to RGB function; then calls Xilinx Gammer LUT IP implements the gamma correction function; then calls Xilinx's VDMA IP to implement the image three-frame cache function. VDMA calls two, one is used for video writing to DDR3, and the other is used for video reading DDR3, so that reading and writing are separated The purpose is to use the bandwidth of AXI; then call Xilinx's Video Timing Controller and AXI4-Stream toVideo Out IP to convert the video stream from AXI4-Stream to VGA timing; finally use a custom HDMI sending IP to display the video output monitor; the entire project Calling MicroBlaze soft core for IP configuration is equivalent to the role of Zynq. MicroBlaze configuration is run in the form of C language software code in Vitis SDK, so the entire project includes two parts: FPGA logic design and Vitis SDK software design, which requires FPGA and The comprehensive capabilities of embedded C language are not suitable for beginners or novices. . .

This article describes in detail the design scheme of Xilinx Artix7-100T low-end FPGA to decode MIPI video. The engineering code is compiled and verified by on-board debugging. It can be directly transplanted into the project. It is suitable for school students to do graduation projects and graduate project development. , also suitable for on-the-job engineers to do project development, and can be used in the fields of digital imaging and image transmission in medical, military and other industries;
Provide complete, run-through engineering source code and technical support; a> Regarding the MIPI protocol, please search by yourself, there are many big names in csdn It’s very detailed, so I won’t write more about it;
How to obtain the project source code and technical support is at the end of the article, please be patient until the end;

Disclaimer

This project and its source code include both parts written by myself and parts obtained from public channels on the Internet (including CSDN, Xilinx official website, Altera official website, etc.). If you feel offended, please send a private message to criticize and educate; based on this, this project The project and its source code are limited to readers or fans for personal study and research, and are prohibited from being used for commercial purposes. If legal issues arise due to commercial use by readers or fans themselves, this blog and the blogger have nothing to do with it, so please use it with caution. . .

2. The MIPI codec scheme I already have here

I currently have a wealth of MIPI codec solutions based on FPGA, mainly MIPI decoding, including MIPI decoding implemented by pure vhdl, MIPI decoding implemented by calling Xilinx official IP, and 2line MIPI Decoding, there is also 4line MIPI decoding, including MIPI decoding with 4K resolution and MIPI decoding with resolution as small as 720P. There are MIPI decoding based on Xilinx platform FPGA, MIPI decoding based on Altera platform FPGA, and Lattice platform. For FPGA MIPI decoding, we will continue to launch MIPI decoding solutions that are better than domestic FPGAs. After all, domestic solutions are currently the mainstream in the future. We will also launch more MIPI encoding DSI solutions in the future, and strive to make FPGA’s MIPI encoding and decoding solutions better. The price is cabbage. . .
Based on this, I created a column specifically for MIPI encoding and decoding, and put all the MIPI encoding and decoding blogs into the column for those who have project needs or are interested in learning about FPGA encoding and decoding MIPI. You can go to my column to have a look. The column address is as follows:
Click to go directly to the column

3. Performance of this MIPI CSI2 module and its advantages and disadvantages

Since Xilinx’s MIPI CSI-2 RX Subsystem IP core is called, the performance depends on your FPGA model. In theory, the higher the FPGA, the higher the IO line rate or GT high-speed interface line rate it supports. , you can run higher-speed MIPI video;
This IP is extremely adaptable, and the supported MIPI camera performance parameters are as follows:


And, in the higher-end On FPGA models, the IP supports more high-end performance;
Due to the use of Xilinx’s MIPI CSI-2 RX Subsystem IP core, NIPI video decoding is very stable and easy to use. It is simple. The disadvantage is that you cannot see the source code, and it is difficult to troubleshoot if there is a problem. You can only add ila to the input and output interfaces for step-by-step tracing;
The FPGA model used in this solution is Xilinx Artix7-100T , which is a MIPI decoding application for low-end FPGAs. Even the smallest Artix7-35T or Spartan7, Spartan6, etc. can no longer use the MIPI CSI-2 RX Subsystem;

4. Detailed design plan

This design is based on Xilinx's Artix7-100T low-end FPGA development board. It collects 2Line MIPI video from the OV5640 camera. The IOV5640 camera is configured in MIPI mode. The pins are connected to the LVDS differential IO of the HS BANK of the FPGA after passing through the weight resistor scheme; calling Xilinx's MIPI CSI-2 RX Subsystem IP implements MIPI's D-PHY function. This IP is provided free of charge by Xilinx. It decodes the MIPI video and outputs it in AXIS video stream format; then calls Xilinx's Sensor Demosaic IP to implement the RAM to RGB function; then calls Xilinx Gammer LUT IP implements the gamma correction function; then calls Xilinx's VDMA IP to implement the image three-frame cache function. VDMA calls two, one is used for video writing to DDR3, and the other is used for video reading DDR3, so that reading and writing are separated The purpose is to use the bandwidth of AXI; then call Xilinx's Video Timing Controller and AXI4-Stream toVideo Out IP to convert the video stream from AXI4-Stream to VGA timing; finally, use a custom HDMI sending IP to display the video output monitor;

Design schematic diagram

The design principle block diagram is as follows:

OV5640 and its configuration

The input video uses a cheap OV5640 camera module, configured in MIPI mode, 2 Line, data format is RAW10, line rate is 1000Mbps, video resolution is 1280*720, one clock per pixel, OV5640 requires SCCB bus configuration to run. The bus is equivalent to the I2C bus. An AXI-GPIO is called to simulate I2C. The Vitis software is used to configure the OV5640. Part of the configuration code is implemented in C language. Please refer to the Vitis program for details. The screenshots of the AXI-GPIO call and SCCB software configuration code are as follows:

Weight resistor hardware solution

Use the weighted resistor hardware solution officially recommended by Xilinx to restore the HS and PL of the input differential MIPI pair. The screenshot of the schematic diagram is as follows:

Note: The weighted resistor solution is only used at low speeds. It is available in MIPI mode. Please use a dedicated chip to implement high-speed MIPI, such as MC20001. MC series solutions can support rates above 2Gbps/Lane, as long as the IO rate of the FPGA is sufficient;

MIPI CSI-2 RX Subsystem

Call Xilinx's MIPI CSI-2 RX Subsystem IP to implement MIPI's D-PHY function. This IP is provided free of charge by Xilinx. It decodes the MIPI video and outputs it in AXIS video stream format; the call and configuration are as follows:

Sensor Demosaic image format conversion

Call Xilinx's Sensor Demosaic IP to implement the RAM to RGB function. The IP is configured through Vitis's C code software. The screenshots of the Sensor Demosaic calling and C code software configuration codes are as follows:

Gammer LUT gamma correction

The Gammer LUT IP of Xilinx is called to implement the gamma correction function. The IP is configured through Vitis's C code software. The screenshots of the Gammer LUT calling and C code software configuration code are as follows:

VDMA image cache

The VDMA IP of Xilinx is called to implement the three-frame cache function of the image. Two VDMA calls are used, one is used for video writing to DDR3, and the other is used for video reading DDR3. The purpose of separate reading and writing is to use the bandwidth of AXI; this IP passes Vitis C code software configuration; VDMA call and C code software configuration code screenshots are as follows:

AXI4-Stream toVideo Out

Then call Xilinx's Video Timing Controller and AXI4-Stream to Video Out IP to convert the video stream from AXI4-Stream to VGA timing; the Video Timing Controller is configured as 1280*720@60Hz. These two IPs do not require software configuration; Video Timing Controller And the screenshot of the AXI4-Stream toVideo Out call is as follows:

HDMI output

Finally, a custom HDMI sending IP is used to display the video output on the monitor. The maximum output resolution of this IP only supports 1920*1080@60Hz; the screenshot of the IP call is as follows:

5. Detailed explanation of vivado project

FPGA logic design

Development board FPGA model: Xilinx–Artix7–xc7a100tfgg484-2;
Development environment: Vivado2021.1;
Input: OV5640 camera–MIPI– 2 Line–RAW10-1280*720;
Output: HDMI, resolution 720P;
Application: Xilinx Artix7-100T low-end FPGA decoding MIPI video; a> The resource consumption and power consumption of the project are as follows: The project code structure is as follows: After expansion, it is as follows: Since there are many calling IPs, it is unsightly to spread them out and is not conducive to reading. We have changed the core IP of MIPI decoding and caching. Then encapsulate it, as shown in the red box mark in the picture above. The package is only beautiful in form and has no special functions. You can expand it as follows to view the detailed composition and structure inside, as follows:
The project Block Design is as follows:

Vitis SDK software design

Vitis SDK C language software is used to configure the IP called by the FPGA. You can view it by opening it with VItis. The code content is as follows:

Vitis HLS patch

Since the IP generated by HLS, Sensor Demosaic and Gammer LUT are used in the project, comprehensive compilation may fail, or the device ID cannot be found in the Vitis SDK after warning. In this case, the computer system time needs to be changed. Or apply an official patch to solve this problem. For specific methods, please refer to this boss’s blog post:Click directly to go

6. Project transplantation instructions

Vivado version inconsistency handling

1: If your vivado version is consistent with the vivado version of this project, open the project directly;
2: If your vivado version is lower than the vivado version of this project, you need to open it After the project, click File –> Save As; however, this method is not safe. The safest way is to upgrade your vivado version to the vivado version of this project or a higher version;

3 : If your vivado version is higher than the vivado version of this project, the solution is as follows:

After opening the project, you will find that the IPs are locked, as follows:

At this time It is necessary to upgrade the IP, please do as follows:

FPGA model inconsistency handling

If your FPGA model is inconsistent with mine, you need to change the FPGA model. The operation is as follows:



After changing the FPGA model, you also need to upgrade the IP. The method of upgrading the IP has been described previously. ;

Other things to note

1: Since the DDR of each board is not necessarily exactly the same, the MIG IP needs to be configured according to your own schematic diagram. You can even directly delete the MIG of my original project here and re-add the IP and reconfigure it;
2: Modify the pin constraints according to your own schematic diagram, just modify it in the xdc file;
3: Transplanting pure FPGA to Zynq needs to be done in the project Add zynq soft core;

7. Board debugging and verification

The equipment that needs to be prepared is as follows:
FPHA development board;
OV5640 camera;
HDMI monitor; a>
The output is as follows:

8. Benefits: Obtaining engineering codes

Benefit: Acquisition of engineering code
The code is too large to be sent by email. It will be sent via a certain network disk link.
How to obtain data: Private, or the V business card at the end of the article.
The network disk information is as follows: