Zynq7020 pure VHDL decodes MIPI video, HDMI output, supports ISP dynamic configuration, and provides vivado project source code and technical support

1 Introduction

The most complex and technically difficult protocol in the field of FPGA image acquisition is probably the MIPI protocol. MIPI decoding is so difficult that it has defeated countless heroes, so much so that Xilinx officially had to launch a dedicated IP core for developers to use, otherwise it would be too high-end. This operation directly scares away a large number of FPGA developers, and no one will play with it.

This design is based on Xilinx's Zynq7020 development board. It collects 2Line MIPI video from the OV5640 camera. The OV5640 camera pin is connected to the BANK35 of the Zynq7020. The D-PHY and CSI2 RX modules implemented in pure VHDL output Bayer video, and then the Bayer to RGB module outputs RGB. The video is then enhanced by the gamma correction module to enhance the image quality, and then the Xilinx official VDMA is called to send the image to DDR3 on the PS side for three-frame buffering and then read out; the Xilinx official VTC module is called to generate the output video timing, and zynq dynamically configures the VTC through the SDK software. The output timing supports three timing outputs: 1080P, 720P and 640P. It calls Xilinx official AXI4-Stream to Video Out for data stream conversion, outputs VGA timing RGB data, and finally outputs the video to the display through the RGB to HDMI module implemented in pure VHDL. ; On the SDK software side, interact with the PC through the serial port, and the PC sends instructions to the SDK software to dynamically configure the ISP. The ISP functions include controlling the OV5640 input resolution, controlling the OV5640 input image format, arbitrarily modifying the OV5640 internal register value, and arbitrarily reading the OV5640 Internal register value, arbitrarily modify the gamma correction coefficient (different output effects), control and switch the automatic white balance mode (there are three modes: advanced, simple and disabled);

This article describes in detail the design scheme of Zynq7020 pure VHDL decoding MIPI video HDMI output. After the engineering code is compiled and passed the on-board debugging and verification, the project can be directly transplanted. It is suitable for school students to do graduation projects and graduate project development, and is also suitable for in-service engineers to do Project development can be used in the fields of digital imaging and image transmission in medical, military and other industries;
it provides complete, run-through engineering source code and technical support;
the method of obtaining the engineering source code and technical support is placed at the end of the article, please be patient and read Finally;
regarding the MIPI protocol, please search by yourself. There are many big guys in csdn who talk about it in detail, so I won’t write more about it;

disclaimer

This project and its source code include both parts written by myself and parts obtained from public channels on the Internet (including CSDN, Xilinx official website, Altera official website, etc.). If you feel offended, please send a private message to criticize and educate; based on this, this project The project and its source code are limited to readers or fans for personal study and research, and are prohibited from being used for commercial purposes. If legal issues arise due to commercial use by readers or fans themselves, this blog and the blogger have nothing to do with it, so please use it with caution. . .

2. The MIPI codec solution I already have here

I currently have a wealth of MIPI encoding and decoding solutions based on FPGA, mainly MIPI decoding. There are both MIPI decoding implemented by pure vhdl, and MIPI decoding implemented by calling Xilinx official IP. There are both 2line MIPI decoding and 4line. MIPI decoding, including MIPI decoding with 4K resolution and MIPI decoding with resolution as small as 720P, MIPI decoding based on Xilinx platform FPGA, MIPI decoding based on Altera platform FPGA, and MIPI decoding based on Lattice platform FPGA. In the future, we will continue to launch MIPI decoding solutions that are better than domestic FPGAs. After all, domestic solutions are currently the mainstream in the future. We will also launch more MIPI encoding DSI solutions in the future, and strive to make MIPI encoding and decoding solutions for FPGAs affordable. . .
Based on this, I created a special column for MIPI encoding and decoding, and put all MIPI encoding and decoding blogs into the column. Brothers who have project needs or learning interests in FPGA encoding and decoding MIPI can go to my column to have a look. The address is as follows:
Click to go directly to the column

3. Performance and advantages of this MIPI CSI2 module

In one word: awesome, the performance is as follows:
1: Pure VHDL code implementation, the learning and readability have reached the ceiling;
2: The portability is okay, as long as the FPGA is compatible with the Xilinx deserialization source language, it can be transplanted;
3: The algorithm has reached the ceiling, The standard CSI2 receiving protocol implements decoding;
4: The practicality reaches the ceiling, using the OV5640 camera as the input (mainly cheap). Different from the verification and experimental projects on the market, this design is directly oriented to practical projects, close to real projects, and Brothers with similar projects can use it directly and get one month’s salary directly. . . 5: Supports MIPI video decoding with resolutions up to
1920
It will be lower, but users only need to know the user interface and do not need to look at the complex internal code;
7: Easy to use, although it is implemented in VHDL code, it has been encapsulated as a custom IP, and users do not need to care about the code implementation For complex logic, you only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still open it directly and watch it;

4. Detailed design plan

Design schematic diagram

The design principle block diagram is as follows:

OV5640 camera and its configuration

The OV5640 camera I use outputs 2 Line MIPI format. The output resolutions are 720p@60Hz and 1080p@30Hz. You can choose which one to configure through the SDK software; OV5640 outputs RAW10 data; zynq is configured through i2c on-chip peripherals. OV5640, this operation is completed in the SDK software, and the PL side hardware does not need to be involved;

D-PHY module

Regarding MIPI D-PHY, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI D-PHY function. In order to take care of everyone who is not used to reading VHDL code According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch; this MIPI D-PHY can only support 1 line or 2 line MIPI video; the
D-PHY custom IP call in this project is as follows:


The MIPI D-PHY pure VHDL source code is as follows:

CSI-2-RX module

Regarding MIPI CSI-2-RX, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI CSI-2-RX function. In order to take care of everyone To get used to the habit of reading VHDL code, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see it The source code can still be opened directly for viewing; this MIPI CSI-2-RX can only support 1 line or 2 line MIPI video, and the data format supports RAW10; the
CSI-2-RX custom IP call in this project is as follows:

MIPI CSI-2 -RX pure VHDL source code is as follows:

Bayer to RGB module

Regarding MIPI Bayer to RGB conversion, there are a lot of articles on the Internet that introduce the principles and concepts. I will not repeat them here. Here we focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI Bayer to RGB conversion function, in order to take care of everyone who is not used to reading VHDL code. According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch;
the Bayer to RGB custom IP call in this project is as follows:

MIPI Bayer to RGB pure VHDL source code is as follows:

Gamma correction module

Regarding MIPI gamma correction, there are a lot of articles on the Internet introducing the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the MIPI gamma correction function, in order to take care of everyone who is not used to reading VHDL code. According to the custom, we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still directly Open and watch;
the gamma correction custom IP call in this project is as follows:

MIPI gamma correction pure VHDL source code is as follows:

RGB to HDMI module

Regarding RGB to HDMI, there are a lot of articles on the Internet introducing the principles and concepts. I will not repeat them here. Here I focus on the implementation using FPGA;
we use pure VHDL code to implement the RGB to HDMI function, in order to take care of everyone who is not used to reading VHDL code. , we have encapsulated some of the code into custom IP. Users do not need to care about the complex logic of code implementation. They only need to call the IP and configure it through the UI interface to use it. Of course, if you want to see the source code inside, you can still open it directly and watch it. ;
The RGB to HDMI pure VHDL source code in this project is as follows:

ISP dynamic configuration

On the SDK software side, it interacts with the PC through the serial port, and the PC sends instructions to the SDK software to dynamically configure the ISP. The ISP functions include controlling the OV5640 input resolution, controlling the OV5640 input image format, arbitrarily modifying the OV5640 internal register values, and arbitrarily reading the OV5640 internal Register value, arbitrarily modify the gamma correction coefficient (different output effects), control and switch the automatic white balance mode (there are three modes: advanced, simple and disabled); the details are as follows: After downloading the bit, the serial port debugging assistant on the PC will receive a
printout prompt information;

Please press the key corresponding to the desired option:
a. Change Resolution
b. Change Liquid Lens Focus
d. Change Image Format (Raw or RGB)
e. Write a Register Inside the Image Sensor
f. Read a Register Inside the Image Sensor
g. Change Gamma Correction Factor Value
h. Change AWB Settings

At this time, you can enter a, b, d, e, f, g, h cases to dynamically configure the camera; the instructions here are as follows:

OV5640 input resolution control

When you enter a, you will see the following printed information:

1. 1280 x 720, 60fps
2. 1920 x 1080, 15fps
3. 1920 x 1080, 30fps

Input 1, the input resolution of OV5640 is 1280 x 720, 60fps;
input 2, the input resolution of OV5640 is 1920 x 1080, 15fps;
input 3, the input resolution of OV5640 is 1920 x 1080, 30fps;

OV5640 focus control

That is the function when b is entered above, but I have been failing to debug this function. I don’t know where the problem lies yet, so I skip it and won’t go into details;

OV5640 input data format control

When you enter d, you will see the following printed information:

Please press the key corresponding to the desired setting:
1. Select image format to be RGB, output still Raw
2. Select image format & output to both be Raw

Input 1, the input data format of OV5640 is RGB;
input 2, the input data format of OV5640 is RAW;
Note: RGB format is only used in debugging, because this project is MIPI, so be sure to configure it as RAW;

OV5640 write register control

When you enter e, you will see the following printed information:

Please enter address of image sensor register, in hex, with small letters:

At this time, you need to enter the OV5640 register address that needs to be written. Note that you enter the hexadecimal number.
After entering the address, you will see the following printed information:

Please enter value of image sensor register, in hex, with small letters:

At this time, you need to enter the OV5640 register data that needs to be written. Note that you enter the hexadecimal number;
from then on, the OV5640 register writing operation is completed;

OV5640 read register control

When you enter f, you will see the following printed information:

Please enter address of image sensor register, in hex, with small letters:

At this time, you need to enter the OV5640 register address that needs to be read. Note that you enter the hexadecimal number;
after entering the address, you will see the data read back;

Gamma correction coefficient control

When you enter g, you will see the following printed information:

Please press the key corresponding to the desired Gamma factor:
1. Gamma Factor = 1
2. Gamma Factor = 1/1.2
3. Gamma Factor = 1/1.5
4. Gamma Factor = 1/1.8
5. Gamma Factor = 1/2.2

Enter 1, the gamma correction coefficient is 1;
enter 2, the gamma correction coefficient is 1/1.2;
enter 3, the gamma correction coefficient is 1/1.5; enter 4, the gamma correction coefficient is 1/1.8; enter 5,
the gamma correction coefficient is 1/ 1.8;
The horse correction coefficient is 1/2.2;

Automatic white balance control

When you enter h, you will see the following printed information:

Please press the key corresponding to the desired AWB change:
1. Enable Advanced AWB
2. Enable Simple AWB
3. Disable AWB

Enter 1, the automatic white balance is in advanced mode, and the effect is best;
input 2, the automatic white balance is in simple mode, and the effect is average;
enter 3, the automatic white balance function is disabled, and the output may be black;

MIPI D-PHY hardware solution

This design uses the resistor network solution officially recommended by xilinx, which requires 800M Hz or less and wiring within 30mm; the onboard MIPI interface schematic is as follows: Then the



differential signal group is connected to BANK35 as follows:

5. Detailed explanation of vivado project

PL side FPGA hardware design

Development board FPGA model: Xilinx–zynq7020–xc7z020clg400-1;
Development environment: Vivado2019.1;
Input: OV5640 MIPI 2 Line RAW10;
Output: HDMI display, 720P;
Application: Zynq7020 pure VHDL decoding MIPI video HDMI output;
Project Block Design is as follows :

The engineering code structure is as follows:

The FPGA resource consumption and power consumption estimates after comprehensive compilation are as follows:

PS SDK software design

The SDK C++ language software code structure is as follows:

6. Project transplantation instructions

Vivado version inconsistency handling

1: If your vivado version is consistent with the vivado version of this project, open the project directly;
2: If your vivado version is lower than the vivado version of this project, you need to open the project and click File –> Save As; but this method does not It is not safe. The safest way is to upgrade your vivado version to the vivado version of this project or a higher version;

3: If your vivado version is higher than the vivado version of this project, the solution is as follows:

After opening the project, you will find that the IP has been It is locked, as follows:

At this time, the IP needs to be upgraded, and the operation is as follows:

FPGA model inconsistency handling

If your FPGA model is inconsistent with mine, you need to change the FPGA model. The operation is as follows:



After changing the FPGA model, you also need to upgrade the IP. The method of upgrading the IP has been described previously;

Other considerations

1: Since the DDR of each board is not necessarily exactly the same, the MIG IP needs to be configured according to your own schematic. You can even directly delete the MIG of my original project and re-add the IP and reconfigure it; 2: According to your
own To modify the pin constraints of the schematic diagram, just modify it in the xdc file;
3: When transplanting pure FPGA to Zynq, you need to add the zynq soft core to the project;

7. Board debugging and verification

The output is as follows:

8. Benefits: Obtaining engineering codes

Bonus: Acquisition of engineering code.
The code is too large to be sent by email. It will be sent via a certain network disk link.
The information acquisition method is: private, or the V business card at the end of the article.
The network disk information is as follows: