Table of contents
1 Introduction
In the field of FPGA image acquisition, the most complicated protocol and the most technically difficult protocol should be the MIPI protocol. The difficulty of MIPI decoding has made countless heroes compete, so that Xilinx officially has to release a dedicated IP core for developers to use, otherwise it is too high-end The operation directly scares away a large number of FPGA developers, and no one will play.
This design is based on Lattice's LCMXO3LF-6900C-5BG256C development board, which collects 1080P 4Line MIPI video from the IMX219 camera. The IMX219 camera is a special camera adapter board. The schematic diagram and PCB source files of the adapter board are provided. You can use the Altium Designer series When the software is opened, it can even be mass-produced directly on the board. Lattice FPGA has its own MIPI decoding source language, which can be directly called to complete MIPI decoding. This is also the unique advantage of Lattice FPGA in MIPI video encoding and decoding. After MIPI video enters FPGA, after D-PHY deserialization, Byte alignment, Line alignment, MIPI CSI packet decoding, RAW10 to Bayer, Bayer to RGB, RGB to YUV, video output correction, the final video data is sent to Cypress FX3 USB3.0 interface output; use Cypress The official USB3.0 camera host computer receives and displays the video; the Lattice FPGA development board provides the schematic diagram and PCB source file of the adapter board, which can be opened with Altium Designer series software, and can even be mass-produced directly on the board;
This article describes the design scheme in detail. After the engineering code is compiled and passed, it can be debugged and verified on the board, and the project can be directly transplanted. It is suitable for students in school to do graduation design, graduate project development, and also for in-service engineers to do project development. It can be applied to medical and military industries. Digital imaging and image transmission fields in other industries;
provide complete and smooth engineering source code and technical support;
the method of obtaining engineering source code and technical support is at the end of the article, please be patient to the end;
about the MIPI protocol, please search by yourself , there are many big guys on csdn who talk about it in detail, so I won’t write more about it;
2. The performance and advantages of Lattice FPGA decoding MIPI
One word: Awesome, the performance is as follows:
1: Lattice FPGA decodes MIPI, the FPGA has its own MIPI decoding source language, and the MIPI decoding can be completed by direct calling, which is also the unique advantage of Lattice FPGA in MIPI video encoding and decoding, and the FPGAs are relatively small, with few users and developers, and fewer project source codes;
2: The portability is not bad, and it can be ported between Lattice series FPGAs;
3: The practicability reaches the ceiling, and it can be used as a batch of USB cameras , and I provide not only the source code but also the schematic diagram and PCB;
4: Support MIPI video decoding up to 4K resolution;
3. The MIPI codec scheme I have here
I currently have a wealth of FPGA-based MIPI encoding and decoding solutions, mainly MIPI decoding, including MIPI decoding implemented by pure vhdl, and MIPI decoding implemented by calling Xilinx official IP, including 2-line MIPI decoding and 4-line decoding. MIPI decoding, including MIPI decoding with 4K resolution and MIPI decoding with resolution as small as 720P, MIPI decoding based on Xilinx platform FPGA, MIPI decoding based on Altera platform FPGA, and MIPI decoding based on Lattice platform FPGA, In the future, we will continue to introduce MIPI decoding solutions that are more advanced than domestic FPGAs. After all, the current localized solutions are the mainstream in the future, and more MIPI-encoded DSI solutions will be launched in the future, and we will strive to make the MIPI encoding and decoding solutions for FPGAs the price of cabbage. . .
Based on this, I specially built a MIPI codec column, and put all the MIPI codec blogs in the column to sort out. Brothers who have project needs or learning interests in FPGA codec MIPI can go to my column. Column The address is as follows:
Click to go directly to the column
4. Detailed design plan
The engineering design structure is as follows:
IMX219 camera and its adapter board
Collect 1080P 4Line MIPI video of IMX219 camera, IMX219 camera is a special camera adapter board, provide the schematic diagram and PCB source file of the adapter board, which can be opened with Altium Designer series software, and can even be mass-produced directly on the board; IMX219 camera The adapter board is inserted on the Lattice FPGA development board; as follows:
The following is the physical picture and PCB screenshot of IMX219;
the following is the MIPI 1920x1080 @30FPS Start of a frame waveform output by IMX219 captured by a high-speed oscilloscope, you can see it here The IMX219 hardware designer has no problem, you can learn from it;
D-PHY
Lattice FPGA comes with MIPI decoding source language, which can be directly called to complete MIPI decoding. This is also the unique advantage of Lattice FPGA in MIPI video encoding and decoding. This module almost directly calls MIPI decoding source language, and the input is differential MIPI video data. , the output is parallel video data, because it is the source language, so there is no need for too much explanation, you just need to know how to use it, you don’t need to know why, the top-level interface of this module is as follows:
mipi_rx_ddr (alignwd, buf_clk_lp0i, buf_clk_lp0o, buf_clk_lp0t,
buf_clk_lp1i, buf_clk_lp1o, buf_clk_lp1t, clk, clk_lp0, clk_lp1,
clk_s, init, reset, rx_ready, sclk, oclk, buf_data_lp0i, buf_data_lp0o,
buf_data_lp0t, buf_data_lp1i, buf_data_lp1o, buf_data_lp1t, data_lp0,
data_lp1, datain, q)/* synthesis NGD_DRC_MASK=1 */;
input wire alignwd;
input wire buf_clk_lp0o;
input wire buf_clk_lp0t;
input wire buf_clk_lp1o;
input wire buf_clk_lp1t;
input wire clk;
input wire clk_s;
input wire init;
input wire reset;
input wire [3:0] buf_data_lp0o;
input wire [3:0] buf_data_lp0t;
input wire [3:0] buf_data_lp1o;
input wire [3:0] buf_data_lp1t;
input wire [3:0] datain;
output wire buf_clk_lp0i;
output wire buf_clk_lp1i;
output wire rx_ready;
output wire sclk;
output wire oclk;
output wire [3:0] buf_data_lp0i;
output wire [3:0] buf_data_lp1i;
output wire [31:0] q;
inout wire clk_lp0;
inout wire clk_lp1;
inout wire [3:0] data_lp0;
inout wire [3:0] data_lp1;
data alignment
Data alignment includes Byte alignment and Line alignment;
the Byte alignment module receives raw unaligned bit output aligned bytes from the D-PHY module, bytes on MIPI lanes do not have any defined byte boundaries, so this module looks up on the line Always constant first byte 0xB8, once found 0xB8 determines the byte boundary offset, sets the output valid to determined, and starts outputting the correct byte when the data lane is in MIPI LP state Keep reset; Line
aligned The module receives multi-Line Byte-aligned data from mipi, and the mipi byte clock outputs Line-aligned data in the multi-Line mipi bus. Data on different Lines may appear at different offsets, so the module will wait until all Lines The valid output starts to output Line alignment data, so all Line bytes are output at the same time scale; in this way, the MIPI video completes the real data alignment;
MIPI CSI2
The function of this module is very simple, it is to analyze and discard the control characters in the MIPI protocol, and extract valid video data. Some screenshots of the source code are as follows:
Video data format conversion
The sequence of video data format conversion is: RAW10 to Bayer, Bayer to RGB888, RGB888 to YUV422; if you want to output a monitor, you only need to do the step of Bayer to RGB888 to output, here achieve the purpose of RGB888 to YUV422 It is to send Cypress FX3 USB3.0 interface output; there are detailed notes on the top of each module, and its function can be understood at a glance. Taking the RGB888 to YUV422 module as an example, the screenshot is as follows:
Video output correction
The function of this module is to convert the bit width of the video data, convert the yuv422 data of 64bit 4 pixel clocks into 32bit yuv422 data of 2 pixel clocks, and the purpose of doing this is also to send Cypress FX3 USB3.0 interface output;
5. Detailed explanation of Lattice Diamond project
Lattice Diamond is a development tool for Lattice FPGA. It is very simple to use, but it is rarely used in China;
development board FPGA model: LCMXO3LF-6900C-5BG256C of Lattice;
development tool: Lattice Diamond3.12;
input: IMX219, 4Line 1080P resolution ;
Output: Cypress FX3 USB3.0 interface output;
The source code structure of the project is as follows:
The project has been compiled, as follows:
The screenshot of the integrated schematic diagram is as follows:
6. Board debugging and verification
Hardware requirements:
One piece of Lattice FPGA development board;
one piece of IMX219 camera adapter board;
one piece of Cypress FX3 USB3.0 adapter board;
connection, IMX219 camera adapter board is plugged into the Lattice FPGA development board as input; Cypress FX3 USB3.0 adapter The board is plugged into the Lattice FPGA development board as an output; as shown in the figure below:
Use Cypress's official USB3.0 camera host computer to receive and display the video; as follows:
7. Welfare: acquisition of engineering code
Benefits: Obtaining the engineering code
The code is too large to be sent by email, and it is sent by a link to a certain network disk.
Data acquisition method 1: private, or the V business card at the end of the article.
Data acquisition method 2: The official account at the end of the article, reply 002006
network disk data is as follows:
the data list is as follows:
