Overview: This article focuses on the mainstream camera solutions in autonomous driving. The mipi_csi2 on the main control soc is matched with the max96712 camera complete link software framework and implementation plan. It mainly discusses two solutions, GMSL1 and GMSL2. This article is a record during my debugging process. If it is wrong Local welcome to discuss: [email protected]

1. The complete framework of the Camera link:

1.1 Link frame diagram:

GMSL1 scheme: 3Gbps

NOTE:

The camera module adopts Senyun SG2-AR0231C-0202-GMSL

GMSL2 solution: 6Gbps

NOTE:

The camera module adopts Senyun SG2-AR0233C-5200-GMSL2

1.2 camera module information:

SG2-AR0231C-0202-GMSL（GMSL1）:

Module block diagram:

Default address information (8bit means, 7bit means need to convert by yourself):

MAX96705: 0x80 converted to 7bit corresponds to 0x40

AP0202 : 0xBA

AR0231 : 0X20

key parameter:

HDR support

1920*1080/22fps

YUV422@8bit output

SG2-AR0233C-5200-GMSL2（GMSL2）：

Module block diagram:

Default address information (8bit means, 7bit means need to convert by yourself):

MAX9295A: 0x80 converted to 7bit corresponds to 0x40

GW5200 : 0xDA

AR0233 : 0X20

key parameter:

HDR support

LFM support ---- LED flicker suppression

The widespread use of LEDs on the road has brought about the phenomenon of "LED flickering". "LED flickering" is a phenomenon caused by the LED driving method. The LED lamp is driven by AC, and the driving frequency is generally above 90Hz, that is, the slowest pulse period is about 11 milliseconds, and the LED turns on and off once within 11 milliseconds. Energy saving and extended service life, the duty cycle usually does not exceed 50%. If the exposure time of the camera is short (for example, 3 milliseconds), it is possible that the exposure time coincides with the period when the LED is turned off. At this time, the image sensor captures the LED If the image is turned off, if it is an LED array, in this case, part of the captured image is bright and part is dark, which is the phenomenon of "LED flickering"

1920*1080/30fps

YUV422@8bit output

2. I2C address determination: Take GMSL1 as an example

max96712:

The first power-on is determined by the pull-up status of CFG0. The specific details are as follows:

Our circuit configuration is 0xD6, bit0 is the direction bit, so the 7bit address after one bit shifted to the right is 0x6B

DEV_ADDR (0x00) is the address register, and a new address can be written through 0x6B. For example, if 0x90 is written, the actual address will be shifted to the right and the direction bit0 will be changed to 0x48, and then the max96712 will be operated through this address.

max96705:

default addr: 0x80, after shifting one bit to the right, it is 0x40. When the i2c configuration opens the lane channel in sequence, the new address will be written to reg (0x00). LaneA, LaneB, LaneC, and LaneD are respectively set to 0x41, 0X42, 0X43, 0X44.

as follows:

regArray[0] = 0x00 ;
sendBuf[0] = 0x80 + 2 * (i + 1);
i2c_ptr->I2cWrite(0x40, regArray, 1, sendBuf, 1); //Configure ABCD channel max96705 i2c address

note : max96705 has an i2c address translation function

//set i2c_source A

regArray[0] = 0x09;
sendBuf[0] = 0xba + 2 * (i + 1);
i2c_ptr->I2cWrite(0x41 + i, regArray, 1, sendBuf, 1);

//set i2c dst A
regArray[0] = 0x0a;
sendBuf[0] = 0xba;
i2c_ptr->I2cWrite(0x41 + i, regArray, 1, sendBuf, 1);

When i=0 in the above example, it is to convert 0xba+2 (0xbc, the corresponding i2c address is 0x5e) to 0xba, which is the default address of ap0202 mentioned below. i from 0 to 3 respectively will convert 0x5e, 0x5f, 0x60, 0x60 into 0xba

AP0202：

Our module is connected to VDDIO_H by default, so it is 0xBA, combined with the i2c address translation function of max96705 above, the purpose of configuring ap0202 can be achieved

max9295A:

default addr: 0x80, after shifting one bit to the right, it will be 0x40, and then change the corresponding MAX9295A to 0x41-0x44 when opening the laneA-laneD channels on the 96712 in turn, the code is as follows:

The i2c-tools tool under linux is used:

The open source code i2c-tools toolkit under linux provides multiple i2c tools, which can scan addresses (i2c-detect) based on the toolkit, test i2c register read and write, etc.

3. Pipeline description

GMSL1 pipeline diagram:

illustrate:

In GMSL1 mode, each serializer cannot contain multiple pipes, and after entering the max96712, the internal pipes cannot be arbitrarily matched. The fixed connection is as shown in the red part above:

GMSL1 A -->PIPE 0

GMSL1 B -->PIPE1

GMSL1 C -->PIPE 2

GMSL1 D -->PIPE3

GMSL2 pipeline diagram:

illustrate:

The serializer can be configured as pipe X/pipe Y/pipe Z/pipe U 4 output pipes. Generally, it can be configured as one (we configure it as pipe Z), but if you need to output RAW original data or HDR data, you can configure it More than one pipe, as shown above, GMSL2 A/GMSL2 D are configured with two output pipes

The 96712 deserializer can select the pipex/y/z/u in the input PORTA-PORTD for pipe0-pipe7, please refer to the register 0xF0, 0xF1, 0xF2, 0xF3 for details

For pipe, map_src, map_dst, mipi-csi2 related registers and channel selection configuration, VC virtual channel configuration, etc., please refer to the related registers starting from 0X90A, such as 0x90a/0x90b/0x92d/0x90d/0x90e, etc.

We have allocated 4 virtual channel numbers such as VC 0/1/2/3 for the 4 channel data of 96712, occupying two bits

4. PWM trigger frame synchronization

Default scheme: do not use external GPIO to trigger FSYNC, use internal FSYNC signal

The configuration is as follows:

reg : 0x4a0 val : 0x02

PWM external GPIO input 96712 to trigger FSYNC

The configuration is as follows:

reg : 0x4a0 val : 0x08

reg : 0x4af val : 0x9f

note:

In this mode, the SoC needs to trigger PWM and output to the FSYNC synchronization pin of max96712, which can control the image trigger time, synchronization and frame rate of multiple cameras. The following figure provides 20Hz/25Hz test conditions respectively.

20Hz: 500ms interval for obtaining 10 frames of pictures in the application layer

25Hz: The interval for obtaining 10 frames of pictures in the application layer is 400ms

5. Image Basics

Brief introduction of YUV

Divided into three components, "Y" represents the brightness (Luminance or Luma), that is, the gray value; while "U" and "V" represent the chroma (Chrominance or Chroma), which are used to describe the image color and Saturation, which specifies the color of the pixel.

YUV is divided into two formats, planar and packed. The planar format stores the Y of all pixels first, and then stores UV, which is divided into 2-planar/3-planar. In packed format, each pixel Y, U, V is stored alternately and continuously

Cb in YCbCr marks U, Cr marks V

YUV422 8bit: Key instructions, learn other formats by yourself

Sample legend:

Transfer style:

frame format: