This article mainly introduces the driver debugging of the RK3588 platform camera sensor HDR and the sharing of problems encountered in the process of debugging the IMX577 HDR mode.
Table of contents
(2) RK3588 ISP supports HDR introduction
3. The driver adds HDR-related IOCTL interface
4. Improve the function of HDR exposure gain control according to the guidance of the manual
5. The addition of other interfaces to drive:
6.dts configuration and xml configuration
(4) IMX577: HDR debugging and troubleshooting
1. Grab the raw image of long and short frames
2. Troubleshoot abnormal effects
(1) Introduction to HDR
High dynamic range imaging achieves a larger exposure dynamic range than ordinary digital image technology. Simply put, it can display a larger brightness range (larger difference between light and dark).
The simple point is to use the camera to take two or three images with different exposure times at the same time (called short exposure, medium exposure and long exposure), and then use an algorithm to extract the parts with excellent light and dark performance from the three photos and fuse them into one. A new image.
For example: the above three images are short exposure time, medium exposure time, and long exposure time, and finally the three images are fused into a better image through the ISP algorithm.
For the specific introduction of HDR, I won’t go into details here. You can refer to the following articles:
What is HDR | Geek Notes (deepinout.com)
(2) RK3588 ISP supports HDR introduction
Through RK official documents, we can know that RK3588 ISP supports HDR, supports 2-in-1, 3-in-1 multi-frame HDR mode:
The HDR types supported by RK3588 are: Sequential / Framebase HDR, Staggered / DOL HDR, DCG HDR.
不支持：SME HDR、BME HDR、 QBC HDR、 Lateral overflow HDR、 Splite-diode pixel HDR。
So when choosing an HDR sensor, you need to pay attention to choosing the type supported by RK3588. Regarding the description of more HDR types, I won’t go into details here, and I can learn it later when I have time.
(3) IMX577: HDR debugging
Here we take debugging the IMX577 HDR mode as an example to share the experience of debugging the HDR process.
1. IMX577 HDR support
It can be seen from the datasheet of the sensor that the IMX577 supports DOL-HDR mode, which meets the requirements of RK3588 and can be verified on RK3588.
IMX577 supports 2-frame mode and 3-frame mode, here we take 2-frame mode as an example for debugging.
Note: RK3588 platform only supports VC mode for receiving long and short frames, that is, the long frame and short frame images are output through MIPI virtual channels VC0 and VC1 respectively, and the ISP receives and merges them.
2. Obtain the HDR configuration sequence, refer to the HDR application manual, and obtain the HDR related registers
The configuration related to HDR is quite different from the linear mode, and the original factory of the sensor needs to provide the corresponding initialization sequence.
Similarly, HDR has long frames and short frames that need to control registers such as exposure gain, so it is necessary to extract related register controls from the application manual or datasheet and add them to the driver.
IMX577 adds long and short frame control registers as follows:
@@ -67,6 +71,25 @@
#define IMX577_GAIN_STEP 0x1
#define IMX577_GAIN_DEFAULT 0x20
+#define IMX577_LF_GAIN_REG_H 0x00f0
+#define IMX577_LF_GAIN_REG_L 0x00f1
+#define IMX577_SEF1_GAIN_REG_H 0x00f2
+#define IMX577_SEF1_GAIN_REG_L 0x00f3
+#define IMX577_SEF2_GAIN_REG_H 0x00f4
+#define IMX577_SEF2_GAIN_REG_L 0x00f5
+
+#define IMX577_LF_EXPO_REG_H 0x00ea
+#define IMX577_LF_EXPO_REG_L 0x00eb
+#define IMX577_SEF1_EXPO_REG_H 0x00ec
+#define IMX577_SEF1_EXPO_REG_L 0x00ed
+#define IMX577_SEF2_EXPO_REG_H 0x00ee
+#define IMX577_SEF2_EXPO_REG_L 0x00ef
+
+#define IMX577_RHS1_REG_H 0x00e6
+#define IMX577_RHS1_REG_L 0x00e7
+#define IMX577_RHS2_REG_H 0x00e8
+#define IMX577_RHS2_REG_L 0x00e9
+
#define IMX577_REG_TEST_PATTERN 0x0600
#define IMX577_TEST_PATTERN_ENABLE 0x02
#define IMX577_TEST_PATTERN_DISABLE 0x0
@@ -79,14 +102,30 @@
#define IMX577_FETCH_AGAIN_H(VAL) (((VAL) >> 8) & 0x03)
#define IMX577_FETCH_AGAIN_L(VAL) ((VAL) & 0xFF)
+#define IMX577_FETCH_GAIN_H(VAL) (((VAL) >> 8) & 0xFF)
+#define IMX577_FETCH_GAIN_L(VAL) ((VAL) & 0xFF)
+
+#define IMX577_FETCH_RHS1_H(VAL) (((VAL) >> 8) & 0xFF)
+#define IMX577_FETCH_RHS1_L(VAL) ((VAL) & 0xFF)
+
#define REG_NULL 0xFFFF
#define IMX577_REG_VALUE_08BIT 1
#define IMX577_REG_VALUE_16BIT 2
#define IMX577_REG_VALUE_24BIT 3
+#define IMX577_GROUP_HOLD_REG 0x0104
+#define IMX577_GROUP_HOLD_START 0x01
+#define IMX577_GROUP_HOLD_END 0x00
+
+/* Basic Readout Lines. Number of necessary readout lines in sensor */
+#define BRL_FULL 3077
+
+#define CIT_MARGIN 22
+
#define OF_CAMERA_PINCTRL_STATE_DEFAULT "rockchip,camera_default"
#define OF_CAMERA_PINCTRL_STATE_SLEEP "rockchip,camera_sleep"
+#define OF_CAMERA_HDR_MODE "rockchip,camera-hdr-mode"Add the corresponding HDR mode support in the driver support mode:
{
.width = 4056,
.height = 3040,
.max_fps = {
.numerator = 10000,
.denominator = 250000,
},
.exp_def = 0x0c10,
.hts_def = 0x11a0,
.vts_def = 0x0e8a,
.bpp = 10,
.bus_fmt = MEDIA_BUS_FMT_SRGGB10_1X10,
.reg_list = imx577_hdr2_10bit_4056x3040_30fps_regs,
.link_freq_idx = 0,
.hdr_mode = HDR_X2,
.vc[PAD0] = V4L2_MBUS_CSI2_CHANNEL_1,
.vc[PAD1] = V4L2_MBUS_CSI2_CHANNEL_0,//L->csi wr0
.vc[PAD2] = V4L2_MBUS_CSI2_CHANNEL_1,
.vc[PAD3] = V4L2_MBUS_CSI2_CHANNEL_1,//M->csi wr2
},3. The driver adds HDR-related IOCTL interface
Including PREISP_CMD_SET_HDRAE_EXP, RKMODULE_GET_HDR_CFG, RKMODULE_SET_HDR_CFG, RKMODULE_GET_CHANNEL_INFO and other interfaces, because RK's camera uses the standard V4L2 framework, and the ctrl of the V4L2 framework does not have an HDR control interface, so it is necessary to add RK's private ioctl interface. The core is the PREISP_CMD_SET_HDRAE_EXP interface, which mainly realizes the control of the exposure gain of long and short frames.
case PREISP_CMD_SET_HDRAE_EXP:
if (imx577->cur_mode->hdr_mode == HDR_X2)
ret = imx577_set_hdrae(imx577, arg);
break;
case RKMODULE_GET_MODULE_INFO:
imx577_get_module_inf(imx577, (struct rkmodule_inf *)arg);
break;
case RKMODULE_GET_HDR_CFG:
hdr = (struct rkmodule_hdr_cfg *)arg;
hdr->esp.mode = HDR_NORMAL_VC;
hdr->hdr_mode = imx577->cur_mode->hdr_mode;
break;
case RKMODULE_SET_HDR_CFG:
hdr = (struct rkmodule_hdr_cfg *)arg;
w = imx577->cur_mode->width;
h = imx577->cur_mode->height;
for (i = 0; i < ARRAY_SIZE(supported_modes); i++) {
if (w == supported_modes[i].width &&
h == supported_modes[i].height &&
supported_modes[i].hdr_mode == hdr->hdr_mode) {
imx577->cur_mode = &supported_modes[i];
break;
}
}
if (i == ARRAY_SIZE(supported_modes)) {
dev_err(&imx577->client->dev,
"not find hdr mode:%d %dx%d config\n",
hdr->hdr_mode, w, h);
ret = -EINVAL;
} else {
mode = imx577->cur_mode;
imx577->cur_vts = mode->vts_def;
w = mode->hts_def - mode->width;
h = mode->vts_def - mode->height;
mutex_lock(&imx577->mutex);
__v4l2_ctrl_modify_range(imx577->hblank, w, w, 1, w);
__v4l2_ctrl_modify_range(imx577->vblank, h,
IMX577_VTS_MAX - mode->height,
1, h);
__v4l2_ctrl_s_ctrl(imx577->link_freq, mode->link_freq_idx);
dst_pixel_rate = (u32)link_freq_items[mode->link_freq_idx] /
mode->bpp * 2 * lane_num;
__v4l2_ctrl_s_ctrl_int64(imx577->pixel_rate,
dst_pixel_rate);
mutex_unlock(&imx577->mutex);
}
break;4. Improve the function of HDR exposure gain control according to the guidance of the manual
According to the manual description of IMX577, in HDR mode, the restrictions on the exposure gain of long and short frames are as follows:
The driver code needs to limit the control of exposure gain according to the above constraints, as follows:
+static int imx577_set_hdrae(struct imx577 *imx577,
+ struct preisp_hdrae_exp_s *ae)
+{
+ struct i2c_client *client = imx577->client;
+ u32 l_exp_time, m_exp_time, s_exp_time;
+ u32 l_a_gain, m_a_gain, s_a_gain;
+ int ret = 0;
+ u32 fll, dol_cit1, dol_cit2, dol_off2;
+
+ if (!imx577->has_init_exp && !imx577->streaming) {
+ imx577->init_hdrae_exp = *ae;
+ imx577->has_init_exp = true;
+ dev_dbg(&imx577->client->dev, "imx577 is not streaming, save hdr ae!\n");
+ return ret;
+ }
+ l_exp_time = ae->long_exp_reg;
+ m_exp_time = ae->middle_exp_reg;
+ s_exp_time = ae->short_exp_reg;
+ l_a_gain = ae->long_gain_reg;
+ m_a_gain = ae->middle_gain_reg;
+ s_a_gain = ae->short_gain_reg;
+ dev_dbg(&client->dev,
+ "rev exp req: L_exp: 0x%x, 0x%x, M_exp: 0x%x, 0x%x S_exp: 0x%x, 0x%x\n",
+ l_exp_time, m_exp_time, s_exp_time,
+ l_a_gain, m_a_gain, s_a_gain);
+
+ if (imx577->cur_mode->hdr_mode == HDR_X2) {
+ l_a_gain = m_a_gain;
+ l_exp_time = m_exp_time;
+ }
+
+ ret = imx577_write_reg(client, IMX577_GROUP_HOLD_REG,
+ IMX577_REG_VALUE_08BIT, IMX577_GROUP_HOLD_START);
+ /* gain effect n+1 */
+ if (l_a_gain > 0x160)
+ l_a_gain = 0x160;
+ if (l_a_gain < 0x10)
+ l_a_gain = 0x10;
+ if (s_a_gain > 0x160)
+ s_a_gain = 0x160;
+ if (s_a_gain < 0x10)
+ s_a_gain = 0x10;
+ l_a_gain = 1024 - 1024 * 16 / l_a_gain;
+ s_a_gain = 1024 - 1024 * 16 / s_a_gain;
+
+ ret |= imx577_write_reg(client, IMX577_LF_GAIN_REG_H,
+ IMX577_REG_VALUE_08BIT, IMX577_FETCH_GAIN_H(l_a_gain));
+ ret |= imx577_write_reg(client, IMX577_LF_GAIN_REG_L,
+ IMX577_REG_VALUE_08BIT, IMX577_FETCH_GAIN_L(l_a_gain));
+ ret |= imx577_write_reg(client, IMX577_SEF1_GAIN_REG_H,
+ IMX577_REG_VALUE_08BIT, IMX577_FETCH_GAIN_H(s_a_gain));
+ ret |= imx577_write_reg(client, IMX577_SEF1_GAIN_REG_L,
+ IMX577_REG_VALUE_08BIT, IMX577_FETCH_GAIN_L(s_a_gain));
+
+ fll = imx577->cur_vts;
+ dol_cit1 = l_exp_time/* >> 1*/;
+ dol_cit2 = s_exp_time/* >> 1*/;
+
+ /*dol_cit1 dol_cit2 dol_off2 should be even*/
+ if (dol_cit1 < 2)
+ dol_cit1 = 2;
+ else if (dol_cit1 > fll - 2 * CIT_MARGIN - 2)
+ dol_cit1 = fll - 2 * CIT_MARGIN - 2;
+ dol_cit1 &= (~0x1);
+
+ if (dol_cit2 < 2)
+ dol_cit2 = 2;
+ else if (dol_cit2 > fll - BRL_FULL - CIT_MARGIN)
+ dol_cit2 = fll - BRL_FULL - CIT_MARGIN;
+ dol_cit2 &= (~0x1);
+
+ dol_off2 = (dol_cit2 + CIT_MARGIN) & (~0x1);
+ if (dol_off2 < dol_cit2 + CIT_MARGIN)
+ dol_off2 = (dol_cit2 + CIT_MARGIN) & (~0x1);
+ else if (dol_off2 > fll - BRL_FULL)
+ dol_off2 = (fll - BRL_FULL) & (~0x1);
+
+ dev_dbg(&client->dev,
+ "l_exp_time=%d,s_exp_time=%d,fll=%d,rhs1=%d,l_a_gain=%d,s_a_gain=%d\n",
+ l_exp_time, s_exp_time, fll, dol_off2, l_a_gain, s_a_gain);
+
+ ret |= imx577_write_reg(client,
+ IMX577_RHS1_REG_L,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_RHS1_L(dol_off2));
+ ret |= imx577_write_reg(client,
+ IMX577_RHS1_REG_H,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_RHS1_H(dol_off2));
+
+ ret |= imx577_write_reg(client,
+ IMX577_SEF1_EXPO_REG_L,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_EXP_L(dol_cit2));
+ ret |= imx577_write_reg(client,
+ IMX577_SEF1_EXPO_REG_H,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_EXP_H(dol_cit2));
+ ret |= imx577_write_reg(client,
+ IMX577_LF_EXPO_REG_L,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_EXP_L(dol_cit1));
+ ret |= imx577_write_reg(client,
+ IMX577_LF_EXPO_REG_H,
+ IMX577_REG_VALUE_08BIT,
+ IMX577_FETCH_EXP_H(dol_cit1));
+
+ ret |= imx577_write_reg(client, IMX577_GROUP_HOLD_REG,
+ IMX577_REG_VALUE_08BIT, IMX577_GROUP_HOLD_END);
+
+ return ret;
+}Among them, the group hold register is used. The function of this register is to make the register take effect at the same time, so as to prevent the register of the long frame from being written, and the long frame has already taken effect before the writing of the short frame starts.
5. The addition of other interfaces to drive:
imx577_enum_frame_interval: The last reverse of this interface needs to report the corresponding HDR mode.
@@ -1329,6 +1910,7 @@ static int imx577_enum_frame_interval(struct v4l2_subdev *sd,
fie->width = supported_modes[fie->index].width;
fie->height = supported_modes[fie->index].height;
fie->interval = supported_modes[fie->index].max_fps;
+ fie->reserved[0] = supported_modes[fie->index].hdr_mode;
return 0;
}imx577_g_mbus_config: Configure the number of MIPI lanes used.
@@ -1337,12 +1919,15 @@ static int imx577_g_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
struct v4l2_mbus_config *config)
{
struct imx577 *imx577 = to_imx577(sd);
+ const struct imx577_mode *mode = imx577->cur_mode;
u32 lane_num = imx577->bus_cfg.bus.mipi_csi2.num_data_lanes;
u32 val = 0;
val = 1 << (lane_num - 1) |
V4L2_MBUS_CSI2_CHANNEL_0 |
V4L2_MBUS_CSI2_CONTINUOUS_CLOCK;
+ if (mode->hdr_mode != NO_HDR)
+ val |= V4L2_MBUS_CSI2_CHANNEL_1;
config->type = V4L2_MBUS_CSI2_DPHY;
config->flags = val;6.dts configuration and xml configuration
dts needs to be added as follows:
rockchip,camera-hdr-mode = <5>; // 0: no_hdr; 5: hdr2; 6: hdr3xml add configuration:
<isoAnalogGain1 value="75"/> <!--Pseudo ISO corresponding analog gain value 1.0. -->
<cITMaxMargin value="10"/> <!--coarse integration time max margin -->
<aiq.workingMode value="HDR2"/> <!-- NORMAL or HDR2 or HDR3 -->(4) IMX577: HDR debugging and troubleshooting
1. Grab the raw image of long and short frames
First set to non-compact mode:
echo 0 0 1 0 > /sys/devices/platform/rkcif-mipi-lvds2/compact_testGrab video0 long frame raw:
v4l2-ctl -d /dev/video0 --set-fmt-video=width=4056,height=3040,pixelformat='BG10' --stream-mmap=3 --stream-skip=4 --stream-to=/data/out_long.raw --stream-count=1 --stream-pollGrab video1 short frame raw:
v4l2-ctl -d /dev/video1 --set-fmt-video=width=4056,height=3040,pixelformat='BG10' --stream-mmap=3 --stream-skip=4 --stream-to=/data/out_short.raw --stream-count=1 --stream-poll2. Troubleshoot abnormal effects
I encountered a fatal problem during the debugging process, and found that the effect of HDR fusion was not good. After investigation, the brightness of the long and short frames captured by the sensor was inconsistent (here: the brightness of the raw captured by the long and short frames is different, RK colleagues You can use a simulation tool to convert the exposure ratio back, and the obtained image finds that the brightness of the long and short frames is still inconsistent), so there is a problem with the image given by the sensor driver.
Check the exposure gain setting of HDR:
The CIT_MARGIN parameter is fixed at 22.
RBL parameters, fixed to 3077:
The limiting conditions of long and short frame exposure gain are as follows:
As shown in the figure above, the default sequence FLL, that is, VTS is 3102, then the short frame exposure is always limited to 2, so the short frame exposure cannot be adjusted, resulting in abnormal effects.
Solution: Increase VTS and increase the adjustment range of short-frame exposure, but increasing VTS will lead to a decrease in frame rate, which requires a trade-off.
(5) Summary
The article mainly introduces the debugging of the sensor driver to increase the HDR mode. At the same time, it also shares a serious problem I encountered during the IMX577 HDR debugging process, and shares the troubleshooting and solution of the problem. I hope it will be helpful to students who are debugging HDR help.