Allwinner T507 platform adds V4L2 sensor API to directly operate I2C

reference

I have very seriously referred to the following articles and listed them in advance:
very good article
Linux V4L2 draft
v4l2 learning suggestions and process analysis
V4L2 framework analysis
Linux V4L2 draft
Linux V4L2 draft

Author:
Full analysis of embedded Max V4L2 framework

00 - Overview of V4L2 framework
01 - V4L2 framework-v4l2 device
02 - V4L2 framework-media-device
03 - V4L2 framework-videobuf2
V4L2 framework-control
Data structure of V4L2 framework-control

Author: Letcos
[Camera]v4l2 framework kernel space analysis

The following articles are for general reference, record them here:
V4L2 source code tour one: struct v4l2_subdev
V4L2 source code tour two: V4L2 sub-device userspace API
V4L2 source code tour three: I2C sub-device drivers
V4L2 source code tour Four: struct video_device
V4L2 source code journey five: The starting point and end point of V4L2 V4L2
source code journey six: source code tracking
V4L2 source code journey seven: controls queryctrl()
V4L2 source code journey eight: ioctl
V4L2 source code journey nine ：videobuf
V4L2 source code tour 10: main structure of videobuf

Xilinx Linux V4L2 Video Pipeline Driver Analysis

Target

For a Sensor encapsulated in V4L2, add an API that directly operates I2C at the AP layer.

background:

The SDK of Allwinner T507 is used here, and the Linux version is 4.9.191

Because on the Allwinner platform, the sensor device is not a standard I2C device by default. It can be/dev/i2c-xThe device can be read and written directly. Instead, it is registered as a V4L2 device (/dev/videox). In this way, the AP layer cannot operate the Sensor's Register through regular I2C operations.

The I2C of the Allwinner platform has its own standard: TWI : Normal Two Wire Interface: "The Allwinner platform is compatible with the bus control protocol of the I2C standard protocol."

But at the software level, the description of I2C is still used, which is too disgusting. I think there is no need to invent a new private system, which defeats the original intention of establishing I2C.

What is the purpose of this?

The reason why it is registered as /dev/videox is to avoid the AP layer directly performing I2C operations on the sensor, which may cause problems.
For I2C operations, Allwinner encapsulates a layer of cci for operation.

So whether to use cci or twi (that is, I2C), you can know from the configuration in the kernel that you can choose. The default is to retain the current cci method, and the option is to use V4L2 as a regular I2C device.

.config - Linux/arm64 4.9.191 Kernel Configuration
Device Drivers > Multimedia support > V4L platform devices > select cci or cci to twi (use internal cci) —>
(X) use internal cci
( ) chenge cci to twi

How to do it?

At the beginning, I didn’t know how the V4L2 commands worked, so I could only study it by dissecting a sparrow.
My understanding of subdevice is that the device can be used as a subdevice of I2C, sensor, and v4l2, and can point to and transform each other, so It’s called subdevice. I’ll add this if I have new understanding.

// 从v4l2_subdev获取i2c_client：
struct i2c_client *client = v4l2_get_subdevdata(sd);

// 从i2c_client获取v4l2_subdev:
 struct v4l2_subdev *sd = i2c_get_clientdata(client);
 
 
## 从sd成员来获取父成员的地址
struct v4l2_subdev *sd;
struct sensor_info *info = to_state(sd);

## 获取info的成员成员sd的地址
struct sensor_info *info;
struct v4l2_subdev *sd = &info->sd;
## 这2个是一个互为 相反的操作

1 Study the initialization of V4L2 sub device

On the sunxi platform, the /dev/video0 device is roughly initialized through the following process.

module_init(vin_init);
└──>vin_init(void);
	│	// drivers/media/platform/sunxi-vin/vin-video/vin_core.c
	├──	sunxi_csi_platform_register();
	└──>sunxi_vin_core_register_driver();
		└──>platform_driver_register(&vin_core_driver);
			└──>static struct platform_driver vin_core_driver = {
    
    
					.probe = vin_core_probe,
					...
					}
				vin_core_probe();
				│	//drivers/media/platform/sunxi-vin/vin-video/vin_video.c
				└──>vin_initialize_capture_subdev();
					│	// drivers/media/v4l2-core/v4l2-subdev.c				
					│	// 此时将device作为链表加入init序列,并开始初始化具体的device,下面是间接的调用				
					├──>v4l2_subdev_init();
					│ sd->internal_ops = &vin_capture_sd_internal_ops;
					│ 	struct v4l2_subdev_internal_ops vin_capture_sd_internal_ops = {
    
    
					│ 	.registered = vin_capture_subdev_registered,
					│	.unregistered = vin_capture_subdev_unregistered,
					│};
					└──v4l2_set_subdevdata(sd)
						└──>vin_capture_subdev_registered()
							| // drivers/media/platform/sunxi-vin/vin-video/vin_video.c
							├──>vin_init_controls()
							│	│ //将3A之类的命令注册一下,并给初始值
							│	├──>v4l2_ctrl_new_std(V4L2_CID_XXX)
							│	├──>v4l2_ctrl_new_std_menu(V4L2_CID_XXX)
							│	└──>v4l2_ctrl_new_custom(V4L2_CID_XXX)
							└──>vin_init_video()
								├──>cap->vdev.name // vin_video0
								├──>cap->vdev.fops = &vin_fops; // 文件接口的ioctl
								├──>cap->vdev.ioctl_ops = &vin_ioctl_ops; // ioctl的命令列表
								├──>video_register_device(cap->vdev)
								├──>video_set_drvdata()
								└──>vb2_queue_init()

2 Specific to the sensor mlx75027 I use, the specific initialization process

// drivers/media/platform/sunxi-vin/modules/sensor/mlx75027_mipi.c
module_init(init_sensor);
│	static struct i2c_driver sensor_driver = {
    
    
│			.probe = sensor_probe,
│			...
│	};
└──>init_sensor(void);
	└──>cci_dev_init_helper(&sensor_driver);
		└──>sensor_driver->probe()
			└──>sensor_probe()
				├──>sensor_init_controls(sensor_ctrl_ops);
				│	│  // 定义sensor对3A命令处理的逻辑,可覆盖父级别
				│	└──>v4l2_ctrl_new_std(V4L2_CID_XXX);			
				└──>cci_dev_probe_helper(sensor_ops); // 设置sensor_ops
					│  // 定义sensor对3A命令处理的逻辑,可覆盖父级别
					└──>struct v4l2_subdev_ops sensor_ops = {
    
    
						.core = &sensor_core_ops,
						...
						};
						└──>struct v4l2_subdev_core_ops sensor_core_ops = {
    
    
							.ioctl = sensor_ioctl, // 上层通过ioctl,最终会调用到这里,我实现的自定义命令,也会到这个处理
							...
							};

3. Type of V4L2 cmd

1. The first is that the number of VIDIOC_XXX commands defined in /include/uapi/linux/videodev2.h
   does not exceed #define BASE_VIDIOC_PRIVATE 192. As commands reserved for v4l2, custom commands are not recommended to be added within this range.
   These commands are the basic functions of v4l2 operations, including opening and closing devices, requesting streaming data, etc.

/*
 *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
 *
 */
#define VIDIOC_QUERYCAP		_IOR('V',  0, struct v4l2_capability)
#define VIDIOC_RESERVED		_IO('V',  1)
#define VIDIOC_ENUM_FMT     _IOWR('V',  2, struct v4l2_fmtdesc)
#define VIDIOC_G_FMT		_IOWR('V',  4, struct v4l2_format)
#define VIDIOC_S_FMT		_IOWR('V',  5, struct v4l2_format)
#define VIDIOC_REQBUFS		_IOWR('V',  8, struct v4l2_requestbuffers)
#define VIDIOC_QUERYBUF		_IOWR('V',  9, struct v4l2_buffer)
#define VIDIOC_G_FBUF		_IOR('V', 10, struct v4l2_framebuffer)
#define VIDIOC_S_FBUF		_IOW('V', 11, struct v4l2_framebuffer)
#define VIDIOC_OVERLAY		_IOW('V', 14, int)
#define VIDIOC_QBUF			_IOWR('V', 15, struct v4l2_buffer)
#define VIDIOC_EXPBUF		_IOWR('V', 16, struct v4l2_exportbuffer)
#define VIDIOC_DQBUF		_IOWR('V', 17, struct v4l2_buffer)
#define VIDIOC_STREAMON		_IOW('V', 18, int)
#define VIDIOC_STREAMOFF	_IOW('V', 19, int)

2. The second one is V4L2_CID_XXX defined in /include/uapi/linux/v4l2-controls.h.
   Because it is used by the application layer, the address is defined here, and the range that different applications can use is distinguished according to a certain size.

#define V4L2_CTRL_CLASS_USER	0x00980000	/* Old-style 'user' controls */
#define V4L2_CID_BASE			(V4L2_CTRL_CLASS_USER | 0x900)
#define V4L2_CID_USER_BASE 		V4L2_CID_BASE
#define V4L2_CID_USER_CLASS 	(V4L2_CTRL_CLASS_USER | 1)
#define V4L2_CID_BRIGHTNESS		(V4L2_CID_BASE+0)
#define V4L2_CID_CONTRAST		(V4L2_CID_BASE+1)

3. The third type is the cmd defined in /include/media/sunxi_camera_v2.h for each Camera.
   You can see that after the private 192, you can add the sensor's private data VIDIOC_ISP_XXX and VIDIOC_VIN_SENSOR_XXX .

#define BASE_VIDIOC_PRIVATE	192
#define VIDIOC_ISP_AE_STAT_REQ  _IOWR('V', BASE_VIDIOC_PRIVATE + 1, struct isp_stat_buf)
#define VIDIOC_ISP_HIST_STAT_REQ _IOWR('V', BASE_VIDIOC_PRIVATE + 2, struct isp_stat_buf)
...
#define VIDIOC_VIN_SENSOR_CFG_REQ _IOWR('V', BASE_VIDIOC_PRIVATE + 60, struct sensor_config)
#define VIDIOC_VIN_SENSOR_EXP_GAIN _IOWR('V', BASE_VIDIOC_PRIVATE + 61, struct sensor_exp_gain)
#define VIDIOC_VIN_SENSOR_SET_FPS  _IOWR('V', BASE_VIDIOC_PRIVATE + 62, struct sensor_fps)
...
// 我新增的命令放在了最后
struct msg_i2c {
    
    
	unsigned short addr;
	unsigned short value;
};
#define VIDIOC_VIN_SET_I2C_DATA  _IOWR('V', BASE_VIDIOC_PRIVATE + 76, struct msg_i2c)

4. Processing process of different types of commands

1. Processing of VIDIOC_XXX

	int sel = 0;
	int mode = 5;
	enum v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
	if (-1 == camera_init(sel, mode))
		return -1;
	if (-1 == camera_fmt_set(mode))
		return -1;
	if (-1 == req_frame_buffers())
		return -1;

	pixformat = TVD_PL_YUV420;
	disp_init(input_size.width, input_size.height, pixformat);
	if (-1 == ioctl(fd, VIDIOC_STREAMON, &type))
	{
    
    
		printf("VIDIOC_STREAMON failed\n");
		return -1;
	}

After ioctl, the process from kernel to driver is as follows

// 在 第1节已经知道了  vin_init_video() 会注册 cap->vdev.fops = &vin_fops; 
// 所以对vin设备的ioctl都会到这个函数
static struct v4l2_file_operations vin_fops = {
    
    
	.unlocked_ioctl = video_ioctl2,
}
static struct v4l2_ioctl_info v4l2_ioctls[] = {
    
    
	IOCTL_INFO_FNC(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
	IOCTL_INFO_FNC(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
}

EXPORT_SYMBOL(video_ioctl2);
video_ioctl2()
└──>__video_do_ioctl()
	├──>v4l2_is_known_ioctl() // 如果是在已知cmd列表里面,就执行注册的func
	├──>struct v4l2_ioctl_info *info = &v4l2_ioctls[_IOC_NR(cmd)];
	└──>info->func(ops, file, fh, arg);
		└──>v4l_streamon()
			└──>ops->vidioc_streamon(file, fh, *(unsigned int *)arg);
				└──>static const struct v4l2_ioctl_ops vin_ioctl_ops = {
    
    
					.vidioc_streamon = vidioc_streamon,
					.vidioc_streamoff = vidioc_streamoff,
					}
					// /drivers/media/platform/sunxi-vin/vin-video/vin_video.c
					vidioc_streamon()
					└──>vb2_streamon()
						│ // /drivers/media/v4l2-core/videobuf2-core.c
						└──>vb2_core_streamon()
							├──>v4l_vb2q_enable_media_source()
							└──>vb2_start_streaming()
								│ // /drivers/media/platform/sunxi-vin/modules/sensor/mlx75027_mipi.c
								└──>struct v4l2_subdev_video_ops sensor_video_ops = {
    
    
									.s_stream = sensor_s_stream,
									}
									sensor_s_stream()
									└──>sensor_s_streamon(sd,1);
										└──>sensor_write(sd, 0x1001, 0x01); // 实际操作寄存器

2. Processing of V4L2_CID_XXX

// drivers/media/platform/sunxi-vin/modules/sensor/mlx75027_mipi.c
module_init(init_sensor);
│	static struct i2c_driver sensor_driver = {
    
    
│			.probe = sensor_probe,
│			...
│	};
└──>init_sensor(void);
	└──>cci_dev_init_helper(&sensor_driver);
		└──>sensor_driver->probe()
			└──>sensor_probe()
				├──>sensor_init_controls(sensor_video_ops);
				│	│  // 定义sensor对3A命令处理的逻辑,可覆盖父级别
				│	└──>v4l2_ctrl_new_std(V4L2_CID_XXX);			
				└──>cci_dev_probe_helper(sensor_ops); // 设置sensor_ops
					│  // 定义sensor对3A命令处理的逻辑,可覆盖父级别
					└──>v4l2_subdev_video_ops sensor_video_ops = {
    
    
							.s_parm = sensor_s_parm,
							.g_parm = sensor_g_parm,
						...
						};
						└──>struct v4l2_ioctl_ops soc_camera_ioctl_ops = {
    
    
							.vidioc_g_parm		 = soc_camera_g_parm,
							}
							└──>soc_camera_g_parm()
								└──>default_g_parm()
									└──>vidioc_g_parm()
										└──>v4l2_subdev_call(vinc->vid_cap.pipe.sd[VIN_IND_SENSOR], video, g_parm, parms);
											└──>in_ctrl_ops = {
    
    
												.g_volatile_ctrl = vin_g_volatile_ctrl,
												};
												└──>vin_g_volatile_ctrl()
													// /drivers/media/platform/sunxi-vin/modules/sensor/mlx75027_mipi.c
													v4l2_ctrl_ops sensor_ctrl_ops = {
    
    
														.g_volatile_ctrl = sensor_g_ctrl,
													};
													└──>int sensor_g_ctrl(struct v4l2_ctrl *ctrl)
															switch (ctrl->id)
															case V4L2_CID_EXPOSURE:
																sensor_g_exp(); // 实际的处理函数

3. Custom cmd processing

 // /include/media/sunxi_camera_v2.h   在头文件增加新的cmd和传入的参数定义
struct msg_i2c {
    
    
	unsigned short addr;
	unsigned short value;
};
#define VIDIOC_VIN_SET_I2C_DATA	_IOWR('V', BASE_VIDIOC_PRIVATE + 76, struct msg_i2c)

// /drivers/media/platform/sunxi-vin/vin-video/vin_video.c
struct v4l2_ioctl_ops vin_ioctl_ops = {
    
    
	.vidioc_default = vin_param_handler, // 走的是这个函数
└──>vin_param_handler()
	└──>switch (cmd) {
    
    
		case VIDIOC_VIN_SET_I2C_DATA:
		vidioc_set_i2c_zhang(struct file *file, struct v4l2_fh *fh, struct msg_i2c *i2c);
		└──>struct vin_core *vinc = video_drvdata(file);
			v4l2_subdev_call(vinc->vid_cap.pipe.sd[VIN_IND_SENSOR], core, ioctl, VIDIOC_VIN_SET_I2C_DATA, i2c);
			│// /drivers/media/platform/sunxi-vin/modules/sensor/mlx75027_mipi.c
			└──>struct v4l2_subdev_ops sensor_ops = {
    
    
				.core = &sensor_core_ops,
				};
				struct v4l2_subdev_core_ops sensor_core_ops = {
    
    
				.ioctl = sensor_ioctl,
				}
				└──>sensor_ioctl()
					└──>switch (cmd) {
    
    
						case VIDIOC_VIN_SET_I2C_DATA:
				        sensor_s_i2c_data(sd, (struct msg_i2c *)arg);
				        └──> sensor_write(sd, i2c->addr, i2c->value);

4. v4l2_subdev_call() This is the focus of implementing custom cmd

/*
 * Call an ops of a v4l2_subdev, doing the right checks against
 * NULL pointers.
 *
 * Example: err = v4l2_subdev_call(sd, video, s_std, norm);
 */
#define v4l2_subdev_call(sd, o, f, args...)				\
	(!(sd) ? -ENODEV : (((sd)->ops->o && (sd)->ops->o->f) ?	\
		(sd)->ops->o->f((sd), ##args) : -ENOIOCTLCMD))
vidioc_set_i2c_zhang(struct file *file, struct v4l2_fh *fh, struct msg_i2c *i2c);		
└──>v4l2_subdev_call(vinc->vid_cap.pipe.sd[VIN_IND_SENSOR], core, ioctl, VIDIOC_VIN_SET_I2C_DATA, i2c);

struct v4l2_subdev_ops sensor_ops = {
    
    
	.core = &sensor_core_ops,
	};
struct v4l2_subdev_core_ops sensor_core_ops = {
    
    
	.ioctl = sensor_ioctl,
	}

v4l2_subdev_call(vinc->vid_cap.pipe.sd[VIN_IND_SENSOR], core, ioctl, VIDIOC_VIN_SET_I2C_DATA, i2c);
Actually equivalent to
v4l2_subdev_call(subdevice, sensor_ops , ioctl, VIDIOC_VIN_SET_I2C_DATA, struct msg_i2c);
The translation is to call the ioctl method of sensor_ops through subdevice. The parameter passed in is VIDIOC_VIN_SET_I2C_DATA, and the structure of the method is struct msg_i2c.

5. Problems encountered

1. The values ​​of is_isp_used and is_bayer_raw will affect the processing logic of V4L2_CID_XXX

static int vin_g_volatile_ctrl(struct v4l2_ctrl *ctrl)
{
    
    
	struct vin_vid_cap *cap = container_of(ctrl->handler, struct vin_vid_cap, ctrl_handler);
	struct sensor_instance *inst = get_valid_sensor(cap->vinc);
	if (inst->is_isp_used && inst->is_bayer_raw) {
    
    
	// 一般而言,V4L2_CID_XXX 会走这个分支
	} else {
    
    
	}
// 	/drivers/media/platform/sunxi-vin/vin.c
static int __vin_handle_sensor_info(struct sensor_instance *inst)
{
    
    
	if (inst->cam_type == SENSOR_RAW) {
    
    
		inst->is_bayer_raw = 1;
		inst->is_isp_used = 1;
	} else if (inst->cam_type == SENSOR_YUV) {
    
    
		inst->is_bayer_raw = 0;
		inst->is_isp_used = 0;
	} else {
    
    
		inst->is_bayer_raw = 0;
		inst->is_isp_used = 0;
	}
	return 0;
}
// 从这个函数可以看到,只有sensor的type是SENSOR_RAW的时候,这2个属性才都是1.
// 所以就需要在初始化列表里面将sensor的属性配置好,如果是新增的sensor,这2个值都是0.
// /drivers/media/platform/sunxi-vfe/utility/sensor_info.c
struct sensor_item sensor_list_t[] = {
    
    
	/* name       i2c_addr      sensor type  sensor size   sensor max pclk */
	{
    
    	"ov2640",	0x60,		SENSOR_YUV,	 PIXEL_NUM_2M, CORE_CLK_RATE_FOR_2M},
	{
    
    	"ov5647_mipi",	0x6c,	SENSOR_RAW,	 PIXEL_NUM_5M, CORE_CLK_RATE_FOR_5M},
	}
// 可以把新增的sensor添加在这个结构体的后面.	

2. The meaning of linux kernel macro container_of

Analysis of linux kernel macro container_of
About linux container_of usage
container_of usage and implementation of
linux driver container_of macro analysis
detailed explanation of bit operations of Linux kernel

struct struct *struct_p= container_of(struct_member_p, struct struct, struct_member);
//container_of 的作用是:已知 struct结构中某个成员struct_member的指针struct_member_p，就可以知道整个struct的指针struct_p

6. Allwinner provides the method for V4L2 devices to directly read and write I2C

I only saw this method on Quanzhi's official website after encapsulating the API... share it now.
[FAQ440] Tina online reading and writing Sensor register example
Release date 2021-10-30 15:47:18
Platform V536 V533 V833 V831 I just don’t know if T507 also supports it. After actually looking at it, all the files in the sensor directory are consistent.

[Command example]
1) cd /sys/devices/gc2053_mipi (enter the target sensor node directory)
2) echo 16 > addr_width; echo 8 > data_width (enter the target sensor register address/data bit width, please refer to the datasheet to obtain)
3) echo 0 > read_flag (read_flag: read and write control node, enabling it to 1 means that the subsequent operation is a read action, and enabling it to 0 means that the subsequent operation is a write action) 4) echo 30350021
> cci_client ("30350021": 0x3035 [target register address] , 0x0021 [Register value to be written], in the case of read_flag = 1, the written value is invalid)
5) cat read_value (print the result of the previous step: register value)

[Notes]
1) IIC works normally on the system and hardware;
2) The above commands need to be executed in the normal working state of the Camera to be effective, such as whether the /sys/devices/sensor node exists, whether the Camera is powered on, etc.;
3 ) The above commands are only valid in the tina-linux system, and are not supported by the rtos system yet;

There are a few good pictures, I’ll share them too