V4L2 driver framework of V4L2 series

foreword

  本篇文章主要介绍Linux内核中的V4L2框架，本篇文章所用内核版本：linux-4.19

  v4L2 (Video for Linux 2) is a set of video framework for linux. The main body is located in the kernel. It can be understood as the video source capture driver framework on the entire linux system. It is widely used in embedded devices, mobile terminals, and personal computer devices. Devices that use video image acquisition on the market, such as mobile phones, IPCs, and driving recorders, will use this framework for video acquisition.

  v4L2 allows the application to control the image sensor and the transmission format, and the application uses this to complete image sensor data applications such as taking pictures, previewing, and video recording.

  Previously, Linux also had the first version of V4L2, which dropped support in kernel version 2.6.38.

1. Overview of the V4L2 driver framework

1. Application layer-"middle layer-"driver layer

  Linux uses a multi-level driver architecture to unify and abstract interfaces. The lowest-level drivers are always directly oriented to hardware, while the highest-level drivers are divided into three categories: character, block, and network devices. The first two types of drivers are in the file A "virtual file" similar to a file is formed in the system, also known as a "node node". These nodes have different names and represent different devices, and are managed in a unified way under the directory /dev. System call functions such as open, close, and read are also called Similar to the operation of ordinary files, the consistency of this interface is abstracted by VFS (Virtual File System Layer).

  V4L2 is an intermediate driver layer for video devices. It provides a common interface for applications to access video devices, and provides a unified V4L2 framework for device driver development . Its video device node path is usually videoX in /dev. The V4L2 driver provides the form of "character device" to user space,The major device number is 81, controlled by various ioctl calls in user space, and can use mmap for memory mapping.

V4L2支持多种设备，有以下接口:
√视频采集接口（video capture interface)
√视频输出接口（video output interface)
√直接传输视频接口（video overlay interface) 
√视频间隔消隐信号接口（VBI interface)
√收音机接口(radio interface).

	应用通过open、ioctl等系统调用操作video设备。在内核空间，Video设备的具体操作
方法由驱动中的struct video_device提供。

	驱动使用video_register_device函数将struct video_device注册到V4L2的核心
层，然后V4L2的核心层再向上注册一个字符设备。这样应用就可以使用系统调用访问虚拟
文件系统中Video设备提供的方法，然后进一步访问V4L2核心层提供的v4l2_fops方法集
合，最后通过struct video_device结构休中的fops和ioctl_ops方法集合访问Video
主设备。

	Video主设备通过V4L2_subdev_call方法访问Video从设备，同时Video从设备可以
通过notify回调方法通知主设备发生了事件。

2. The main code file (Linux version 4.19 kernel)

  The driver source code of v4L2 is in the kernel/drivers/media/v4l2-core directory, and the main code files are:

(1)v4l2-dev.c //视频设备硬件的操作，包含video_device的注册、释放等，主要包括
//以下函数：
	videodev_init
		register_chrdev_region
		class_register
	videodev_exit
		class_unregister
		unregister_chrdev_region
	__video_register_device
	video_unregister_device
	
(2)v4l2-common.c //一些通用操作，V4l2的子设备一般是摄像头和摄像头控制器，
//它们和主机的控制操作是通过i2c总线完成的。V4l2驱动框架中跟i2c相关的代码在
//v4l2_common.c中
	v4l2_ctrl_query_fill
	v4l2_i2c_subdev_init
	v4l2_i2c_new_subdev_board
	v4l2_i2c_new_subdev
	v4l2_i2c_subdev_addr
	v4l2_i2c_tuner_addrs
	v4l2_spi_subdev_init
	v4l2_spi_new_subdev
	clamp_align
	v4l_bound_align_image
	__v4l2_find_nearest_size
	v4l2_get_timestamp
	v4l2_g_parm_cap
	v4l2_s_parm_cap
	
(3)v4l2-device.c //V4L2的设备支持，主要是注册v4I2_device，包括以下函数：
	v4l2_device_register
	v4l2_device_unregister
	
	v4l2_device_put
	v4l2_device_release
	
	v4l2_device_register_subdev
	v4l2_device_unregister_subdev
	
	v4l2_device_register_subdev_nodes
	v4l2_device_release_subdev_node	

(4)v4l2-ioctl.c  //处理V4L2的ioctl命令的一个通用的框架。

(5)v412-subdev.c //v4l2子设备

(6)v4l2-mem2mem.c //使用videobuf缓冲区的设备辅助函数。

V4L2 buffer management

	V4L2维护着两个缓冲队列:一个用于驱动（INPUT队列)，另一个用于用户程序
（OUTPUT队列）。
	缓冲区（由VIDIOC_REQBUFS命令申请）被用户空间的应用程序放入驱动的队
列中（通过VIDIOC_QBUF ioctl命令）以便填充数据。驱动按顺序填充缓冲区后，
缓冲区由INPUT队列放入OUTPUT队列。

	当用户程序调用VIDIOC_DQBUF命令后，驱动会在OUTPUT队列中寻找可用的
缓冲，如果可用则推送到用户程序，不可用则等待直到有可用缓冲后再推送给用
户程序。缓冲区使用完后，必须调用VIDIOC_QBUF将缓冲区重新加入INPUT队列
以便再次填充数据。

	注意驱动程序会独立自主的填充INPUT队列中的缓冲区，如果用户程序对缓
冲数据使用不及时，INPUT队列被填满，驱动暂停等待，会产生丢帧。

2. How to write V4L2 driver

1. How to write a device driver?

  Taking the character device driver as an example, please read an article I wrote before: Three Implementation Methods of Character Device Drivers

2. The main structure of Video equipment

  struct v4l2_device : A hardware device may contain multiple sub-devices. For example, a TV may have a VBI device or an FM tunneler in addition to a capture device. And v4l2_device is the root node of all these devices, responsible for managing all sub-devices.

/ * *
*struct v4l2_device -用于V4L2设备驱动程序的主结构
*
* @dev:指向设备结构体的指针。
* @mdev:指向结构体media_device的指针，可以为NULL。
* @subdevs:用于跟踪已注册的子设备
* @lock:锁定这个结构体;如果该结构嵌入到一个更大的结构中，驱动程序也可以使用
* 该结构。
* @name:唯一的设备名称，默认为驱动器名称+总线ID
* @notify:通知进行了某个操作(某些子设备被调用)
* @ctrl_handler:控制处理程序。可能是NULL。
* @prio:设备的优先级状态
* @ref:跟踪对这个结构体的引用。
* @release:当ref计数变为0时调用的释放函数。
*
* V4L2设备的每个实例都应该创建v4l2_device结构体，无论是独立的还是嵌入到
* 更大的结构体中。
*
*它允许轻松访问子设备(参见V4L2 -subdev.h)，并提供基本的V4L2设备级支持。
*
* . .注意::
*
* #) @dev->driver_data指向该结构体。
* #)如果没有父设备，@dev可能是%NULL
* /
struct v4l2_device {
    
    
	struct device *dev;
	struct media_device *mdev;
	struct list_head subdevs;
	spinlock_t lock;
	char name[V4L2_DEVICE_NAME_SIZE];
	void (*notify)(struct v4l2_subdev *sd,
			unsigned int notification, void *arg);
	struct v4l2_ctrl_handler *ctrl_handler;
	struct v4l2_prio_state prio;
	struct kref ref;
	void (*release)(struct v4l2_device *v4l2_dev);
};

  struct video_device : The main function of this structure is to provide /dev/videoX or /dev/v4l-subdevx device nodes, and at the same time abstract the capture interface to describe a frame-out device. In addition, the Video sub-device is also inherited from this structure. This structure contains operation object pointers to v4l2_file_operations, v4l2_ioctl_ops, etc.

/ * *
* struct video_device——用于创建和管理V4L2设备节点的结构。
*
* @entity: &struct media_entity
* @intf_devnode:指向&struct media_intf_devnode的指针
* @pipe: &struct media_pipeline
* @fops:指向视频设备的&struct v4l2_file_operations的指针
* @device_caps: v4l2_capabilities中使用的设备能力
* @dev: &struct设备用于视频设备
* @cdev:字符设备
* @v4l2_dev:指向struct v4l2_device父设备的指针
* @dev_parent:指向&结构设备父设备的指针
* @ctrl_handler:与该设备节点关联的控制处理程序。可能为NULL。
* @queue: &struct vb2_queue与该设备节点相关联。可能为NULL。
* @prio:指向带有设备优先级状态的struct v4l2_prio_state的指针。
* 如果为NULL，则使用v4l2_dev->prio。
* @name:视频设备名称
* @vfl_type: V4L设备类型，由&enum vfl_devnode_type定义
* @vfl_dir: V4L接收器、发射器或m2m
* @minor:设备节点“minor”。如果注册失败，则设置为-1
* @num:视频设备节点编号
* @flags:视频设备标志。使用bitops来设置/清除/测试标志。
* 包含一组&enum v4l2_video_device_flags。
* @index:属性用于区分一个物理设备上的多个索引
* @fh_lock:对所有v4l2_fhs进行锁
* @fh_list: struct v4l2_fh的列表
* @dev_debug:内部设备调试标志，驱动程序不使用
* @tvnorm:支持的电视规范
*
* @release:视频设备release()回调函数
* @ioctl_ops:带有ioctl回调函数的指向&struct v4l2_ioctl_ops的指针
*
* @valid_ioctls:该设备有效的ioctl的位图
* @lock:指向& &struct mutex 序列化锁的指针
*
* . .注意::
*只有在无法从@v4l2_dev推导出@dev_parent时才设置它。
* /
* 
struct video_device
{
    
    
#if defined(CONFIG_MEDIA_CONTROLLER)
	struct media_entity entity;
	struct media_intf_devnode *intf_devnode;
	struct media_pipeline pipe;
#endif
	const struct v4l2_file_operations *fops;

	u32 device_caps;

	/* sysfs */
	struct device dev;
	struct cdev *cdev;

	struct v4l2_device *v4l2_dev;
	struct device *dev_parent;

	struct v4l2_ctrl_handler *ctrl_handler;

	struct vb2_queue *queue;

	struct v4l2_prio_state *prio;

	/* device info */
	char name[32];
	enum vfl_devnode_type vfl_type;
	enum vfl_devnode_direction vfl_dir;
	int minor;
	u16 num;
	unsigned long flags;
	int index;

	/* V4L2 file handles */
	spinlock_t		fh_lock;
	struct list_head	fh_list;

	int dev_debug;

	v4l2_std_id tvnorms;

	/* callbacks */
	void (*release)(struct video_device *vdev);
	const struct v4l2_ioctl_ops *ioctl_ops;
	DECLARE_BITMAP(valid_ioctls, BASE_VIDIOC_PRIVATE);

	struct mutex *lock;
};

3. How to write V4L2 driver

(1)分配/设置/注册v4l2_device(调用函数注册v4l2_device_register),有辅助作用，提供自旋锁以及引用计数
(2)分配video_device:video_device_alloc()或kzalloc()；
(3)设置video_device:.fops、.ioctl_ops、dev；
(4)注册video_device: video_register_device()

参考阅读：https://blog.csdn.net/seiyaaa/article/details/120199720

Video设备注册时的执行流程可总结如下:
1.设置设备注销时资源释放回调和v4l2_device结构体。

2.检查设备类型并确定设备节点基本名称。

3.设置设备类型、次设备号及设备节点数量。

4.将video_device结构体指针保存到全局video_device数组中。

5.根据设备类型验证哪那些ioctl函数可以使用。

6.分配字符设备结构体。

7.设置字符设备的操作函数集合为v4l2_fops。

8.将video设备注册为字符设备，并注册设备;

9.设置设备引用计数为0时的回调函数，该函数主要的工作是删除注册的字符设备，
回调v4l2_device中的release函数（通常是video_device_release函数）释放
video_device结构体内存，最后减少v4l2_device的引用计数;

10.增加video_device所属v4l2_device的引用计数。

11.设置已注册标志V4L2_FL_REGISTERED。

Video device access process

(1）首先通过系统调用访问/dev/videox用户空间设备节点。

(2）进入到内核空间，访问字符设备struct file_operations中的方法。对于Video
设备，该操作集合被v4L2子系统初始化为v4l2_fops集合。

(3）通过v4L2子系统提供的v4l2_fops集合，可直接调用底层驱动实现的Video主设备
struct v4l2_file_operations方法，对于ioctl方法，则需要借助中间函数
__video_do_ioctl调用底层驱动实现的struct v4l2_ioctl_ops中的ioctl功能。
struct v4l2_file_operations方法和struct v4l2_ioctl_ops方法属于主设备
方法，需要主设备的驱动实现。

( 4) struct v4l2_file_operations和struct v4l2_ioctl_ops中的函数都可以
通过v4l2_subdev_call调用video从设备struct v4l2_subdev_core_ops、struct
v4l2_subdev_video_ops、struct v4l2_subdev_pad_ops等方法，这些方法都要在
从设备驱动中实现。

Three, V4L2 debugging tools

1、v4l2-ctl

  To facilitate development, common Linux distributions come with a v4l2-ctl command line tool that can be used to test the camera subsystem. This tool can list the devices in the system, query device capabilities, adjust device properties, set pixel format, resolution, frame rate, etc., and also perform actions such as capturing images.

  Reference reading: Basic usage of v4l2-ctl

2、dev_debug

V4L2调试
由丁video系统的配置较复杂，为便于调试，V4L2提供了简单但庞大的用户空间调试手段，用于跟踪框架或用户空间API的信息。
椎架的调试信息可通过下述命令开启（通过dmesg查看):
# echo 0x3 > /sys/module/videobuf2_v4l2/parameters/debugt
# echo 0x3 > /sys/module/videobuf2_common/parameters/debug

V4L2的用户空问API跟踪通过下述命令开启:
# echo 0x3 >/sys/class/video4linux/video0/dev_debug

3、v4l2-compliance

  For Video devices to work normally, driver compatibility is an important aspect. The v4l2-compliance tool can judge its driver compatibility by testing various aspects of V4L2 devices.

  For usage, please read: v4l2-compliance of v4l-utils

  After a preliminary understanding of the driver framework of V4L2, the next article will use "Virtual Camera Driver: drivers\media\platform\vivid" for detailed analysis and interpretation, further in-depth understanding, so stay tuned.

The content of this article is mainly from Baidu Encyclopedia and teacher Wei Dongshan's course notes, if there is any infringement, please contact to delete! Welcome everyone to guide and communicate in the comment area! ! !