I. Overview
Video4Linux2 is a kernel driver framework for video devices in the Linux kernel, which provides a unified interface for the upper layer to access the underlying video device. All subsystems in the kernel abstract the differences of the underlying hardware, provide a unified interface for the upper layer and extract common codes to avoid code redundancy and other benefits. Just like the bosses of a company generally do not directly talk to the bottom employees, but to the department managers to understand the situation. One is because there are a large number of low-level dicks, their opinions are different, and the wording is inaccurate. The department managers will summarize the situation afterwards. Report upward; second, the boss's time is precious.
V4L2 supports three types of devices: video input and output devices, VBI devices, and radio devices (in fact, more types of devices are supported, which will not be discussed for the time being). VideX, radioX and vbiX device nodes will be generated in the /dev directory respectively. Our common video input device is mainly a camera, which is also the main analysis object of this article.
1.1 The video input device in Linux system mainly includes the following four parts:
The core of the character device driver: V4L2 itself is a character device, with all the characteristics of a character device, exposing the interface to the user space;
V4L2 driver core: It is mainly to build a framework of standard video device drivers in the kernel, and provide a unified interface function for video operations;
Platform V4L2 device driver: Under the V4L2 framework, the V4L2 driver part related to the platform according to the characteristics of the platform itself, including registration of video_device and v4l2_dev.
Specific sensor driver: Mainly power on, provide working clock, video image cropping, stream IO start, etc., realize various device control methods for the upper layer to call and register v4l2_subdev.
1.2 The core source code of V4L2 is located in drivers/media/v4l2-core. The functions implemented by the source code can be divided into four categories:
Core module implementation: implemented by v4l2-dev.c, mainly used to apply for character major device number, register class and provide video device registration and cancellation related functions;
V4L2 framework: implemented by v4l2-device.c, v4l2-subdev.c, v4l2-fh.c, v4l2-ctrls.c and other files to build the V4L2 framework;
Videobuf management: realized by videobuf2-core.c, videobuf2-dma-config.c, videobuf2-dma-sg.c, videobuf2-memops.c, videobuf2-vmalloc.c, v4l2-mem2mem.c and other files to complete the videobufffer Assignment, management and cancellation.
Ioctl framework: It is implemented by the v4l2-ioctl.c file to build the framework of V4L2 ioctl.
Two, V4L2 frame
2.1 Structural frame diagram
The structures v4l2_device, video_device, v4l2_subdev and v4l2_fh are the main elements of the framework. The following figure is the structure diagram of the V4L2 frame:
From the above figure, the V4L2 framework is a standard tree structure, v4l2_device acts as the parent device, and manages all the child devices registered under it through a linked list. These devices can be GRABBER, VBI and RADIO. v4l2_subdev is a sub-device. The v4l2_subdev structure contains ops and ctrls that operate on the device. This part of the code is related to the hardware and needs to be implemented by the driver engineer according to the hardware. Camera devices need to implement control of power on and off, read ID, saturation, contrast The interface function of opening and closing the video data stream.
Video_device is used to create child device nodes and expose the interface of the operating device to the user space. v4l2_fh is the file handle of each sub-device. It is set when opening the device node file to facilitate the upper index to v4l2_ctrl_handler. v4l2_ctrl_handler manages the ctrls of the device. These ctrls (camera devices) include adjustment of saturation, contrast, and white balance.
2.2 v4l2_device
v4l2_device acts as the parent device of all v4l2_subdev in the v4l2 framework and manages the child devices registered under it. The following is the prototype of the v4l2_device structure (with unrelated members removed):
struct v4l2_device {
struct list_head subdevs; //用链表管理注册的 subdev
char name[V4L2_DEVICE_NAME_SIZE]; //device 名字
struct kref ref; //引用计数
... ...
};It can be seen that the main function of v4l2_device is to manage the sub-devices registered under it to facilitate the system to find and reference.
Registration and cancellation of v4l2_device:
int v4l2_device_register(struct device *dev, struct v4l2_device *V4l2_dev);
static void v4l2_device_release(struct kref *ref);2.3 V4l2_subdev
v4l2_subdev stands for sub-device and contains the related attributes and operations of the sub-device. Let's first look at the structure prototype:
struct v4l2_subdev {
struct v4l2_device *v4l2_dev; //指向父设备
const struct v4l2_subdev_ops *ops; //提供一些控制 v4l2 设备的接口
const struct v4l2_subdev_internal_ops; *internal_ops; //向 v4l2 框架提供的接口函数
//subdev 控制接口
struct v4l2_ctrl_handler *ctrl_handler;
/* name nust be unique */
char name[V4L2_SUBDEV_NAME_SIZE];
/* subdev device node */
struct video_device *devnode;
};Each sub-device driver needs to implement a v4l2_subdev structure. v4l2_subdev can be embedded in other structures or used independently. The structure contains members v4l2_subdev_ops and v4l2_subdev_internal_ops that operate on sub-devices.
The prototype of the v4l2_subdev_ops structure is as follows:
struct v4l2_subdev_ops {
const struct v4l2_subdev_core_ops *core; //视频设备通用的操作：初始化、加载FW、上电和 RESET 等
const struct v4l2_subdev_tuner_ops *tuner; //tuner 特有的操作
const struct v4l2_subdev_audio_ops *audio; //audio 特有的操作
const struct v4l2_subdev_video_ops *video; //视频设备的特有操作：设置帧率，裁剪图像、开关视频流等
... ...
};Video devices usually need to implement core and video members. The operations in these two ops are optional, but for video streaming devices video->s_stream (open or close stream IO) must be implemented.
The prototype of the v4l2_subdev_internal_ops structure is as follows:
struct v4l2_subdev_internal_ops {
//当 subdev 注册时被调用，读取 IC 的 ID 来进行识别
int (*registered)(struct v4l2_subdev *sd);
void (*unsigned)(struct v4l2_subdev *sd);
//当设备节点被打开时调用，通常会给设备上电和设置视频捕捉 FMT
int (*open)(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh);
int (*close)(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh);
};v4l2_subdev_internal_ops is an interface provided to the V4L2 framework and can only be called by the V4L2 framework layer. When registering or opening a sub-device, perform some auxiliary operations.
Registration and deregistration of subdev: When we have implemented all the members that need to be implemented in v4l2_subdev, we can call the following function to register the sub-device to the V4L2 core layer:
int v4l2_device_register_subdev(struct v4l2_device *v4l2_dev, struct v4l2_subdev *sd);When the child device is unloaded, the following function can be called to log out:
void v4l2_device_unregister_subdev(struct v4l2_subdev *sd);2.4 video_device
The video_device structure is used to generate device node files in the /dev directory, exposing the interface of operating the device to the user space.
struct video_device {
const struct v4l2_file_operations *fops;
/* sysfs */
struct device dev; /* v4l device */
struct cdev *cdev; // 字符设备
/* Seteither parent or v4l2_dev if your driver uses v4l2_device */
struct device *parent; /* device parent */
struct v4l2_device *v4l2_dev; /* v4l2_device parent */
/* Control handler associated with this device node. May be NULL */
struct v4l2_ctrl_handler *ctrl_handler;
/* 指向 video buffer 队列 */
struct vb2_queue *queue;
int vfl_type; /* device type */
int minor; //次设备号
/* V4L2 file handles */
spin lock_t fh_lock; /* lock for all v4l2_fhs */
struct list_head fh_list /* List of struct v4l2_fh */
/* ioctl 回调函数集，提供 file_operations 中的 ioctl 调用 */
const struct v4l2_ioctl_ops *ioctl_ops;
... ...
};video_device allocation and release, used to allocate and release the video_device structure:
struct video_device *video_device_alloc(void);
void video_device_release(struct video_device *vdev);Video_device registration and deregistration. After implementing the relevant members of the video_device structure, you can call the following interface to register:
static inline int __must_checkvideo_register_device(struct video_device *vdev, int type, int nr);
void void_unregister_device(struct video_device *vdev);vdev: video_device that needs to be registered and unregistered;
type: Device type, including VFL_TPYE_GRABBER, VFL_TYPE_VBI, VFL_TYPE_RADIO and VFL_TYPE_SUBDEV.
nr: Device node name number, such as /dev/video[nr].
2.4 v4l2_fh
v4l2_fh is a unique operation method used to save sub-devices, that is, the v4l2_ctrl_handler to be analyzed below. The kernel provides a set of v4l2_fh operation methods, and v4l2_fh registration is usually performed when the device node is opened.
Initialize v4l2_fh, add v4l2_ctrl_handler to v4l2_fh:
void v4l2_fh_init(struct v4l2_fh *fh, struct video_device *vdev);
Add v4l2_fh to video_device to facilitate the core layer to call:
void v4l2_fh_add(struct v4l2_fh *fh);2.5 v4l2_ctrl_handler
v4l2_ctrl_handler is a structure used to save the set of sub-device control methods. For video devices, these ctrls include setting brightness, saturation, contrast, and sharpness. The ctrls are saved in a linked list. You can add ctrls to the linked list through the v4l2_ctrl_new_std function.
struct v4l2_ctrl *v4l2_ctrl_new_std(struct v4l2_ctrl_handler *hdl, const struct v4l2_ctrl_ops *ops, u32 id, s32 min, s32 max, u32 step, s32 def);hdl is the initialized v4l2_ctrl_handler structure;
ops is the v4l2_ctrl_ops structure, which contains the specific implementation of ctrls;
id is the command passed through the arg parameter of IOCTL, defined in the v4l2-controls.h file;
min and max are used to define the scope of an operation object. Such as:
v4l2_ctrl_new_std(hdl, ops, V4L2_CID_BRIGHTNESS, -208, 127, 1, 0);The user space can be called to v4l2_ctrl_handler through the VIDIOC_S_CTRL instruction of ioctl, and the id is passed through the arg parameter.
Three, ioctl framework
You may observe that user space operations on V4L2 devices are basically realized by ioctl. V4L2 devices have a large number of operable functions (configuration registers), so the ioctl of V4L2 is also very large. What kind of framework is it and how is it implemented?
The Ioctl framework is implemented by the v4l2_ioctl.c file. The structure defined in the file is the v4l2_ioctls array, which can be regarded as the relationship table between the ioctl command and the callback function. The user space calls the system call ioctl, passes down the ioctl instruction, and then finds the corresponding callback function by looking up the relationship table.
The following are two items of the intercepted array:
IOCTL_INFO_FNC(VIDIOC_QUERYBUF, v4l2_querybuf, v4l_printf_buffer, INFO_FL_CLEAR(v4l2_buffer, length)),
IOCTL_INFO_STD(VIDIOC_G_FBUF, vidioc_g_fbuf, v4l_print_framebuffer, 0),The kernel provides two macros (IOCTL_INFO_FNC and IOCTL_INFO_STD) to initialize the structure. The parameters are ioctl instruction, callback function or v4l2_ioctl_ops structure member, debug function, and flag. If the callback function is a member of the v4l2_ioctl_ops structure, use IOCTL_INFO_STD; if the callback function is implemented by v4l2_ioctl.c, use IOCTL_INFO_FNC.
The flow chart of IOCTL call is as follows:
The user space obtains the file structure of the file by opening the device node under the /dev/ directory, and passes cmd and arg to the kernel through the system call ioctl. After a series of calls, the _video_do_ioctl function is finally called, and then v4l2_ioctls[] is retrieved through cmd to determine whether it is INFO_FL_STD or INFO_FL_FUNC. If it is INFO_FL_STD, it will directly call the video_device->v4l2_ioctl_ops function set in the video device driver. If it is INFO_FL_FUNC, it will first call the standard callback function implemented by v4l2 itself, and then call the video_device->v4l2_ioctl_ops or v4l2_fh->v4l2_ctrl_handler function set according to the arg.
Expansion:
1. v4l2 control framework: Since video input is involved, there will be many ISP-related effects, such as contrast, saturation, color temperature, white balance, etc., these are common and necessary control items, and most of them only You need to set an integer value. V4L2 very close to providing such a number of interfaces for us for use (can be said to be a very intimate), inside the kernel, these controls are abstracted as one of the control ID, respectively
V4L2_CID_XXXnamed.2. v4l2 ioctl framework: user space operations on V4L2 devices are basically realized through wrong ioctl, such as VIDIOC_QUERYCAP, VIDIOC_S_FMT, VIDIOC_DQBUF, etc.
Four, IO access
V4L2 supports three different IO access methods (other access methods are also supported in the kernel and will not be discussed for the time being):
- Read and write are basic frame IO access methods. Each frame of data is read through read. The data needs to be copied between the kernel and the user. This method of access may be very slow;
- The memory mapped buffer (V4L2_MEMORY_MMAP), which actually opens up the buffer in the kernel space, is mapped to the user address space by the application through the mmap() system call. These buffers can be large and continuous DMA buffers, virtual buffers created by vmalloc(), or buffers directly opened in the device's IO memory (if supported by the hardware);
- The user space buffer (V4L2_MEMORY_USERPTR) enables the user space application to open up a buffer, and exchange buffer pointers between the user and the kernel space. Obviously, mmap() is not needed in this case, but the driver will effectively support user space buffers, and its work will be more difficult.
The read and write methods belong to the frame IO access method. Each frame must be operated through IO, which requires data copy between the user and the kernel. The latter two are stream IO access methods, which do not require memory copy, and the access speed is relatively fast. The memory-mapped buffer access method is a more commonly used method.
4.1 Memory mapped buffer mode
Data streaming at the hardware layer
The image data captured by the camera sensor is transmitted to CAMIF (camera interface) through the parallel port or MIPI, and CAMIF can adjust the image data (flip, crop, format conversion, etc.). Then the DMA controller sets the DMA channel to request AHB to transfer the image data to the allocated DMA buffer.
After the image data is transferred to the DMA buffer, the mmap operation maps the buffer to the user space, and the application can directly access the data in the buffer.
4.2 vb2_queue
In order for the device to support stream IO, the driver needs to implement struct vb2_queue, look at this structure:
struct vb2_queue {
enum v4l2_buf_type type; //buffer 类型
unsigned int io_modes; //访问 IO 的方式：mmap、userptr etc
const struct vb2_ops *ops; //buffer 队列操作函数集合
const struct vb2_mem_ops *mem_ops; //buffer memory 操作集合
struct vb2_buffer *bufs[VIDEO_MAX_FRAME]; //代表每个 buffer
unsigned int num_buffers; //分配的 buffer 个数
... ...
};vb2_queue represents a video buffer queue, vb2_buffer is a member of this queue, vb2_mem_ops is the operation function set of buffer memory, and vb2_ops is used to manage the queue.
4.3 vb2_mem_ops
vb2_mem_ops contains the memory operation methods of memory mapped buffer and user space buffer:
struct vb2_mem_ops {
void *(*alloc)(void *alloc_ctx, unsigned long size); //分配视频缓存
void (*put)(void *buf_priv); //释放视频缓存
//获取用户空间视频缓冲区指针
void *(*get_userptr)(void **alloc_ctx, unsigned long vaddr, unsigned long size, int write);
void (*put_userptr)(void *buf_priv); //释放用户空间视频缓冲区指针
//用于缓冲同步
void (*prepare)(void *buf_priv);
void (*finish)(void *buf_priv);
void *(*vaddr)(void *buf_priv);
void *(*cookie)(void *buf_priv);
unsigned int (*num_users)(void *buf_priv); //返回当期在用户空间的buffer数
int (*mmap)(void *buf_priv, struct vm_area_struct *vma); //把缓冲区映射到用户空间
};This is a very large structure. So many structures need to be implemented. Fortunately, one of the kernels helped us realize it. Three types of video caching operations are provided:
- Continuous DMA buffer;
- Distributed DMA buffer;
- Buffer created by vmalloc.
They are implemented by the videobuf2-dma-contig.c, videobuf2-dma-sg.c and videobuf-vmalloc.c files respectively, and can be used according to the actual situation.
4.4 vb2_ops
vb2_ops is a collection of functions used to manage the buffer queue, including queue and buffer initialization
struct vb2_ops {
//队列初始化
int (*queue_setup)(struct vb2_queue *q, const struct v4l2_format *fmt, unsigned int *num_buffers, unsigned int *num_planes, unsigned int sizes[], void *alloc_ctxs[]);
//释放和获取设备操作锁
void (*wait_prepare)(struct vb2_queue *q);
void (*wait_finish)(struct vb2_queue *q);
//对 buffer 的操作
int (*buf_init)(struct vb2_buffer *vb);
int (*buf_prepare)(struct vb2_buffer *vb);
int (*buf_finish)(struct vb2_buffer *vb);
int (*buf_cleanup)(struct vb2_buffer *vb);
//开始视频流
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
//停止视频流
int (*stop_streaming)(struct vb2_queue *q);
//把 VB 传递给驱动
void (*buf_queue)(struct vb2_buffer *vb);
};vb2_buffer is the basic unit of the buffer queue, and v4l2_buffer embedded in it is the core member. When streaming IO is started, the frame is transmitted between the application and the driver in the format of v4l2_buffer. A buffer can have three states:
-
In the incoming queue of the driver, the driver will process the buffer in this queue, and the user space will put the buffer into the queue through IOCTL:VIDIOC_QBUF. For a video capture device, the buffer of the incoming queue is empty, and the driver will fill it with data;
-
In the outgoing queue of the driver , these buffers have been processed by the driver. For a video capture device, the buffer has been filled with video data and is waiting for the user space to claim it;
-
The queue of the user space status has been transmitted to the user space buffer through IOCTL:VIDIOC_DQBUF. At this time, the buffer is owned by the user space and cannot be accessed by the driver.
The switching of these three states is shown in the figure below:
The structure of v4l2_buffer is as follows:
struct v4l2_buffer {
__u32 index; //buffer 序号
__u32 type; //buffer 类型
__u32 byteused; //缓冲区已使用 byte 数
__u32 flags;
__u32 field;
struct timeval timestamp； //时间戳，代表帧捕获的时间
struct v4l2_timecode timecode;
__u32 sequence;
/* memory location */
__u32 memory; //代表缓冲区是内存映射缓冲区还是用户空间缓冲区
union {
__u32 offset; //内核缓冲区的位置
unsigned long userptr; //缓冲区的用户空间地址
struct v4l2_plane *planes;
__s32 fd;
}m;
__u32 length; //缓冲区大小，单位 byte
};When the user space gets v4l2_buffer, the relevant information of the buffer can be obtained. byteused is the number of bytes occupied by image data:
- If it is the V4L2_MEMORY_MMAP method, m.offset is the starting address of image data storage in kernel space, which will be passed to the mmap function as an offset, and a buffer pointer p is returned through mmap mapping, p+byteused is the virtual address space of the image data in the process Occupied area
- If it is a user pointer buffer mode, the start address pointer of the image data that can be obtained is m.userptr, userptr is a user space pointer, userptr+byteused is the occupied virtual address space, and the application can directly access it.
Five, user space access equipment
The following is the process of accessing the video device (capture device) through the kernel mapping buffer.
1> Open the device file
fd = open(dev_name, O_RDWR /* required */ | O_NONBLOCK, 0);
//dev_name[ /dev/videoX ]2> Query the capabilities supported by the device
struct v4l2_capability cap;
ioctl(fd, VIDIOC_QUERYCAP, &cap);3> Set video capture format
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
fmt.fmt.pix.width = 640;
fmt.fmt.pix.height = 480;
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV; //像素格式
fmt.fmt.pix.field = V4L2_FIELD_INTERLACED;
ioctl(fd, VIDIOC_S_FMT, &fmt);4> Apply for buffer to the driver
struct v4l2_buffer req;
req.count = 4;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;
if(-1 == xioctl(fd, VIDIOC_REQBUFS, &req))5> Get the information of each buffer and map it to user space
struct buffer {
void *start;
size_t length;
} *buffers;
buffers = calloc(req.count, sizeof(*buffers));
for(n_buffers = 0; n_buffers < req.count; ++n_buffers){
struct v4l2_buffer buf;
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
buf.index = n_buffers;
if(-1 == xioctl(fd, VIDIOC_QUERYBUF, &buf))
errno_exit("VIDIOC_QUERYBUF");
buffers[n_buffers].length = buf.length;
buffers[n_buffers].start =
mmap(NULL /* start anywhere */,
buf.length,
PROT_READ | PROT_WRITE /* required */,
MAP_SHARED /* recommended */,
fd,
buf.m.offset);
}6> Put the buffer into the incoming queue, open the stream IO, and start video capture
for(i=0; i<n_buffers; ++i){
struct v4l2_buffer buf;
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
buf.index = i;
if(-1 == xioctl(fd, VIDIOC_QBUF, &buf))
errno_exit("VIDIOC_QBUF");
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
if(-1 == xioctl(fd, VIDIOC_STREAMON, &type))
}7> Call select to monitor the file descriptor, whether the data in the buffer is filled, and then check the video data
for(;;) {
fd_set fds;
struct timeval tv;
int r;
FD_ZERO(&fds);
FD_SET(fd, &fds);
/* timeout */
tv.tv_sec = 2;
tv.tv_usec = 0;
//监测文件描述是否变化
r = select(fd + 1, &fds, NULL, NULL, &tv);
if(-1 == r) {
if(EINTR == errno)
continue;
errno_exit("select");
}
if(0 == r){
fprintf(stderr, "select timeout\r\n");
exit(EXIT_FAILURE);
}
//对视频数据进行处理
if(read_frame())
break;
/* EAGAIN - continue select loop. */
}8> Take out the filled buffer, get the size of the video data, and then process the data. The buffer taken out here only contains the information of the buffer, and the video data is not copied.
buf.type = V4L2_BUF_TPYE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
if(-1 == ioctl(fd, VIDIOC_DQBUF, &buf)) //取出缓存
errno_exit("VIDIOC_QBUF");
process_image(buffers[buf.index].start, buf.byteused); //视频数据处理
if(-1 == xioctl(fd, VIDIOC_QBUF, &buf)) //然后又放入到传入队列
errno_exit("VIDIOC_QBUF");9> Stop video capture
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
ioctl(fd, VIDIOC_STREAMOFF, &type);10> Turn off the device
close(fd);Attached:
-
For specific usage examples, please refer to: vivi.c
-
blog: <V4L2 (2) In-depth analysis of virtual camera driver vivi> - https://blog.csdn.net/lizuobin2/article/details/53006927