[V4L2] Brief description of V4L2 framework

Table of Contents of Series Articles

[V4L2] Brief introduction of V4L2 framework
[V4L2] Driver structure of V4L2 framework
[V4L2] V4L2 sub-device

---

Introduction: V4L2 is a set of video framework specially designed for linux devices. Its main framework is in the linux kernel, which can be understood as the video source capture driver framework on the entire linux system. It is widely used in embedded devices, mobile terminals, and personal computer equipment. Coding products on the market such as: SDV, mobile phones, IPC, and driving recorders will use this framework for video collection. Of course, there are more powerful manufacturers. Just use a set of video collection framework implemented by yourself, which is the best among manufacturers. The following mainly refers to the linux-4.4 kernel documentation for a global introduction to the V4L2 framework.

Introduction to V4L2 framework

Almost all devices have multiple IC modules, which may be physical (for example, a USB camera contains ISP, sensor, etc.), or abstract (such as the abstract topology in a USB device), and they are under the /dev directory Several device nodes are generated, and these IC modules also create some non-v4l2 devices: DVB, ALSA, FB, I2C, and input devices. It is precisely due to the complexity of the hardware that the v4l2 driver has also become very complex.

In particular, the v4l2 driver needs to support IC modules for audio/video mixing/encoding operations, which makes the v4l2 driver even more complicated. Usually, some IC modules are connected to the main bridge driver through one or more I2C buses, while other buses are still available. These ICs are called 'sub-devices'. For example, the sensor sensor in the camera device uses I2C to Communicate commands and use interfaces such as MIPI or LVDS for image data transmission.

For a long time, the framework (referring to the old V4L2 framework) was limited to creating v4l device nodes and video_buf through the video_device structure to process video data. This means that all drivers must set up the device instance and map it to the child device. Sometimes these steps are very complex and difficult to complete correctly, and some drivers are never written to follow these steps correctly. Due to the lack of a framework, a lot of common code cannot be refactored, resulting in this part of the code being rewritten and inefficient.

Therefore, this framework abstractly constructs the code required by all drivers and encapsulates it into modules, simplifying the reconstruction of common code for device drivers. v4l2-pci-skeleton.c is a very good reference routine. It is a driver framework for PCI capture cards. This routine demonstrates how to use the v4l2 driver framework, and this routine can be used as a driver template for a PCI video capture card. At the very beginning, you can also refer to this code writing method to make contact. Of course, the most suitable code is the code drivers/media/video/omap3ispin the folder. This code can basically be used as a complete input device example code (because it includes ISP, CSI, video and other devices, and has a complete data flow pipeline, which uses almost all aspects of the V4L2 framework, which is of great reference value) to write your own device driver code for reference.

---

V4L2 Framework Blueprint

Blueprint deconstruction

This is a very large picture, but I only selected one of them. This picture simplifies the sub-modules in V4L2 (simplified to only the name of the sub-module, without the introduction of the internal implementation), the big picture is as follows:

V4L2 Device topology
What do you think of this picture? It has the following key factors:

• v4l2_device: This is the overall structure of the entire input device. It can be considered as the entrance to the entire V4L2 framework, acting as the driver manager and entrance guardian. v4l2_subdev is derived from this structure. It is used for the overall management of video input devices. There are as many v4l2_device abstractions as there are input devices (for example, a USB camera as a whole can be regarded as a V4L2 device). Further down are the input sub-devices, which correspond to devices such as ISP, CSI, MIPI, etc. They are subordinate to a V4L2 device.

• media_device: Used for runtime data flow management, embedded in the V4L2 device, runtime means: a V4L2 device may have many sub-devices of the same type (two or more sensors, ISP, etc.), then in the device How do I know which type of sub-device my data flow needs to use when running? At this time, it is the turn of media_device. It establishes a virtual connection for this bunch of sub-devices, establishes a runtime pipeline (pipeline), and can dynamically change and manage the connected devices during runtime.

• v4l2_ctrl_handler: Control module, which provides sub-device (mainly video and ISP device) special effect operation interface in user space. For example, if you want to change the brightness, contrast, saturation, etc. of the output image, you can do it through this.

• vb2_queue: Provides a buffer transfer interface between the kernel and user space. The input device generates a bunch of image data. Where should it be placed in the kernel? How many can be placed? Is it continuous as a whole, continuous in sections, or physically discontinuous? How do users access it? It is all managed.

Hierarchical deconstruction

1. You can see the entry custom_v4l2_dev in the picture. It is a structure defined by the user. The important thing is not how it is defined. The important thing is that it contains a v4l2_device structure. As mentioned above, this structure provides an overview and strategizing. , equivalent to the position of the central management office, then the central government decides that it is the abstraction of the entire input device (such as the entire USB camera input device, such as the entire IPC camera input device). It also has a media_device structure, as mentioned above, it manages the data flow line and is engaged in structural route planning and management.

2. In the future, there is a linked list in v4l2_device, which maintains a huge sub-device chain, and all sub-devices are closely united with v4l2_device as the center through the bidirectional circular linked list structure of the kernel. In addition, media_device is one by one media_entity (you don’t need to understand its specific meaning now, you just need to know that it is an abstract body similar to the components on the circuit board), and media_entity has established its own small circle. In their small circle, the data flow flows unimpeded in a certain order and travels freely.

3. At the end, the /dev/videoX device node is abstracted, which is the role of the Ministry of Foreign Affairs, which is responsible for providing a communication hub between the kernel and user space. It should be noted that the essence of the device node is still a character device, and its internal set of operations is the same as that of the character device, but it is just a layer of encapsulation.

4. So far, a rough four-layer structure of V4L2 has been abstracted, as shown in the following figure: