【V4L2】V4L2 framework-v4l2 device

Table of Contents of Series Articles

【V4L2】V4L2 framework-v4l2 device

---

---

This article introduces the easier-to-understand backbone structure of V4L2, involving two core structures: v4l2_device and
v4l2_subdev. The article focuses on these two structures and
uses the omap3isp code of the Linux-4.4 kernel as an example to introduce them. The so-called introduction still plays a supporting role.

The "routine" below refers to the omap3isp routine.

V4L2 Framework Supplement

First look at the picture:

V4L2 sub-module division.
This is not an official division. I divided it based on my own feelings. That's it. Then the theme of this article is the sub-device system, which is the third "subdev" in the above figure. It can also be seen from the figure that this article mainly talks about how to manage many input devices. Note: It is all devices, not the devices on the data flow path at runtime.

---

main device

The main device is represented abstractly using v4l2_device. This routine uses the device tree for device resolution, and uses the platform driver for the corresponding driver probe. There is the isp_probe function in the file drivers/media/platform/omap3isp/isp.c. The first parameter definition of this function is isp_device. Recall the previous article, which mentioned that in most cases, the v4l2_device structure needs to be embedded into a larger structure. This In order to drive the custom structure, that is it, then its definition is as follows (delete the part that is not related to the analysis):

struct isp_device {
    
    
    struct v4l2_device v4l2_dev;
    struct v4l2_async_notifier notifier;
    struct media_device media_dev;

    ... ...
}

Among them, the one related to this section is struct v4l2_device v4l2_dev, which will serve as the manager of the entire omap3isp in series.

At the same time, its usage is consistent with the "embedded" mentioned above. struct isp_device is a driver-defined structure. This structure can be regarded as the device abstraction structure of the entire omap3isp. In other words: it represents the big omap3isp device of. You will see its usage below.

Register v4l2_device

There is a call function in the isp_probe function of isp.c isp_register_entities, and the content at the beginning is roughly as follows:

... ...
    isp->v4l2_dev.mdev = &isp->media_dev;
    ret = v4l2_device_register(isp->dev, &isp->v4l2_dev);
    if (ret < 0) {
    
    
        dev_err(isp->dev, "%s: V4L2 device registration failed (%d)\n",
            __func__, ret);
        goto done;
    }
... ...

In fact, you can see that the registration function does not add high-level devices to a linked list, but initializes structure members, such as initializing the head of the sub-device linked list, adding dev references, etc. Before all sub-devices are connected in series, v4l2_devicethe overall device must first be initialized and registered.

v4l2_subdev

This structure is an abstract sub-device and is used for sub-device management. It has several operations such as "initialization", "registering sub-device", "registering sub-device node", etc. Usually, the "initialization" function instances are defined inside each sub-device driver module, and the entities of the two functions "register sub-device" and "register sub-device node" are defined inside the parent device module, such as: isp .c in.

Above code (in isp_probe function):

ret = isp_initialize_modules(isp);
if (ret < 0)
    goto error_iommu;
ret = isp_register_entities(isp);
if (ret < 0)
    goto error_modules;

Let’s first look at isp_initialize_modulesthe implementation in the function:

static int isp_initialize_modules(struct isp_device *isp)
{
    
    
    int ret;

    ret = omap3isp_csiphy_init(isp);

    ret = omap3isp_csi2_init(isp);

... ...

    ret = omap3isp_h3a_aewb_init(isp);

    ret = omap3isp_h3a_af_init(isp);
... ...
    return 0;
... ...

Go into omap3isp_xxxx_initthe function and you will see similar v4l2_subdev_initfunction calls. The article should try to post as little code as possible and focus more on the implementation and usage mechanism. I believe you will be able to find the relevant code on your own.

So the structure is roughly like this:

1. You have an input device omap3isp, its manager is listed in a separate code file named isp.c;

2. Define a custom abstract structure to represent the omap3isp device, named isp_device, and embed v4l2_device internally as a sub-device management tool;

3. Abstract sub-devices - similar to csi, preview, 3a, etc. into a sub-device. Each sub-device has a code file named ispxxx.c. Each sub-device has its own abstract structure named isp_xxx_dev, and v4l2_subdev is embedded internally as a sub-device. Device abstract tool usage. At the same time, implement your own device initialization function, named xxx_init_eneities;

4. Call the xxx_init_entities of the sub-device in the probe function of the manager isp.c, and the sub-device initialization function will initialize v4l2_subdev;

5. Register v4l2_device in the manager's probe function, register the sub-device, and if necessary, register the sub-device node to generate /dev/nodeX in the user space;

6. You're done, now you can manage all sub-devices through v4l2_device. The framework itself provides very useful management methods and related callback functions, structure members, etc.

equipment management

The management of the device inevitably requires the interconnection and interoperability between the main device and the sub-device. Otherwise, there is no way to manage it. This section will introduce how to realize the data interconnection and interoperability between the main device and the sub-device.

Master device and sub-device intercommunication

After establishing the connection through the above steps, how to find the sub-device from the main device? How to find master device from sub device? How to get from v4l2_device to a custom main device abstract structure? How to get from v4l2_subdev to the subdevice custom structure?

• How to find sub-devices from the main device. First, you need to obtain the v4l2_device structure. Then you can use list_for_each_entry to traverse the sub-devices. There is a name member inside the sub-device structure, with a length of 32 bytes. This field is required to be the entire v4l2_device. It is unique among subordinates, so if you want to find a specified sub-device, you can compare the name field of the sub-device during traversal to see if it is what you want to find.

• How to find the main device from the sub-device. There is a pointer member of v4l2_dev in the structure of v4l2_subdev. This member will point to the v4l2_device member when the sub-device is registered. The registration function is v4l2_device_register_subdev. After this step is completed, the main device structure can be obtained by obtaining the v4l2_dev member inside the sub-device structure.

• How to instantiate the structure from the main device to the main device. You can see that there is no private pointer inside v4l2_device. So how to find the instantiation structure of the main device? At this time, you can obtain it through another side method. , for example, when defining the structure, put v4l2_device first in the structure member, and then obtain v4l2_device through v4l2_subdev, and then its address can be forced to be converted into a custom instantiated structure for the main device to achieve access.

• How to instantiate a structure from subdevice to subdevice There are two private pointers inside the subdevice: dev_priv, host_priv. The former one is easy to understand and use. When using it, just call v4l2_set_subdevdatathe function to point dev_priv to the sub-device instantiation structure, and then use it to v4l2_get_subdevdataobtain the structure data of the sub-device structure instantiation from v4l2_subdev. The latter one is not very easy to understand its use, but it can also v4l2_set_subdev_hostdata/v4l2_get_subdev_hostdatabe set/obtained through. The host is also the main control end. For example, the controller on the SOC side of a camera sensor can be used as the main control end. Another example is a camera that uses I2C for communication. The I2C controller on the SOC side of the sensor can be used as host_priv to control the behavior of the sub-device through I2C when necessary. Or you can simply use the structure instantiated by the main device as host data.

Information exchange between master and sub-device

This section uses multiple practical use cases to explain in depth various information exchange methods and scenarios. For example: How to control access to specific types of sub-devices? How does a child device return notifications to the master device?

• Access all sensor devices and close their data streams

1. When registering the sub-device, the relevant operation function should be provided, which is the v4l2_subdev_ops structure. In this example, we only set the s_stream member of its video member.

2. Provide the subdevice group id, which is the grp_id member of v4l2_subdev. Here we set it to an enumeration type that I assumed (you only need to ensure that this enumeration type is unique for the entire v4l2_device subordinate). I assume it is OMAP3ISP_CAMSENSOR.

3. I won’t go into details about initializing and registering sub-devices. The methods of initialization and registration have been mentioned before.

4. Execute v4l2_device_call_all(v4l2_device, OMAP3ISP_CAMSENSOR, video, s_stream, 0);. At this time, all sub-devices of the OMAP3ISP_CAMSENSOR group hanging under the name of v4l2_device will be traversed, and their s_stream module function will be called to close the data stream.

• After the data flow of the sub-device is closed, a notification is returned to the main device.

1. The notify member operation function of the main device needs to be provided.

2. Define the notification format, such as my own definition. The upper 8 bits indicate which sub-device, the second 8 bits indicate which type of operation (here is the video type ops), and the third 8 bits indicate the specific operation function (s_stream). , the lower 8 bits represent the operation value (0 turns off).

3. The sub-device calls the v4l2_subdev_notify function to send a formal notification. At this time, it can also take some parameters. It only needs to pass its address. The main and sub-device sides agree on the format of the data.

4. The main device performs relevant operations after receiving the notification.

Transportation hub video_device

This structure integrates the terminal module function of data flow management and is responsible for providing data exchange from kernel space to user space. It is a very important function and is indispensable.

Normally, all sub-devices can register a video_device structure to generate a device node in user space for user operation, but the difference is that only the module responsible for actually transmitting video data needs to register the video type device node name. (such as the DMA data terminal of the kernel input device data link), and other types of v4l-subdev can be used.

video_device is only bound and associated with v4l2_device, through which the resources of the entire sub-device network can be accessed.

How to register video type node

Use video_register_devicethe match VFL_TYPE_GRABBERparameter to register. At this time, when the function completes execution and returns, you can see the /dev/videoXdevice node in the shape of in the user space.

Note that you need to provide its operating function, similar to the following:

static struct v4l2_file_operations isp_video_fops = {
    
    
    .owner = THIS_MODULE,
    .unlocked_ioctl = video_ioctl2,
    .open = isp_video_open,
    .release = isp_video_release,
    .poll = isp_video_poll,
    .mmap = isp_video_mmap,
};

videobuf2How its members are implemented is not introduced in detail in this section. It will be introduced in detail in a later article.

How to register other types of nodes

For v4l2 input devices, use v4l2_device_register_subdev_nodesto register device nodes in batches. It will still call video_register_devicethe function internally, but it will use VFL_TYPE_SUBDEVthe type instead of the above VFL_TYPE_GRABBER. Then the device node name generated in the user space is v4l- subdevX.

In this case, the operating function of the registered device node is the default operating function defined in v4l2-device.c. Its definition is as follows:

const struct v4l2_file_operations v4l2_subdev_fops = {
    
    
    .owner = THIS_MODULE,
    .open = subdev_open,
    .unlocked_ioctl = subdev_ioctl,
#ifdef CONFIG_COMPAT
    .compat_ioctl32 = subdev_compat_ioctl32,
#endif
    .release = subdev_close,
    .poll = subdev_poll,
};

Conclusion

So far, basically the basic operations of the equipment have been introduced. As for some more advanced operations, they will be introduced in subsequent articles. As a preview, the next article is about media framework, step by step. After reading this article and practicing it, you can see that there is a /dev/video device node in the user space. You can write a small test case. When opening the device node, it traverses the sub-devices and types out the names of the sub-devices. You can also add more detailed information. In short, the purpose of this article is to implement a device topology under the v4l2_device management framework. You can print out this topology to complete the task perfectly.