This article gives a detailed introduction to V4L2's runtime data stream device management, including what is called [runtime device management], what is it used for, how to use it, and so on. The goal of this article is to master the coding usage and functional application of meida device.

一、media framework

1.1 Introduction

The related control API is in Documentation/DocBook/media/v4l/media-controller.xml. This document focuses on the implementation of the core test media framework. Note: You can't see anything by directly viewing it. Make htmldocs or other things in the root directory of the kernel will do. It's easy to view.

1.2. Control equipment at runtime

That is, the data flow line control after the device is started, just like a factory pipeline, the nodes (labeling, printing, and packaging) on the pipeline are just like the sub-devices in the input device. The device control at runtime is To achieve the effect of being able to control the node, for example, there are several machines for labeling, which one should be selected for this pipeline processing, whether to add the silk screen, which machine to add, and so on.

1.3, role

Provide real-time pipeline management. Pipeline is understood as a pipeline. Imagine a water pipe. The water inside is the data stream. The csi->isp->video in the input device forms a pipeline line. The media framework provides functions such as opening, closing, effect control, and node control of the pipeline.

1.4 How to use

The kernel mainly uses four structures to organize numerous nodes: media_device, media_entity, media_link, and media_pad. The entire media framework is used around these four structures, which will be introduced in detail below.

1.5, abstract device model

One of the purposes of the media framework is to discover device topology and configure it at runtime. In order to achieve this goal, the media framework abstracts hardware devices into entities, which are connected through links.

1. Entity: hardware device module abstraction (analogous to the various components and chips on the circuit board)

2, pad: hardware device port abstraction (analog components, pins on the chip)

3. Link: The connection abstraction of the hardware device, the two ends of the link are pads (connections between the pins of analog components)

#------------#                #------------#
|          __|__            __|__          |
|         |  |  |   link   |  |  |         |
|         | pad |<-------->| pad |         |
|         |__|__|          |__|__|         |
|            |                |            |
|   entity   |                |   entity   |
#------------#                #------------#

Imagine that if you need to establish a connection between entities, you need to store link and entity information in the pad. The link needs to store the pad and entity information, and the entity needs to store the link and pad information. It belongs to you and me. In your case.

Two, media equipment

A media device is represented by a media_device structure. Normally, the structure needs to be embedded in a larger device-defined structure, and most of the time, media_device and v4l2_device are at a parallel level, or the code of omap3isp is example:

struct isp_device {
    struct v4l2_device v4l2_dev;
    struct v4l2_async_notifier notifier;
    struct media_device media_dev;
    struct device *dev;
    u32 revision;
    ... ...
}

2.1. Use the following function to register the media device:

media_device_register(struct media_device *mdev);

The caller of the function needs to set the following structure members before registration (it is the caller's responsibility to initialize the structure in advance):

dev: Must point to a parent device, usually the device member of the platform device.
model: the name of the model.

The following members are optional:

serial: serial number, must be unique
bus_info: bus information, if it is a PCI device, it can be set to "PCI:"
hw_revision: hardware version. If possible, it should be formatted with the KERNEL_VERSION macro definition
driver_version: driver version. The name of the final device node is media[0-9], and the node number is automatically generated by the kernel.

2.2. Use the following function to uninstall the device

media_device_unregister(struct media_device *mdev);

It should be noted that uninstalling a device that has not been registered is ***unsafe***. There are several main reasons why personal viewing code guess is not safe:

If it is not registered, the entity member inside media_device may not be initialized. If its value is an indeterminate value, it will cause illegal access;
If it is not registered, the internal devnode members will not be initialized, and problems will occur during uninstallation.

三、entities、pads、links

3.1、entities

Entities are represented by a media_entity structure, which is usually embedded in a larger structure, such as v4l2_subdev or video_device structure (you do not need to allocate space by yourself, the structure is already included), of course, you can also directly allocate an entity . Use the following function to initialize the entity:

media_entity_init(struct media_entity *entity, u16 num_pads, struct media_pad *pads, u16 extra_links);

The parameters that need to be noted before executing the initialization function are:

num_pads: the number of pads, which is related to the structure of the driver sub-device;
pads: media_pad structure array, usually pad is embedded in the driver-defined structure, the array address is passed to this parameter, and the pad needs to be initialized in advance;
extra_links: This function allocates the number of links based on num_pads, this parameter indicates how many extra links are needed in addition to the pre-allocated number;
entity: The name, type, flags, revision, and group_id of media_entity need to be set before or after initialization. If the structure is embedded in a higher-level structure, these members may also be set by higher-level framework code. The id is filled in during registration (if the id member is set in advance, the preset value will be maintained during registration). The entity has related flags [flags] to identify its status and function, MEDIA_ENT_FL_DEFAULT means that this is a default entity. You can set the group ID of multiple entities to the same integer to identify that they belong to the same category. For the kernel, the group ID is useless, but the group ID will be passed to the user space when the entity is enumerated. Useful in certain situations in user space.

3.2、pads

The pad is represented by a media_pad structure, and the pads data is managed by the driver (array form). Pads use entity and array subscripts for unique identification. The id inside the entity will not be repeated, but the pad id between different entities may be repeated, so the index of the pad must be jointly confirmed by the entity and id.

Since the number of pads is known in advance (the chip you make, you must know how many pins it has), the media_pad structure is no longer dynamically allocated, and the driver should be responsible for managing the structure array (avoid dynamic allocation) . The driver must set the direction attribute of pads before the media_entity_init function is called. Pads has flags bits to indicate its attributes. You only need to set this member during initialization, and the rest is done by the media_entity_init function:

MEDIA_PAD_FL_SINK：    目的 pad
MEDIA_PAD_FL_SOURCE：  源 pad

3.3、links

The links are represented by a media_link structure. All the pads of each entity store all the links related to it. A link will be stored by the source pad and the destination pad respectively, so as to realize the traversal in both positive and negative directions. Use the following function to create links:

media_entity_create_link(struct media_entity *source, u16 source_pad, struct media_entity *sink, u16 sink_pad, u32 flags);

Links has some flags bits to identify its attributes:

MEDIA_LNK__FLENABLED：link 被使能，可以用来传输数据，多个 link 连接到同一个 sink pad 时，只有一个 link 可能被使能。
MEDIA_LNK_FL_IMMUTABLE：link 的使能状态不能在运行时被改变，一般情况下这两个标志位同时被设置。
MEDIA_LNK_FL_DYNAMIC：link 的状态是动态可变的。

Unlike pads, the number of links is not always confirmed in advance (sometimes on the circuit board you cannot fully confirm how many devices need to be connected to the pins, it is very likely that there will be temporary changes), so the media_entity_init function is based on The incoming parameters pre-allocate a certain amount of media_link structure, if it is not enough, it will be dynamically allocated in media_entity_create_link (if the number of links is greater than or equal to max_link, the number of links will be expanded).

Four, registration and uninstall

The driver needs to use the following functions to register and uninstall the entity (does not need to be executed manually, it is done in the v4l2_device_un/register_subdev function):

media_device_register_entity(struct media_device *mdev, struct media_entity *entity);
media_device_unregister_entity(struct media_entity *entity);

The kernel uses a unique positive integer to represent each entity (unique under the same media_device), and the driver can also specify the ID value of the entity by filling in the media_entity->id member, but it must be unique. If the IDs are automatically generated by the kernel, there is no guarantee that they are continuous. In fact, the IDs automatically generated by the kernel are assigned by the entity's media_device->entity_id ++, and the value is initialized to 1 in the media_device_register function Liman.

After unloading the entity, you need to call the following functions to release the related resources applied for, mainly to release the dynamically allocated media_link structure memory:

media_entity_cleanup(struct meida_entity *entity); //与 media_entity_init 结对使用

To traverse entities, you can perform the MEDIA_IOC_ENUM_ENTITIES system call in user space. You need to set the id of media_entity_desc to (0 | MEDIA_ENT_ID_FLAG_NEXT). During the loop, you only need to set id |= MEDIA_ENT_ID_FLAG_NEXT to complete the entity traversal process. If you need to enumerate the specified Entity, you need to set the id to the id value of the specified entity (the id of the kernel entity starts from 1), this can be seen in the entity registration function. The code example is as follows, I tried to streamline the posted code to prevent occupying too much space:

int enum_media_device_entities(int iFd)
{
    int iRet;
    struct media_entity_desc desc;
    
    desc.id = 0 | MEDIA_ENT_ID_FLAG_NEXT;
    while(1) {
        iRet = ioctl(iFd, MEDIA_IOC_ENUM_ENTITIES, &desc);
        if(iRet < 0) {
            MODULE_WRN("enum media entities end \r\n");
            break;
        }
        MODULE_DBG("entity name [%s]\r\n", desc.name);
        desc.id |= MEDIA_ENT_ID_FLAG_NEXT;
    }
    return 0;
}

int main(int argc, char *argv[])
{
    int iErr = 0, ivFd;
    ivFd = open("/dev/media0", O_RDWR);
    iErr = enum_media_device_entities(ivFd);
    
    close(ivFd);
  open_err:
    return iErr;
}

5. Graph traversal (depth first)

What does graph traversal do? It provides us with a way to access each specified entity at runtime. As for why we need access, it is because we may need to manage them at runtime .

The following functions can be used to traverse the entities belonging to the same media device (linear traversal, atypical graph traversal, that is, the same traversal method as the linked list):

struct media_entity *entity;
media_device_for_ench_entity(entity, mdev) {
    /* entity will point to each entity in turn */
    ... ...
}

The driver may need to traverse all accessible entities from a given entity through the enabled links. The media framework provides a depth-first API to accomplish this task. It should be noted that it is necessary to avoid traversing the closed-loop graph, otherwise it will fall into an infinite loop. In order to avoid this situation, the function limits the maximum traversal depth to MEDIA_ENTITY_ENUM_MAX_DEPTH. The latest definition of this macro is 16** [As of Linux-4.4. 138] **.

media_entity_graph_walk_start(struct media_entity_graph *graph, struct media_entity *entity);
meida_entity_graph_walk_next(struct media_entity_graph *graph);

When using, first initialize the graph with the first function, and then call the second function to traverse in a loop. After the traversal is completed, the second function will return NULL. The traversal process can be interrupted at any moment, and there is no need to call the cleanup function.

There is a corresponding helper function to find the link of two given pads, or to find another pad connected to it through one pad.

media_entity_find_link(struct media_pad *source, struct media_pad *sink);
media_entity_remote_pad(struct media_pad *pad);

Six, supplement the settings of links

The attributes of link can be changed at runtime, just call the following function to complete:

media_entity_setup_link(struct media_link *link, u32 flags);

The flags parameter is used to set the attributes of the specified link. The allowed attributes are from the MEDIA_LINK_FL_ENABLED attribute to the MEDIA_LINK_FL_ENABLE or MEDIA_LINK_FL_DISABLE flag. If the link is set with the MEDIA_LINK_FL_IMMUTABLE flag, it cannot be enabled or disabled. When a link is enabled or closed, the media framework will call the sink and link_setup on the source side into two calls for related settings. If the second call fails, the first call will also be restored.

The media device driver can set media_device->link_notify to point to a callback function, which will be called after the link_setup operation is completed. If any link is non-immutable, the entity driver needs to implement the link_setup operation by itself. The configuration of a link should not affect other links. If a link is connected to the sink pad and the link is enabled, then other links to the pad can no longer be enabled, and -EBUSY should be returned at this time.

6.1 pipeline given media style

The concept of pipeline has been introduced before and will not be repeated. Here is an explanatory diagram (in fact, this is not very clear and easy to understand, and there are more clear and easy to understand that cannot be released due to some reasons. Please use your imagination, according to the figure. The above description abstracts a picture by itself, as long as it only focuses on one point - pipeline is the abstraction of data flow links, I believe you):

When streaming is turned on, the driver should notify all entities on the pipeline to maintain the current state unchanged, and you can call the following function to complete the notification (this function will only call the validate callback on the sink side):

media_entity_pipeline_start(struct media_entity *entity, struct media_pipeline *pipe);

This function will mark all entities on the enable link connection as streaming state, whether directly or indirectly. The media_pipeline structure pointed to by the second parameter will be passed to all entities on the pipeline. The driver needs to embed the media_pipeline structure into a higher-level structure and can access the pipeline from the media_entity structure. When you need to stop streaming, you need to call:

media_entity_pipeline_stop(struct media_entity *entity);

Since the start function can be called in a nested manner, corresponding to it, the stop function should also keep a corresponding number of calls. The media_entity_pipeline_start function will check the validity of the link. At this time, the link_validate member of media_entity will be used to complete the check. The following picture is an illustration of traversal:

pipeline traversal

The circle number in the figure above refers to the order of visits. It is worth mentioning that the kernel's implementation of breadth-first graph traversal is very intriguing and of reference value, and it is worth finding the kernel code by yourself. Among them, concepts such as stacks and bitmaps are used. The specific code is in the media-entity.c/media_entity_pipelien_start function.

Since the above traversal is breadth-first and all traversal, that is to say, if your ISP has two input sources, then these two input sources may be turned on at the same time, more often we do not want this to happen What we expect is that the specified input source and the specified single-link pipeline are opened. At this time, you need to manage the pipeline yourself, and you can implement your own pipeline structure for related management. The implementation is very simple, this pipeline structure is similar to the following:

struct spec_pipeline {
    struct list_head entities;    //pipeline 线上的 entity 链表头
    struct media_entity entities[PIPE_LENGTH];     //与上一个类似，随自己想法去实现
    int (*open_fun)(struct media_entity entity, int onoff);
    int (*s_stream)(struct media_entity entity, int onoff);
    ... ...
};

Tips:

1. Finding v4l2_subdev from media_entity can be done by using the media_entity_to_v4l2_subdev function.

2. You can add the enable flag for link in the streamon part of the video module. The media_entity_pipeline_start function will add value to the entity.stream_count member and pass the pipe in the second parameter to the entity.pipe member on the link line. After the enable flag is completed, graph traversal can be performed on each entity to call its set_stream member to start the stream. In fact, when s_param, s_fmt and other ioctl are called, graph traversal is required to call the callback functions of each entity on the entire pipeline for related settings.

3. After all the sub-device nodes are registered, the connection of each entity can be completed through media_entity_create_link, waiting for the entire data stream to be opened later.

4. Why is there a media framework? Because there is only subdev, each sub-device class is in a level state, and there is no difference in the direction of the data flow. If you need to establish a data flow pipeline, you need to implement it yourself, which will be more troublesome. With the media framework, these managements It will be much more convenient, because it provides all the management operations of the pipeline, so that many subdevs can be connected in series as a complete data stream, so as to carry out correct and directed data transfer.

5. Both v4l2_subdev and video_device have a media_entity structure (non-pointer type). The entity in video_devcie will be registered with the registration of the video device, and the name will be the video_device name. The two are different. When used Need special attention.

[Linux Basics] - V4L2 framework-media device