Article directory
This article is based on the linux kernel
4.19.4
, and the relevant source code files of the abstract media device model framework are as follows:
（1）include/media/media-device.h
（2）include/media/media-entity.h
（3）/drivers/media/media-device.c
（4）/drivers/media/media-devnode.c
（5）/drivers/media/media-entity.c
In the linux kernel 6.2.7version, the abstract media device framework description file is placed /drivers/media/mcin the directory (that is, a separate directory is opened for storage), and the file naming has also changed:
-
mc-dev-allocator.c - Media Controller Device Allocator API
-
mc-device.c: The code in this file contains the definition of the core logic and data structure of the media device subsystem, as well as the related function implementation. It provides interfaces for registration, initialization, configuration and control of media devices.
-
mc-devnode.c: The code in the file contains the logic and function implementation related to the device node in the media device subsystem. It is responsible for creating and managing device nodes in the media device subsystem so that applications can communicate with media devices through device nodes.
-
mc-entry.c: The code in the file contains the logic and function implementation related to the media entity in the media device subsystem. It defines the data structure of media entities and the interface functions for managing and manipulating media entities.
-
mc-request.c: This file is a newly added file, which provides the Request API for the Media Controller and V4L2 interfaces. Some stateless codec drivers currently require the functions implemented in this file.
1. Abstract Media Device Model
The media controller device is called the media controller framework in this paper. One of the design goals of the framework is to discover the internal topology of the device and configure it at runtime. To achieve this goal, the hardware device is modeled as a directed graph, represented by pad Connected entities are composed together. That is, media devices are abstracted into objects containing entity, pad, linkand .
The Media Controller of the Linux kernel is a general-purpose multimedia framework that provides a standardized way to handle multiple media types, including audio, video, images, and more. The Media Controller is mainly used to describe the topology of media devices, that is, to describe the connection relationship between multiple media devices .
In Media Controller, each media device is described as a media entity (Media Entity), each media entity has one or more media ports (Media Pad), and each media port can be connected to another media device on the media ports, thus forming a topology of media devices.
Media Controller also provides a set of APIs so that users can easily access the topology of media devices and use the same API to access different types of media devices.
The main components of the Media Controller include:
-
Media Controller device: The Media Controller device is a virtual device that represents the topology of the entire media device, including all media entities, media ports, and connection relationships. During development, the topology of the entire media device can be accessed through the Media Controller device.
-
Media Entity device: Media Entity device is an actual media device, which can be a camera, a microphone, a video capture card, etc. Each Media Entity device is described as a media entity, which includes one or more media ports and some attribute information.
-
Media Pad device: Media Pad device is a media port, which describes the input or output port of a Media Entity device. Each Media Pad device includes some attribute information, such as media type, format, direction, and so on.
-
Media Link device: Media Link device describes the connection relationship between two media ports. Each Media Link device includes some attribute information, such as connected source port, destination port, media type, format, etc.
2. Abstract Media Devices
A media device is struct media_devicerepresented by an instance, defined in include/media/media-device.h. Allocation of this structure is done by the media device driver, which typically embeds the media_device instance within a larger driver-specific structure.
struct media_device {
struct device *dev; //父设备
struct media_devnode *devnode; //媒体设备节点
char model[32]; //设备模型的名称
char driver_name[32]; //设备驱动名称（可选）。如果没有设置，调用MEDIA_IOC_DEVICE_INFO将返回dev->driver->name。
char serial[40]; // 设备序列号 （可选）
char bus_info[32]; //唯一和稳定的设备位置标识符
u32 hw_revision; //硬件设备修订
u64 topology_version; //单调计数器用于存储图形拓扑的版本，
u32 id; //在最后注册的图对象上使用的唯一ID
struct ida entity_internal_idx; //图遍历算法使用的唯一内部实体ID
int entity_internal_idx_max; //分配的内部实体索引
struct list_head entities; //用于存储已注册的entity的链表
struct list_head interfaces; //用于存储已注册的接口的链表
struct list_head pads; //用于存储已注册的pad的链表
struct list_head links; //用于存储已注册的link的链表
struct list_head entity_notify; //用于存储已注册的entity_notify回调的链表
struct mutex graph_mutex; //用于保护对struct media_device数据的访问
struct media_graph pm_count_walk;
void *source_priv; //驱动程序用于启用/禁用源处理程序的私有数据
int (*enable_source)(struct media_entity *entity, struct media_pipeline *pipe);//启用源处理程序函数指针
void (*disable_source)(struct media_entity *entity); //禁用源处理程序函数指针
const struct media_device_ops *ops; //media设备的操作回调集合
struct mutex req_queue_mutex; //互斥锁
atomic_t request_id; //用于生成唯一的请求ID
};
The driver program initializes the media device instance by calling , and registers the media device with the linux kernel through the macro function call media_device_init()after the media device instance is initialized .media_device_register()__media_device_register()
Call media_device_unregister()to unregister the media device instance. The initialized media device must be called media_device_cleanup()to clean up at the end.
Note that it is not allowed to unregister an unregistered media device instance, nor to clear an uninitialized media device instance.
3. Entity
Entities are struct media_entityrepresented by instances, defined in include/media/media-entity.h:
struct media_entity {
struct media_gobj graph_obj; //包含媒体对象通用数据的嵌入结构。
const char *name; //实体的名称。
enum media_entity_type obj_type; //实现media_entity的对象类型。
u32 function; //实体主函数，定义在include/uapi/linux/media.h (命名类似于MEDIA_ENT_F_*)。
unsigned long flags;//实体标志，在include/uapi/linux/media.h定义(命名类似于MEDIA_ENT_FL_*)。
u16 num_pads; //sink和source pad的数量。
u16 num_links; //link的总数，转发和返回，启用和禁用。
u16 num_backlinks; //反向的backlink的数量。
int internal_idx; //唯一的内部实体特定编号，如果实体未注册或重新注册，这些号码将被重用。
struct media_pad *pads; //数组的大小由num_pads定义。
struct list_head links; //用于存储数据link的链表。
const struct media_entity_operations *ops; //实体操作。
int use_count; //对实体使用count。
union {
struct {
u32 major;
u32 minor;
} dev; /包含设备major和minor信息。
} info;
};
Entity can be allocated directly in the driver, but the structure is usually embedded in a larger structure, such as v4l2_subdevor video_deviceinstance.
The driver media_entity_pads_init()initializes the entity pad by calling it. After the pad is initialized, the driver media_device_register_entity()registers the entity with the media device by calling it.
If you want to unregister a registered media_entity, call media_device_unregister_entity()unregister.
4. Interfaces
Interfaces are struct media_interfacerepresented by instances, defined in include/media/media-entity.h. Currently only one type of interface is defined: the device node, which is struct media_intf_devnoderepresented by .
struct media_interface {
struct media_gobj graph_obj; //嵌入式的图对象
struct list_head links; //指向图实体的link列表
u32 type; //在include/uapi/linux/media.h中定义的接口类型
u32 flags; //在include/uapi/linux/media.h中定义的接口标志。（以MEDIA_INTF_FL_*进行命名）
};
The driver media_devnode_create()initializes and creates the device node interface by calling.
Remove a device node by calling media_devnode_remove().
5.Pad
Pad is struct media_padrepresented by an instance, defined in include/media/media-entity.h.
struct media_pad {
struct media_gobj graph_obj; //包含媒体对象通用数据的嵌入式结构。
struct media_entity *entity; //本pad所属的Entity。
u16 index; //Pad在实体Pad数组中的索引，编号从0到n。
enum media_pad_signal_type sig_type; //media pad的信号类型。
unsigned long flags; //在include/uapi/linux/media.h中定义的Pad标志。
struct media_pipeline *pipe; //该pad属于的Pipeline。可以使用media_entity_pipeline()访问该字段。
};
Each entity stores its pads in a pad array managed by the entity driver, which typically embeds the pad array in a driver-specific structure. Pads are identified by their entity and their 0-based index in the Pads array, both of which are stored in struct media_pad, making struct media_padpointers the canonical way to store and pass link references. Pad has flags that describe the function and state of the pad: MEDIA_PAD_FL_SINKindicates that the pad supports sink data, and MEDIA_PAD_FL_SOURCEindicates that the pad supports source data.
Each pad must set one of
MEDIA_PAD_FL_SINKor and only one flag can be set.MEDIA_PAD_FL_SOURCE
6. Link
Link is struct media_linkrepresented by an instance, defined in include/media/media-entity.h:
struct media_link {
struct media_gobj graph_obj; //包含媒体对象通用数据的嵌入式结构。
struct list_head list; //与拥有该链路的实体或接口相关联的链表。
union {
struct media_gobj *gobj0; //用于获取链接的第一个graph_object的指针。
struct media_pad *source; //仅当第一个对象(gobj0)是pad时使用。在这种情况下，它表示源pad。
struct media_interface *intf; //仅当第一个对象(gobj0)是pad时使用。在这种情况下，它表示源pad。
};
union {
struct media_gobj *gobj1; //用于获取链接的第二个graph_object的指针。
struct media_pad *sink; //仅在第二个对象(gobj1)是pad时使用。在这种情况下，它表示sink pad。
struct media_entity *entity; //仅在第二个对象(gobj1)是实体时使用。
};
struct media_link *reverse; //指向pad到pad link的反向链接的指针。
unsigned long flags; //link标志，在uapi/media.h中定义。
bool is_backlink; //表示link是否是反向链路。
};
There are two types of Links:
-
(1) Pad-to-Pad Links
Relate two entities by their pads, and each entity has a list of all links from or to any of its pads. Therefore, a given link is stored twice, once in the source entity and once in the target entity. -
(2) Link from interface to entity
This type of link is used to associate an interface with a Link. media_create_intf_link()Create an interface-to-entity link in the driver by calling it , and media_remove_intf_links()delete the created link with .
The driver media_create_pad_link()creates a pad-to-pad link by calling , and media_entity_remove_links()deletes the created link with .
A link can only be created when both ends have been created.
Links have flags that describe their functionality and status: valid values are described in media_create_pad_link()and media_create_intf_link().
Seven, Media graph traversal
In the media framework, an API is provided to iterate over the entities in the graph. To traverse all entities belonging to a media device, the driver can use media_device_for_each_entitythe macro, which is defined in include/media/media-device.h:
#define media_device_for_each_entity(entity, mdev) \
list_for_each_entry(entity, &(mdev)->entities, graph_obj.list)
For example the following usage code:
struct media_entity *entity;
media_device_for_each_entity(entity, mdev) {
//entity will point to each entity in turn
...
}
In the driver program, it may also be necessary to traverse all entities in the graph, which can only be accessed through links starting from a given entity, so the media framework provides a deeply optimized graph traversal API for this.
Note: The graph traversal API does not support graphs with cycles (whether directed or undirected). To prevent infinite loops, the graph traversal code limits the maximum depth
MEDIA_ENTITY_ENUM_MAX_DEPTHto 16, which is currently defined.
In the driver, media_graph_walk_start()start a graph traversal by calling The driver can media_graph_walk_next()retrieve the next entity by calling , which will return NULL when graph traversal is complete.
The graph traversal operation can be interrupted at any time without calling the cleanup function, and the graph structure can be released normally.
The following helper functions can be used to find a link between two given pads, or to find a pad connected to another pad by a link:
media_entity_find_link() //用于查找两个pad之间的连接。
media_pad_remote_pad_first() //在连接的远端查找第一个pad。
media_entity_remote_source_pad_unique() //查找一个连接到实体的远程 source pad。
media_pad_remote_pad_unique() //查找一个连接到pad的远程pad。
8. Usage Counting and Power Handling
Power management is not implemented in the media controller framework because there are huge differences between drivers regarding power management needs. However, struct media_entitythe structure includes a use_countfield that media drivers can use to track the number of users per entity for power management needs.
media_entity.use_countThe field is owned by the media driver, the entity driver cannot use it, otherwise access to the field must be media_device.graph_mutexprotected by a lock.
Nine, link settings
media_entity_setup_link()The properties of a link can be modified at runtime by calling ( struct media_link). The function prototype is as follows:
int media_entity_setup_link(struct media_link *link, u32 flags)
-
link : Pointer to struct media_link.
-
flags : Flags.
Ten, Pipeline and Media flow
A media stream is a stream of pixels or metadata that originates from one or more source devices (such as sensors), flows through a media entity pad, and flows to a final sink. The media stream can be modified by the device on the route (for example: scaling or pixel format conversion), and can also be split into multiple branches, or multi-branch merged.
A media pipeline is a set of interdependent media streams, this interdependence may be caused by hardware (for example, if the first stream is enabled, the configuration of the second stream cannot be changed), or by the driver of. Most commonly: a media pipeline consists of a single stream with no branches.
When starting a stream, the driver must notify all entities in the pipeline to prevent calls during the stream from media_pipeline_start()modifying the link state. In this function, all pads that are pipelines will be marked as streams.
The instance pointed to by the pipe parameter struct media_pipelinewill be stored in each pad in the pipeline. Drivers should be embedded in a higher-level pipeline structure struct media_pipeline, and then struct media_pad pipethe pipeline can be accessed through fields.
media_pipeline_start()calls can be nested. The pipe pointer must be the same for all nested calls of the function.
media_pipeline_start()May return an error, in which case it will erase all changes it has made.
When the stream is stopped, the driver must media_pipeline_stop()notify the entity with . If called multiple times media_pipeline_start(), it needs to be called the same number of times media_pipeline_stop()to stop the stream. media_entity.pipeFields will be reset to NULL when the last nested call stops.
By default, using -EBUSYa configured link will fail if either end of the link is a flow entity.
Links that can be modified while streaming must be
MEDIA_LNK_FL_DYNAMICmarked with a flag.
If other operations on the streaming entity need to be disallowed (for example: changing entity configuration parameters), the driver can explicitly check the media_entity stream_countfield to determine if the entity is undergoing a streaming operation, which must media_device graph_mutexbe done while holding.
11. Link Verification
In media_pipeline_start()the function, link validation is performed for entities with sink pads in the pipeline. media_entity.link_validate()The callback is used to implement the verification operation. In link_validate()the callback, the driver of the entity should check whether the properties of the source pad of the connected entity and its own sink pad match. What a match actually means depends on the type of entity (and ultimately on the properties of the hardware).
Subsystems should facilitate link verification by providing subsystem-specific helper functions to access commonly needed information and ultimately provide a way to use driver-specific callbacks.
12. Allocator API for media controller devices
When a media device belongs to multiple drivers, the shared media device is assigned as shared struct deviceas a lookup key, and the shared media device should remain registered until the last driver unregisters it. Also, the media device should be freed when all references are released. During this probeperiod, when the media device is allocated, each driver gets a reference to the media device. If the media device has already been allocated, the allocate API increments the refcount and returns the current There are media devices.
Calling media_device_delete()the function, the media device will be unregistered and cleared from the kref put handler to ensure that the media device remains in the registered state until the last driver unregisters the media device.
How to use it in the driver
The driver should use the appropriate media core to manage the lifecycle of the shared media device, handling the following two states:
-
1、 allocate -> register -> delete
-
2. Get the reference of the registered device -> delete
Call media_device_delete()function to ensure proper handling of deletion of shared media devices.
For driver .probeoperations: If the media devnode is not registered, it needs to be called media_device_usb_allocate()to allocate or obtain a reference to the media device.
For disconnectthe operation of the driver: call to media_device_delete()release media_device, and the release is completed by the kref put handler.
【Reference link】
https://docs.kernel.org/driver-api/media/mc-core.html#c.media_device