1. Why is there such a subsystem as media?
In the framework of multimedia, there are always complex and diverse. In order to solve the complexity and data fluidity of multimedia devices, a media subsystem is created. Media uses a tree structure to connect each device of the multimedia data path to facilitate the management and control of each device.
Two, media frame
When booting up, in [media-devnode.c], the media device will be assigned a major device number through the media_devnode_init() function. The minor device number ranges from 0 to 255, and a bus named "media" will be registered at the same time. At this time, it is just an application to assign a device number called media. The specific media device registration will be registered in the corresponding device driver.
Let's take camera as an example to introduce the media framework process. Before understanding the media framework, first understand the registration process of the camera device. When registering the /dev/videoX node, the main operation process is as follows:
1. Initialize each v4l2_subdevice through the platform probe;
2. First register v4l2_device through the v4l2_device_register() function;
3. Register the /dev/mediaX device by calling the media_device_init() and media_device_register() functions successively. At this time, set v4l2_dev->mdev = media_dev;
4. Register each v4l2_subdevice. At the same time, when registering each subdevice, the subdevice will be registered as an entity to media_dev;
5. Register the /dev/videoX node through the video_register_device() function. So far, each v4l2_subdevice has been registered. The next step is to make the corresponding connection and set the data path according to the needs of each subdevice;
3. Media equipment and entity
In fact, the /dev/mediaX device is a character device, but it is encapsulated by a layer of the media framework, but the major device number and minor device number of the registered device are obtained from the previously pre-allocated media device.
How to understand media deivce and entity can be simply understood. Media device is the director of a department, and entity is each subordinate. Media device will be responsible for coordinating the entity and managing the flow of each data path . So, let’s take a look at how entities are registered.
We take the camera as an example to understand the media framework. In the /dev/videoX node device, the node is managed through the struct video_device structure, and in this structure, the structure struct v4l2_device of v4l2_device will be included, and in the v4l2 device In the structure of, it will contain the structure struct media_device of media device, and that’s it, layer by layer.
When registering a v4l2 subdevice through the v4l2_device_register_subdev() function, the media_device_register_entity() function will be called to register the v4l2 subdevice as an entity to the media device. At the same time, there will be operations such as the initialization of the entity pads.
One of the goals of the media framework is to discover the internal topology of the device and configure it in real time. To achieve this goal, hardware devices are modeled as graphical building blocks, called entities connected by pads. An entity is a basic media hardware building block. It can communicate with a large number of logical modules-such as physical hardware devices (CMOS sensors), logical hardware devices (a SOC graphics processing pipeline first building block), DMA channels or physical connectors. The pad is the connection endpoint used to communicate between the entity and other entities. The data generated by an entity (not only video) flows from the output of the entity to the input of one or more entities. The pads should not be confused with the physical pins of the chip. A link is a point-to-point connection between two pads. Each entity has been registered with media. Then, let's take a look at how the link between entities operates.
四、entity link
The media entity link is completed by the media_create_pad_link(struct media_entity *source, u16 source_pad, struct media_entity *sink, u16 sink_pad, u32 flags) function. For the declaration of this function, you need to input source entity and source pad, sink entity and sink pad. The number of each entity pad is known in advance, so when you connect, you naturally know which pad you need to use. When linking, it is actually to create a media link for source and sink respectively, add them to media device, and assign link to source and sink. At this point, I have simply understood media device, entity, pad, and link. Then, why do we use this set of frameworks, continue to look down.
5. What is the use of the media frame?
5.1. Negotiation
We know that in the process of using the camera, there are various modules that need to cooperate with each other. The simple data flow is as follows:
sensor ---> CPHY ---> csi ---> isp ---> stream
Then, when we set the image output format of the camera, we will need to confirm that the entire pipeline supports this format. At this time, we can use the functions media_entity_graph_walk_init(), media_entity_graph_walk_start(), media_entity_graph_walk_next(), etc. provided by the media framework. The function starts to traverse the various entities of the pipeline, confirms that each entity supports the format, and completes a data negotiation process.
At the same time, when the platform supports scaler scaling, it can output different resolution sizes from the same source after the scaler, such an entity, multiple outputs, which is also very convenient for data management through the media framework.
5.2, control
Most SOC manufacturers do not open source their ISP algorithms, but the Linux kernel is open source. At this time, SOC manufacturers will put their ISP algorithms on the application layer. At this time, they need to operate to the kernel through the application layer. The media framework manages the entire path of data, so here, it is very convenient to operate ISP and other hardware through the media framework.
When registering a media device, in the _media_device_register() function, assign media devnode fops to media_device_fops. And media_device_fops mainly operates media devices through ioctl. The following operations are supported:
static const struct meida_ioctl_info ioctl_info[] = {
MEDIA_IOC(DEVICE_INFO, media_device_get_info, MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(ENUM_ENTITIES, media_device_enum_entities, MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_GRAPH_MUTEX),
MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEIDA_IOC_FL_GRAPH_MUTEX),
};Through these ioctls, you can obtain the handle of the corresponding entity to control the operation of the corresponding module.
At the same time, each entity is opened in the application layer, and event monitoring is set through VIDIOC_SUBSCRIBE_EVENT. When the corresponding event is generated in the kernel, the event is passed to the application layer through the v4l2_event_queue() function.