Openmax frame diagram:
openmax il API overview
openmax il API is a component-based multimedia API, including core API and component API
core:
Dynamic loading and unloading of components, allowing the user layer to communicate directly with the component, allowing the user to establish tunnel communication between the two
components
component：
In the il layer, the component represents an independent functional module, which can have four roles: sources, sinks, codecs, filters, splitters, mixers or other data processing modules, depending on its implementation. But in our multimedia processing, a component is likely to be
a hardware, software codec, processor or a combination of the above. An overview of system components. The parameter description includes Buffer status, error, a series of callback functions, etc.
The communication interface between components, called Port, represents the link between the component and the data stream, and the buffers that need to be maintained to maintain the link.
There are four main components:
Sink: only one input port
Souece: only one output port
host: running entirely on the host, with one input and one output
accelerator: running on loosely coupled accelerators, with one input
Three communication methods between components:
Non-tunneled: Mechanism for data exchange between IL client and conponent>Tunneling: Data exchange between
components Proprietary: Optional communication method between components
Refer to the screenshot below for the communication method:
Component state (OMX_STATETYPE)
The state of the component changes, as shown below:
have to be aware of is:
1. The initial state of each component is ubloaded
. 2. LOADED->IDLE state migration requires all required resources to be applied for. If resources are insufficient, LOADED->WAIT FOR RESOURCES state migration will occur. There is a vendor-customized resource manager in the system to monitor resource changes, issue notifications, and control state transitions. Being in IDLE means that the component has acquired all the resources it needs. The IL client will issue state transition controls other than INVALID.
3, EXECUTING means that the component is waiting to receive the buffer to process the data, IDLE will not process the data
4, PAUSED: maintains the context of the buffer execution, and does not process the data or exchange the buffer. From PAUSED->EXECUTING causes cache processing to resume where the component left off.
5, PAUSED->IDLE EXECUTING->IDLE will cause the context of data processing to be true
6, IDLE->LOADED will lead to the loss of operation resources, such as the loss of communication buffers
Communication behavior:
Data communication with components is non-blocking, and communication is enabled once the number of ports is configured. Each port is configured with a specific data format, and components are placed in the correct state. The communication methods of component ports are different, for example: IL Client calls the input port through OMX_EmptyThisBuffer; calls the output port through OMX_FillThisBuffer, the corresponding OMX_EmptyBufferDone OMX_FillBufferDone callback function is triggered before the call returns, as shown in the following figure:
Regarding ports, you need to understand:
1. Each port has a minimum number of buffers that need to be allocated and used based on component definitions.
2. A port associates a buffer header with a buffer. A buffer header points to the data in the middle of the buffer and provides metadata related to the buffer content.
Application and sharing of tunneled buffer:
In a tunnel, one port provides the buffer to another non-supplier port and passes it to it. Providers also apply for buffers. In some normal cases, the tunneling component may choose to reuse the buffer from another port to avoid memory copying and optimize memory usage. This approach is buffer sharing.
In the case of buffer sharing, a component transfers the buffer from the output port to the input port through OMX_EmptyThisBuffer.
component profiles
base profile: only supports Proprietary communication, does not support tunnled
interop profile: The difference from base is that it supports tunnled communication.
Some professional terms:
Supplier port: Between two ports in a pair of tunneling, the supplier port that calls UseBuffer on the neighboring port is the supplier
port. This buffer supplier does not need to apply for a new buffer, but reuses the buffers of other ports.
tunneling port:
sharing port:
tunneling component:
For the required buffer requirements, refer to the structure OMX_PORTDEFINITIONTYPE, port through OMX_GetParameter.
In the middle of the study, the problems to be solved are as follows:
Question 1: How to integrate a new codec?
Question 2: How to determine which codec to use?
Question 3: The openmax decoding process?
Integrate into the platform vendor's custom framework: such as QCOM
integrated into Android (platform irrelevant):