1. Introduction:
ACodec is directly connected to the superior of OMX. The maintenance of its state machine is more complicated than that of MediaCodec. ACodec separates the state machine to ensure that it is in sync with OMX. This article will introduce in detail how the state machine of ACodec is. maintained.
2. Introduction of ACodec's state machine: Let's first look at the state machine of
the OMX standard header file ( ):OMX_CORE.h
typedef enum OMX_STATETYPE
{
OMX_StateInvalid, /**< component has detected that it's internal data
structures are corrupted to the point that
it cannot determine it's state properly */
OMX_StateLoaded, /**< component has been loaded but has not completed
initialization. The OMX_SetParameter macro
and the OMX_GetParameter macro are the only
valid macros allowed to be sent to the
component in this state. */
OMX_StateIdle, /**< component initialization has been completed
successfully and the component is ready to
to start. */
OMX_StateExecuting, /**< component has accepted the start command and
is processing data (if data is available) */
OMX_StatePause, /**< component has received pause command */
OMX_StateWaitForResources, /**< component is waiting for resources, either after
preemption or before it gets the resources requested.
See specification for complete details. */
OMX_StateKhronosExtensions = 0x6F000000, /**< Reserved region for introducing Khronos Standard Extensions */
OMX_StateVendorStartUnused = 0x7F000000, /**< Reserved region for introducing Vendor Extensions */
OMX_StateMax = 0X7FFFFFFF
} OMX_STATETYPE;
The most important of these state machines are OMX_StateLoaded, OMX_StateIdleand OMX_StateExecuting, and ACodec controls the OMX components based on these state machines.
Let's look at the constructor of ACodec:
ACodec::ACodec()
: mSampleRate(0),
mNodeGeneration(0),
mUsingNativeWindow(false),
mNativeWindowUsageBits(0),
mLastNativeWindowDataSpace(HAL_DATASPACE_UNKNOWN),
mIsVideo(false),
mIsImage(false),
mIsEncoder(false),
mFatalError(false),
mShutdownInProgress(false),
mExplicitShutdown(false),
mIsLegacyVP9Decoder(false),
mEncoderDelay(0),
mEncoderPadding(0),
mRotationDegrees(0),
mChannelMaskPresent(false),
mChannelMask(0),
mDequeueCounter(0),
mMetadataBuffersToSubmit(0),
mNumUndequeuedBuffers(0),
mRepeatFrameDelayUs(-1ll),
mMaxPtsGapUs(0ll),
mMaxFps(-1),
mFps(-1.0),
mCaptureFps(-1.0),
mCreateInputBuffersSuspended(false),
mLatency(0),
mTunneled(false),
mDescribeColorAspectsIndex((OMX_INDEXTYPE)0),
mDescribeHDRStaticInfoIndex((OMX_INDEXTYPE)0),
mStateGeneration(0),
mVendorExtensionsStatus(kExtensionsUnchecked) {
// Mediatek Android Patch Begin
sVerboseLog = (bool)property_get_int32("vendor.media.acodec.verbose.log", 0);
// Mediatek Android Patch End
memset(&mLastHDRStaticInfo, 0, sizeof(mLastHDRStaticInfo));
/* 1.实例化各个状态机 */
mUninitializedState = new UninitializedState(this);
mLoadedState = new LoadedState(this);
mLoadedToIdleState = new LoadedToIdleState(this);
mIdleToExecutingState = new IdleToExecutingState(this);
mExecutingState = new ExecutingState(this);
mOutputPortSettingsChangedState =
new OutputPortSettingsChangedState(this);
mExecutingToIdleState = new ExecutingToIdleState(this);
mIdleToLoadedState = new IdleToLoadedState(this);
mFlushingState = new FlushingState(this);
mPortEOS[kPortIndexInput] = mPortEOS[kPortIndexOutput] = false;
mInputEOSResult = OK;
mPortMode[kPortIndexInput] = IOMX::kPortModePresetByteBuffer;
mPortMode[kPortIndexOutput] = IOMX::kPortModePresetByteBuffer;
memset(&mLastNativeWindowCrop, 0, sizeof(mLastNativeWindowCrop));
/* 2.改变状态 */
changeState(mUninitializedState);
}
1. ACodec's state machine:
ACodec maintains a total of 9 state machines, which are as follows:
sp<UninitializedState> mUninitializedState;
sp<LoadedState> mLoadedState;
sp<LoadedToIdleState> mLoadedToIdleState;
sp<IdleToExecutingState> mIdleToExecutingState;
sp<ExecutingState> mExecutingState;
sp<OutputPortSettingsChangedState> mOutputPortSettingsChangedState;
sp<ExecutingToIdleState> mExecutingToIdleState;
sp<IdleToLoadedState> mIdleToLoadedState;
sp<FlushingState> mFlushingState;
sp<SkipCutBuffer> mSkipCutBuffer;
As mUninitializedStatean example, take a look at the inheritance relationship of this class:
struct ACodec::UninitializedState : public ACodec::BaseState {
explicit UninitializedState(ACodec *codec);
protected:
virtual bool onMessageReceived(const sp<AMessage> &msg);
virtual void stateEntered();
private:
void onSetup(const sp<AMessage> &msg);
bool onAllocateComponent(const sp<AMessage> &msg);
sp<DeathNotifier> mDeathNotifier;
DISALLOW_EVIL_CONSTRUCTORS(UninitializedState);
};
It can be seen that each state machine will inherit from BaseState, and there are two important member functions onMessageReceivedand in the class stateEntered, the former is used to process msg independent of each state, and the latter is used to process some necessary when entering this state things, and the member functions in private are unique to each state, and also positively reflect the characteristics of each state machine.
2. BaseState analysis:
struct ACodec::BaseState : public AState {
explicit BaseState(ACodec *codec, const sp<AState> &parentState = NULL);
protected:
enum PortMode {
KEEP_BUFFERS,
RESUBMIT_BUFFERS,
FREE_BUFFERS,
};
ACodec *mCodec;
virtual PortMode getPortMode(OMX_U32 portIndex);
virtual void stateExited();
virtual bool onMessageReceived(const sp<AMessage> &msg);
virtual bool onOMXEvent(OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2);
virtual void onOutputBufferDrained(const sp<AMessage> &msg);
virtual void onInputBufferFilled(const sp<AMessage> &msg);
void postFillThisBuffer(BufferInfo *info);
private:
// Handles an OMX message. Returns true iff message was handled.
bool onOMXMessage(const sp<AMessage> &msg);
// Handles a list of messages. Returns true iff messages were handled.
bool onOMXMessageList(const sp<AMessage> &msg);
// returns true iff this message is for this component and the component is alive
bool checkOMXMessage(const sp<AMessage> &msg);
bool onOMXEmptyBufferDone(IOMX::buffer_id bufferID, int fenceFd);
bool onOMXFillBufferDone(
IOMX::buffer_id bufferID,
size_t rangeOffset, size_t rangeLength,
OMX_U32 flags,
int64_t timeUs,
int fenceFd);
virtual bool onOMXFrameRendered(int64_t mediaTimeUs, nsecs_t systemNano);
void getMoreInputDataIfPossible();
DISALLOW_EVIL_CONSTRUCTORS(BaseState);
};
It can be seen that ACodec::BaseStateit is a specific operation on the OMX component, where onOutputBufferDrainedthe sum onInputBufferFilledis the operation on the buffer. In other words, ACodec's state machine is capable of buffer operations on OMX.
3. changeState analysis:
All state changes need to changeStatebe realized by passing, look at the implementation of this function:
void AHierarchicalStateMachine::changeState(const sp<AState> &state) {
/* 1.确认设置的状态是否是当前状态 */
if (state == mState) {
// Quick exit for the easy case.
return;
}
/* 2.将当前状态推入容器A中 */
Vector<sp<AState> > A;
sp<AState> cur = mState;
for (;;) {
A.push(cur);
if (cur == NULL) {
break;
}
cur = cur->parentState();
}
/* 3.将下一个状态推入容器B中 */
Vector<sp<AState> > B;
cur = state;
for (;;) {
B.push(cur);
if (cur == NULL) {
break;
}
cur = cur->parentState();
}
// Remove the common tail.
while (A.size() > 0 && B.size() > 0 && A.top() == B.top()) {
A.pop();
B.pop();
}
/* 4.更新状态 */
mState = state;
/* 5.调用上一个状态的退出函数 */
for (size_t i = 0; i < A.size(); ++i) {
A.editItemAt(i)->stateExited();
}
/* 6.调用待设置状态的进入函数 */
for (size_t i = B.size(); i > 0;) {
i--;
B.editItemAt(i)->stateEntered();
}
}
3. Analysis of ACodec state machine:
The various state machines of ACodec are closely connected. In order to prevent him from explaining each state machine below in a cloud, here is the state machine diagram of the entire ACodec:
1.UninitializedState:
a.Construction of ACodec:
When constructing ACodec, we have seen that after instantiating 9 states, the final call will be made to changeStateset the current state to UninitializedState:
ACodec::ACodec()...{
...
changeState(mUninitializedState);
}
b. LoadedStateWhen in the state, if it is called at this time onShundown, it will also be placed in this state. Take a look at the implementation of this state stateEntered:
void ACodec::LoadedState::onShutdown(bool keepComponentAllocated) {
/* release()函数会调用此处 */
if (!keepComponentAllocated) {
(void)mCodec->mOMXNode->freeNode();
mCodec->changeState(mCodec->mUninitializedState);
}
...
}
2.LoadedState:
a. When MediaCodec configures the codec, it will call onAllocateComponentthe function. Here it will apply for the omx component. After the application is successful, the state will be changed to LoadedState:
bool ACodec::UninitializedState::onAllocateComponent(const sp<AMessage> &msg) {
...
sp<CodecObserver> observer = new CodecObserver;
sp<IOMX> omx;
sp<IOMXNode> omxNode;
status_t err = NAME_NOT_FOUND;
OMXClient client;
if (client.connect(owner.c_str()) != OK) {
mCodec->signalError(OMX_ErrorUndefined, NO_INIT);
return false;
}
omx = client.interface();
pid_t tid = gettid();
int prevPriority = androidGetThreadPriority(tid);
androidSetThreadPriority(tid, ANDROID_PRIORITY_FOREGROUND);
err = omx->allocateNode(componentName.c_str(), observer, &omxNode);
...
mCodec->mOMX = omx;
mCodec->mOMXNode = omxNode;
mCodec->mCallback->onComponentAllocated(mCodec->mComponentName.c_str());
mCodec->changeState(mCodec->mLoadedState);
return true;
}
b. LoadedToIdleStateAfter entering the state, it will apply for a buffer. If the buffer application fails, it will fall back to this state:
void ACodec::LoadedToIdleState::stateEntered() {
ALOGV("[%s] Now Loaded->Idle", mCodec->mComponentName.c_str());
status_t err;
/* 申请buffer */
if ((err = allocateBuffers()) != OK) {
ALOGE("Failed to allocate buffers after transitioning to IDLE state "
"(error 0x%08x)",
err);
mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
mCodec->mOMXNode->sendCommand(
OMX_CommandStateSet, OMX_StateLoaded);
if (mCodec->allYourBuffersAreBelongToUs(kPortIndexInput)) {
mCodec->freeBuffersOnPort(kPortIndexInput);
}
if (mCodec->allYourBuffersAreBelongToUs(kPortIndexOutput)) {
mCodec->freeBuffersOnPort(kPortIndexOutput);
}
mCodec->changeState(mCodec->mLoadedState);
}
}
The release phase of c.codec, that is, when the message IdleToLoadedStateis received OMX_EventCmdComplete:
bool ACodec::IdleToLoadedState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventCmdComplete:
{
if (data1 != (OMX_U32)OMX_CommandStateSet
|| data2 != (OMX_U32)OMX_StateLoaded) {
ALOGE("Unexpected command completion in IdleToLoadedState: %s(%u) %s(%u)",
asString((OMX_COMMANDTYPE)data1), data1,
asString((OMX_STATETYPE)data2), data2);
mCodec->signalError(OMX_ErrorUndefined, FAILED_TRANSACTION);
return true;
}
mCodec->changeState(mCodec->mLoadedState);
return true;
}
default:
return BaseState::onOMXEvent(event, data1, data2);
}
}
3.LoadedToIdleState:
LoadedState In the state, when onStarthe t function is called, the OMX state will be set to OMX_StateIdle, if the state is set successfully, ACodec will switch to this state:
void ACodec::LoadedState::onStart() {
ALOGV("onStart");
status_t err = mCodec->mOMXNode->sendCommand(OMX_CommandStateSet, OMX_StateIdle);
if (err != OK) {
mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
} else {
mCodec->changeState(mCodec->mLoadedToIdleState);
}
}
4. IdleToExecutingState:
LoadedToIdleState In the state, when the message is received OMX_EventCmdComplete, it indicates that the state of the bottom layer of OMX is switched successfully at this time, and OMX_StateExecutingthe state setting will be sent to the bottom layer at the same time, and the state of ACodec at this time will be changed to IdleToExecutingState:
bool ACodec::LoadedToIdleState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventCmdComplete:
{
status_t err = OK;
if (data1 != (OMX_U32)OMX_CommandStateSet
|| data2 != (OMX_U32)OMX_StateIdle) {
ALOGE("Unexpected command completion in LoadedToIdleState: %s(%u) %s(%u)",
asString((OMX_COMMANDTYPE)data1), data1,
asString((OMX_STATETYPE)data2), data2);
err = FAILED_TRANSACTION;
}
if (err == OK) {
err = mCodec->mOMXNode->sendCommand(
OMX_CommandStateSet, OMX_StateExecuting);
}
if (err != OK) {
mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
} else {
mCodec->changeState(mCodec->mIdleToExecutingState);
}
return true;
}
...
}
5. ExecutingState:
In IdleToExecutingStatethe state, when the message is received OMX_EventCmdComplete, it indicates that the underlying OMX is already in the OMX_StateExecutingstate at this time, then ACodec changes the state synchronously to ExecutingState:
bool ACodec::IdleToExecutingState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventCmdComplete:
{
if (data1 != (OMX_U32)OMX_CommandStateSet
|| data2 != (OMX_U32)OMX_StateExecuting) {
ALOGE("Unexpected command completion in IdleToExecutingState: %s(%u) %s(%u)",
asString((OMX_COMMANDTYPE)data1), data1,
asString((OMX_STATETYPE)data2), data2);
mCodec->signalError(OMX_ErrorUndefined, FAILED_TRANSACTION);
return true;
}
mCodec->mExecutingState->resume();
mCodec->changeState(mCodec->mExecutingState);
return true;
}
default:
return BaseState::onOMXEvent(event, data1, data2);
}
}
When ACodec is changed to this state machine, it indicates that the buffer operation with OMX is ongoing.
6.OutputPortSettingsChangedState:
This is a special state machine that can only ExecutingStatebe switched with each other. It is usually used for input or output port parameter changes and needs to be reconfigured.
In ExecutingStatethe event reception of , if OMX_EventPortSettingsChangedan event is received, the state will be switched to OutputPortSettingsChangedState:
bool ACodec::ExecutingState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventPortSettingsChanged:
{
...
/* 给OMX组件发送OMX_CommandPortDisable指令 */
if (data2 == 0 || data2 == OMX_IndexParamPortDefinition) {
mCodec->mMetadataBuffersToSubmit = 0;
CHECK_EQ(mCodec->mOMXNode->sendCommand(
OMX_CommandPortDisable, kPortIndexOutput),
(status_t)OK);
mCodec->freeOutputBuffersNotOwnedByComponent();
mCodec->changeState(mCodec->mOutputPortSettingsChangedState);
}
...
}
...
}
In the above code, you can see that when switching to this state, OMX_CommandPortDisablean instruction will be sent to the OMX component, and after the OMX component executes the instruction, it will call back and notify ACodec:
bool ACodec::OutputPortSettingsChangedState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventCmdComplete:
{
if (data1 == (OMX_U32)OMX_CommandPortDisable) {
if (data2 != (OMX_U32)kPortIndexOutput) {
ALOGW("ignoring EventCmdComplete CommandPortDisable for port %u", data2);
return false;
}
ALOGV("[%s] Output port now disabled.", mCodec->mComponentName.c_str());
status_t err = OK;
if (!mCodec->mBuffers[kPortIndexOutput].isEmpty()) {
ALOGE("disabled port should be empty, but has %zu buffers",
mCodec->mBuffers[kPortIndexOutput].size());
err = FAILED_TRANSACTION;
} else {
mCodec->mAllocator[kPortIndexOutput].clear();
}
/* OMX组件执行指令完成后将继续发送OMX_CommandPortEnable指令 */
if (err == OK) {
err = mCodec->mOMXNode->sendCommand(
OMX_CommandPortEnable, kPortIndexOutput);
}
if (err == OK) {
err = mCodec->allocateBuffersOnPort(kPortIndexOutput);
ALOGE_IF(err != OK, "Failed to allocate output port buffers after port "
"reconfiguration: (%d)", err);
mCodec->mCallback->onOutputBuffersChanged();
}
if (err != OK) {
mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
ALOGE("Error occurred while disabling the output port");
}
return true;
} else if (data1 == (OMX_U32)OMX_CommandPortEnable) {
if (data2 != (OMX_U32)kPortIndexOutput) {
ALOGW("ignoring EventCmdComplete OMX_CommandPortEnable for port %u", data2);
return false;
}
ALOGV("[%s] Output port now reenabled.", mCodec->mComponentName.c_str());
if (mCodec->mExecutingState->active()) {
mCodec->mExecutingState->submitOutputBuffers();
}
/* 再次失能端口成功将切换至ExecutingState */
mCodec->changeState(mCodec->mExecutingState);
return true;
}
return false;
}
default:
return BaseState::onOMXEvent(event, data1, data2);
}
}
It can be seen that this state switch is completed by ACodec itself. After 发送指令-等待回调all operations are completed, it will eventually switch back to OMX ExecutingState.
7. ExecutingToIdleState:
When the application decides that the decoder is no longer needed for encoding and decoding, initiateShutdownthe function of ACodec will be called at this time:
void ACodec::initiateShutdown(bool keepComponentAllocated) {
/* 发送kWhatShutdown消息 */
sp<AMessage> msg = new AMessage(kWhatShutdown, this);
msg->setInt32("keepComponentAllocated", keepComponentAllocated);
msg->post();
if (!keepComponentAllocated) {
// ensure shutdown completes in 3 seconds
(new AMessage(kWhatReleaseCodecInstance, this))->post(3000000);
}
}
ExecutingStatewill onMessageReceivedreceive this message:
bool ACodec::ExecutingState::onMessageReceived(const sp<AMessage> &msg) {
bool handled = false;
switch (msg->what()) {
case kWhatShutdown:
{
int32_t keepComponentAllocated;
CHECK(msg->findInt32(
"keepComponentAllocated", &keepComponentAllocated));
mCodec->mShutdownInProgress = true;
mCodec->mExplicitShutdown = true;
mCodec->mKeepComponentAllocated = keepComponentAllocated;
mActive = false;
/* 向OMX发送OMX_StateIdle状态的设置CMD */
status_t err = mCodec->mOMXNode->sendCommand(
OMX_CommandStateSet, OMX_StateIdle);
if (err != OK) {
if (keepComponentAllocated) {
mCodec->signalError(OMX_ErrorUndefined, FAILED_TRANSACTION);
}
// TODO: do some recovery here.
} else {
/* 设置成功将切换状态 */
mCodec->changeState(mCodec->mExecutingToIdleState);
}
handled = true;
break;
}
...
}
It can be seen that the state setting ExecutingStatewill be sent to the bottom layer OMX_StateIdle, and the state of ACodec will be switched to ExecutingToIdleState.
8. IdleToLoadedState:
When OMX_StateIdlethe state is set successfully, it will be notified by ExecutingToIdleStatecallback onOMXEvent:
bool ACodec::ExecutingToIdleState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
switch (event) {
case OMX_EventCmdComplete:
{
if (data1 != (OMX_U32)OMX_CommandStateSet
|| data2 != (OMX_U32)OMX_StateIdle) {
ALOGE("Unexpected command completion in ExecutingToIdleState: %s(%u) %s(%u)",
asString((OMX_COMMANDTYPE)data1), data1,
asString((OMX_STATETYPE)data2), data2);
mCodec->signalError(OMX_ErrorUndefined, FAILED_TRANSACTION);
return true;
}
mComponentNowIdle = true;
/* 注意这个函数 */
changeStateIfWeOwnAllBuffers();
return true;
}
...
}
...
}
Follow up and take a look changeStateIfWeOwnAllBuffers:
void ACodec::ExecutingToIdleState::changeStateIfWeOwnAllBuffers() {
if (mComponentNowIdle && mCodec->allYourBuffersAreBelongToUs()) {
/* 向OMX发送OMX_StateLoaded指令 */
status_t err = mCodec->mOMXNode->sendCommand(
OMX_CommandStateSet, OMX_StateLoaded);
if (err == OK) {
err = mCodec->freeBuffersOnPort(kPortIndexInput);
status_t err2 = mCodec->freeBuffersOnPort(kPortIndexOutput);
if (err == OK) {
err = err2;
}
}
...
}
...
/* 状态切换 */
mCodec->changeState(mCodec->mIdleToLoadedState);
}
When the underlying OMX state is successfully switched, ACodec will switch to the `LoadedState state through a callback.
9. FlushingState:
This state can only ExecutingStatebe switched with each other.
When a message ExecutingStateis received kWhatFlush, the state will be switched:
case kWhatFlush:
{
ALOGV("[%s] ExecutingState flushing now "
"(codec owns %zu/%zu input, %zu/%zu output).",
mCodec->mComponentName.c_str(),
mCodec->countBuffersOwnedByComponent(kPortIndexInput),
mCodec->mBuffers[kPortIndexInput].size(),
mCodec->countBuffersOwnedByComponent(kPortIndexOutput),
mCodec->mBuffers[kPortIndexOutput].size());
mActive = false;
/* 向OMX底层发送OMX_CommandFlush指令 */
status_t err = mCodec->mOMXNode->sendCommand(OMX_CommandFlush, OMX_ALL);
if (err != OK) {
mCodec->signalError(OMX_ErrorUndefined, FAILED_TRANSACTION);
} else {
/* 同时切换状态 */
mCodec->changeState(mCodec->mFlushingState);
}
handled = true;
break;
}
When the bottom layer of OMX is completed OMX_CommandFlush, it will call back to ACodec:
bool ACodec::FlushingState::onOMXEvent(
OMX_EVENTTYPE event, OMX_U32 data1, OMX_U32 data2) {
...
case OMX_EventCmdComplete:
{
...
changeStateIfWeOwnAllBuffers();
...
}
...
}
Follow up to changeStateIfWeOwnAllBuffers(), eventually returning to ExecutingState:
void ACodec::FlushingState::changeStateIfWeOwnAllBuffers() {
...
mCodec->changeState(mCodec->mExecutingState);
...
}
4. Summary:
To master the state machine of ACodec, you need to start from the following aspects:
1. Pay attention to the close connection of each state machine of ACodec, you can refer to the overview diagram above; 2. Pay attention to the implementation and implementation
of each state ; 3. Pay attention How does ACodec set the status of OMX components through the standard interface of OMX, and how does OMX notify ACodec through callbacks;stateEntered()stateExited()