一、引言：
OMX组件的标准接口（OMX_Core.h）中，OMX_FillThisBuffer和OMX_EmptyThisBuffer共同完成了OMX的buffer运转。OMX_FillThisBuffer是操作解码完后数据（PCM/YUV）的，OMX_EmptyThisBuffer是操作解码前（es）数据的。本博客将分析OMX_FillThisBuffer，下篇博客再分析OMX_EmptyThisBuffer。
整个buffer机制的分析需要从两个方面进行，一方面是OMX回调通知ACodec和MediaCodec，另一方面是MediaCodec使用完buffer后通知OMX这个过程，两个过程组合起来实际上就是生产者与消费者的关系。对于OMX_FillThisBuffer而言，MediaCodec是消费者，OMX组件是生产者。
之前讲OMX消息机制的时候，说过OMXNodeInstance会实例化一个OMX_CALLBACKTYPE的结构体，指向了三个函数：
OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = {
&OnEvent, &OnEmptyBufferDone, &OnFillBufferDone
};
OMX_CALLBACKTYPE就是OMX_Core.h中的标准接口：
typedef struct OMX_CALLBACKTYPE
{
OMX_ERRORTYPE (*EventHandler)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_EVENTTYPE eEvent,
OMX_IN OMX_U32 nData1,
OMX_IN OMX_U32 nData2,
OMX_IN OMX_PTR pEventData);
OMX_ERRORTYPE (*EmptyBufferDone)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
OMX_ERRORTYPE (*FillBufferDone)(
OMX_OUT OMX_HANDLETYPE hComponent,
OMX_OUT OMX_PTR pAppData,
OMX_OUT OMX_BUFFERHEADERTYPE* pBuffer);
} OMX_CALLBACKTYPE;
EmptyBufferDone和FillBufferDone就是OMX通知给ACodec回调接口。
二、ACodecBufferChannel介绍：
ACodecBufferChannel这个类是供ACodec用于更新buffer信息的，类的实例化操作是在MediaCodec的init函数中完成：
status_t MediaCodec::init(const AString &name) {
...
/* 1.实例化ACodecBufferChannel */
mBufferChannel = mCodec->getBufferChannel();
/* 2.设置ACodecBufferChannel的回调函数 */
mBufferChannel->setCallback(
std::unique_ptr<CodecBase::BufferCallback>(
new BufferCallback(new AMessage(kWhatCodecNotify, this))));
...
}
看一下ACodec中getBufferChannel函数的操作：
std::shared_ptr<BufferChannelBase> ACodec::getBufferChannel() {
if (!mBufferChannel) {
mBufferChannel = std::make_shared<ACodecBufferChannel>(
new AMessage(kWhatInputBufferFilled, this),
new AMessage(kWhatOutputBufferDrained, this));
}
return mBufferChannel;
}
构造函数中实例化了两个msg，分别和OMX组件的标准回调消息EmptyBufferDone及FillBufferDone配合使用。
回到前面看回调的设置，当ACodecBufferChannel实例化完成后，MediaCodec为了能和ACodec同步获知OMX对buffer的更新情况，会设置一个BufferCallback（主要是一个msg）到ACodecBufferChannel中：
BufferCallback::BufferCallback(const sp<AMessage> ¬ify)
: mNotify(notify) {
}
notify就是上面的kWhatCodecNotify，有了这个msg之后，ACodecBufferChannel中的消息就可以由MediaCodec来处理了。
三、OMX通过FillBufferDone通知至MediaCodec：
不论是vdec组件还是adec组件，当某个buffer的数据填充完之后，OMX组件会调用callbacks.FillBufferDone通知至上层，下面看下芯片平台的具体实现：
pMSComponent->callbacks.FillBufferDone(pOMXComponent, pMSComponent->callbackData, bufferHeader);
参数中bufferHeader包括了解码完后buffer的所有信息。下面看下OMXNodeInstance中OnFillBufferDone的实现：
OMX_ERRORTYPE OMXNodeInstance::OnFillBufferDone(
OMX_IN OMX_HANDLETYPE /* hComponent */,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer) {
if (pAppData == NULL) {
ALOGE("b/25884056");
return OMX_ErrorBadParameter;
}
OMXNodeInstance *instance = static_cast<OMXNodeInstance *>(pAppData);
if (instance->mDying) {
return OMX_ErrorNone;
}
int fenceFd = instance->retrieveFenceFromMeta_l(pBuffer, kPortIndexOutput);
omx_message msg;
msg.type = omx_message::FILL_BUFFER_DONE;
msg.fenceFd = fenceFd;
msg.u.extended_buffer_data.buffer = instance->findBufferID(pBuffer);
msg.u.extended_buffer_data.range_offset = pBuffer->nOffset;
msg.u.extended_buffer_data.range_length = pBuffer->nFilledLen;
msg.u.extended_buffer_data.flags = pBuffer->nFlags;
msg.u.extended_buffer_data.timestamp = pBuffer->nTimeStamp;
instance->mDispatcher->post(msg);
return OMX_ErrorNone;
}
OnFillBufferDone的函数实现比较简单，就是将buffer的相关信息填充到omx_message结构体中并发送出去。下面看下omx_message::FILL_BUFFER_DONE的消息处理：
void OMXNodeInstance::onMessages(std::list<omx_message> &messages) {
for (std::list<omx_message>::iterator it = messages.begin(); it != messages.end(); ) {
/* OMXNodeInstance的消息处理 */
if (handleMessage(*it)) {
messages.erase(it++);
} else {
++it;
}
}
/* ACodec的消息处理 */
if (!messages.empty()) {
mObserver->onMessages(messages);
}
}
先看handleMessage：
if (msg.type == omx_message::FILL_BUFFER_DONE) {
/* 1.获取buffer信息 */
OMX_BUFFERHEADERTYPE *buffer =
findBufferHeader(msg.u.extended_buffer_data.buffer, kPortIndexOutput);
if (buffer == NULL) {
ALOGE("b/25884056");
return false;
}
{
Mutex::Autolock _l(mDebugLock);
mOutputBuffersWithCodec.remove(buffer);
CLOG_BUMPED_BUFFER(
FBD, WITH_STATS(FULL_BUFFER(
msg.u.extended_buffer_data.buffer, buffer, msg.fenceFd)));
unbumpDebugLevel_l(kPortIndexOutput);
}
BufferMeta *buffer_meta =
static_cast<BufferMeta *>(buffer->pAppPrivate);
if (buffer->nOffset + buffer->nFilledLen < buffer->nOffset
|| buffer->nOffset + buffer->nFilledLen > buffer->nAllocLen) {
CLOG_ERROR(onFillBufferDone, OMX_ErrorBadParameter,
FULL_BUFFER(NULL, buffer, msg.fenceFd));
}
/* 2.将OMX的buffer拷贝到framework层共享buffer中 */
buffer_meta->CopyFromOMX(buffer);
// fix up the buffer info (especially timestamp) if needed
codecBufferFilled(msg);
}
之前分析消息机制的时候，已经说过，这里的mObserver就是ACodec传递下来的，中间的消息传递过程就不再赘述：
case omx_message::FILL_BUFFER_DONE:
{
IOMX::buffer_id bufferID;
CHECK(msg->findInt32("buffer", (int32_t*)&bufferID));
int32_t rangeOffset, rangeLength, flags, fenceFd;
int64_t timeUs;
CHECK(msg->findInt32("range_offset", &rangeOffset));
CHECK(msg->findInt32("range_length", &rangeLength));
CHECK(msg->findInt32("flags", &flags));
CHECK(msg->findInt64("timestamp", &timeUs));
CHECK(msg->findInt32("fence_fd", &fenceFd));
return onOMXFillBufferDone(
bufferID,
(size_t)rangeOffset, (size_t)rangeLength,
(OMX_U32)flags,
timeUs,
fenceFd);
}
继续看BaseState::onOMXFillBufferDone：
bool ACodec::BaseState::onOMXFillBufferDone(
IOMX::buffer_id bufferID,
size_t rangeOffset, size_t rangeLength,
OMX_U32 flags,
int64_t timeUs,
int fenceFd) {
...
PortMode mode = getPortMode(kPortIndexOutput);
switch (mode) {
...
case RESUBMIT_BUFFERS:
{
...
/* 更新buffer池 */
mCodec->mBufferChannel->drainThisBuffer(info->mBufferID, flags);
...
}
...
}
...
}
这里就需要用到ACodecBufferChannel了，看一下drainThisBuffer的实现：
void ACodecBufferChannel::drainThisBuffer(
IOMX::buffer_id bufferId,
OMX_U32 omxFlags) {
ALOGV("drainThisBuffer #%d", bufferId);
std::shared_ptr<const std::vector<const BufferInfo>> array(
std::atomic_load(&mOutputBuffers));
BufferInfoIterator it = findBufferId(array, bufferId);
if (it == array->end()) {
ALOGE("drainThisBuffer: unrecognized buffer #%d", bufferId);
return;
}
if (it->mClientBuffer != it->mCodecBuffer) {
it->mClientBuffer->setFormat(it->mCodecBuffer->format());
}
uint32_t flags = 0;
if (omxFlags & OMX_BUFFERFLAG_SYNCFRAME) {
flags |= MediaCodec::BUFFER_FLAG_SYNCFRAME;
}
if (omxFlags & OMX_BUFFERFLAG_CODECCONFIG) {
flags |= MediaCodec::BUFFER_FLAG_CODECCONFIG;
}
if (omxFlags & OMX_BUFFERFLAG_EOS) {
flags |= MediaCodec::BUFFER_FLAG_EOS;
}
it->mClientBuffer->meta()->setInt32("flags", flags);
/* 调用BufferCallback的BufferCallback函数 */
mCallback->onOutputBufferAvailable(
std::distance(array->begin(), it),
it->mClientBuffer);
}
可以看到该函数的重点是回调通知到onOutputBufferAvailable：
void BufferCallback::onOutputBufferAvailable(
size_t index, const sp<MediaCodecBuffer> &buffer) {
sp<AMessage> notify(mNotify->dup());
notify->setInt32("what", kWhatDrainThisBuffer);
notify->setSize("index", index);
notify->setObject("buffer", buffer);
notify->post();
}
消息处理中，最重要的就是下面这句：
case kWhatDrainThisBuffer:
{
/* size_t index = */updateBuffers(kPortIndexOutput, msg);
...
}
updateBuffers实现：
size_t MediaCodec::updateBuffers(
int32_t portIndex, const sp<AMessage> &msg) {
CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
size_t index;
CHECK(msg->findSize("index", &index));
sp<RefBase> obj;
CHECK(msg->findObject("buffer", &obj));
sp<MediaCodecBuffer> buffer = static_cast<MediaCodecBuffer *>(obj.get());
{
Mutex::Autolock al(mBufferLock);
if (mPortBuffers[portIndex].size() <= index) {
mPortBuffers[portIndex].resize(align(index + 1, kNumBuffersAlign));
}
mPortBuffers[portIndex][index].mData = buffer;
}
/* 将可用buffer的index入队 */
mAvailPortBuffers[portIndex].push_back(index);
return index;
}
mAvailPortBuffers这个队列存储了目前可用的buffer数量，实际上OMX组件与MediaCodec的最终交互就体现在这个队列中。到这里，整个OMX->MediaCodec的buffer通知机制就分析完了，总结来说，就是当底层OMX组件有解码完后的数据时，会将buffer信息层层回调，通知至MediaCodec并更新到可用buffer队列中，进而供MediaCodec查询使用。
四、dequeueOutputBuffer分析：
在上一个章节中讲到当OMX底层完成了解码并将buffer信息更新到mAvailPortBuffers后，此时，上层应用就可以使用这个buffer中的数据进行播放或者送显，而首先需要做的，就是调用dequeueOutputBuffer获得可用buffer的index：
status_t MediaCodec::dequeueOutputBuffer(
size_t *index,
size_t *offset,
size_t *size,
int64_t *presentationTimeUs,
uint32_t *flags,
int64_t timeoutUs) {
...
/* 发送消息查询index */
sp<AMessage> msg = new AMessage(kWhatDequeueOutputBuffer, this);
msg->setInt64("timeoutUs", timeoutUs);
sp<AMessage> response;
status_t err;
if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
return err;
}
CHECK(response->findSize("index", index));
CHECK(response->findSize("offset", offset));
CHECK(response->findSize("size", size));
CHECK(response->findInt64("timeUs", presentationTimeUs));
CHECK(response->findInt32("flags", (int32_t *)flags));
...
return OK;
}
看一下kWhatDequeueOutputBuffer的做法：
case kWhatDequeueOutputBuffer:
{
...
/* 获取输出index */
if (handleDequeueOutputBuffer(replyID, true /* new request */)) {
break;
}
...
}
可以看到函数是通过handleDequeueOutputBuffer拿到index的：
bool MediaCodec::handleDequeueOutputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
...
else {
sp<AMessage> response = new AMessage;
ssize_t index = dequeuePortBuffer(kPortIndexOutput);
if (index < 0) {
CHECK_EQ(index, -EAGAIN);
return false;
}
const sp<MediaCodecBuffer> &buffer =
mPortBuffers[kPortIndexOutput][index].mData;
response->setSize("index", index);
response->setSize("offset", buffer->offset());
response->setSize("size", buffer->size());
int64_t timeUs;
CHECK(buffer->meta()->findInt64("timeUs", &timeUs));
statsBufferReceived(timeUs);
response->setInt64("timeUs", timeUs);
int32_t flags;
CHECK(buffer->meta()->findInt32("flags", &flags));
response->setInt32("flags", flags);
response->postReply(replyID);
}
return true;
}
再次跟进到dequeuePortBuffer：
ssize_t MediaCodec::dequeuePortBuffer(int32_t portIndex) {
CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
/* 1.获取buffer队列的临时指针对象 */
List<size_t> *availBuffers = &mAvailPortBuffers[portIndex];
if (availBuffers->empty()) {
return -EAGAIN;
}
/* 2.获得队首index */
size_t index = *availBuffers->begin();
/* 3.将此index从队列中删除 */
availBuffers->erase(availBuffers->begin());
/* 4.通过index拿到对应的buffer信息并填充 */
BufferInfo *info = &mPortBuffers[portIndex][index];
CHECK(!info->mOwnedByClient);
{
Mutex::Autolock al(mBufferLock);
info->mOwnedByClient = true;
// set image-data
if (info->mData->format() != NULL) {
sp<ABuffer> imageData;
if (info->mData->format()->findBuffer("image-data", &imageData)) {
info->mData->meta()->setBuffer("image-data", imageData);
}
int32_t left, top, right, bottom;
if (info->mData->format()->findRect("crop", &left, &top, &right, &bottom)) {
info->mData->meta()->setRect("crop-rect", left, top, right, bottom);
}
}
}
return index;
}
dequeuePortBuffer函数的目的很简单，就是从mAvailPortBuffers中获取可用buffer的index，如果没有的话，会返回-EAGAIN。
五、releaseOutputBuffer分析：
当输出buffer使用完之后，需要OMX进行继续填充，这个过程就是由releaseOutputBuffer来完成的：
status_t MediaCodec::releaseOutputBuffer(size_t index) {
sp<AMessage> msg = new AMessage(kWhatReleaseOutputBuffer, this);
msg->setSize("index", index);
sp<AMessage> response;
return PostAndAwaitResponse(msg, &response);
}
跟进kWhatReleaseOutputBuffer：
case kWhatReleaseOutputBuffer:
{
sp<AReplyToken> replyID;
CHECK(msg->senderAwaitsResponse(&replyID));
/* 检查当前状态是否正常 */
if (!isExecuting()) {
PostReplyWithError(replyID, INVALID_OPERATION);
break;
} else if (mFlags & kFlagStickyError) {
PostReplyWithError(replyID, getStickyError());
break;
}
/* 继续往下调用 */
status_t err = onReleaseOutputBuffer(msg);
PostReplyWithError(replyID, err);
break;
}
继续onReleaseOutputBuffer：
status_t MediaCodec::onReleaseOutputBuffer(const sp<AMessage> &msg) {
...
/* 是否需要渲染后再释放 */
if (render && buffer->size() != 0) {
...
mBufferChannel->renderOutputBuffer(buffer, renderTimeNs);
} else {
/* 直接释放 */
mBufferChannel->discardBuffer(buffer);
}
return OK;
}
renderOutputBuffer和discardBuffer的区别为是否需要渲染后再去释放，通常，视频数据是借助MediaCodec绑定的Surface去渲染的，所以会调用renderOutputBuffer，而音频数据此时已经copy到audiotrack中了，故直接释放作废即可。
接下来看一下mBufferChannel（ACodecBufferChannel.cpp）中的具体实现：
status_t ACodecBufferChannel::discardBuffer(const sp<MediaCodecBuffer> &buffer) {
std::shared_ptr<const std::vector<const BufferInfo>> array(
std::atomic_load(&mInputBuffers));
bool input = true;
BufferInfoIterator it = findClientBuffer(array, buffer);
if (it == array->end()) {
array = std::atomic_load(&mOutputBuffers);
input = false;
it = findClientBuffer(array, buffer);
if (it == array->end()) {
return -ENOENT;
}
}
ALOGV("discardBuffer #%d", it->mBufferId);
/* 获取kWhatOutputBufferDrained消息 */
sp<AMessage> msg = input ? mInputBufferFilled->dup() : mOutputBufferDrained->dup();
msg->setObject("buffer", it->mCodecBuffer);
msg->setInt32("buffer-id", it->mBufferId);
/* 打上废弃标志 */
msg->setInt32("discarded", true);
msg->post();
return OK;
}
mInputBufferFilled->dup()对应的是kWhatInputBufferFilled消息实体，mOutputBufferDrained->dup()对应的是kWhatOutputBufferDrained消息实体。如果调用renderOutputBuffer的话会往msg中设置render为true的键值对，调用discardBuffer的话会往msg中设置discarded为true的键值对。
继续往下，看下kWhatOutputBufferDrained的消息处理：
case kWhatOutputBufferDrained:
{
onOutputBufferDrained(msg);
break;
}
onOutputBufferDrained函数比较长，我们只关注重点部分：
void ACodec::BaseState::onOutputBufferDrained(const sp<AMessage> &msg) {
...
PortMode mode = getPortMode(kPortIndexOutput);
switch (mode) {
...
case RESUBMIT_BUFFERS:
{
if (!mCodec->mPortEOS[kPortIndexOutput]) {
if (info->mStatus == BufferInfo::OWNED_BY_NATIVE_WINDOW) {
// We cannot resubmit the buffer we just rendered, dequeue
// the spare instead.
info = mCodec->dequeueBufferFromNativeWindow();
}
if (info != NULL) {
ALOGV("[%s] calling fillBuffer %u",
mCodec->mComponentName.c_str(), info->mBufferID);
info->checkWriteFence("onOutputBufferDrained::RESUBMIT_BUFFERS");
/* 调入到ACodec中 */
status_t err = mCodec->fillBuffer(info);
if (err != OK) {
mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
}
}
}
break;
}
...
}
...
}
因为此时ACodec的状态机为ExecutingState，所以会进入RESUBMIT_BUFFERS的case中去，最终会调用到fillBuffer通知到OMX组件进行buffer的填充：
status_t ACodec::fillBuffer(BufferInfo *info) {
status_t err;
// Even in dynamic ANW buffer mode, if the graphic buffer is not changing,
// send sPreset instead of the same graphic buffer, so that OMX server
// side doesn't update the meta. In theory it should make no difference,
// however when the same buffer is parcelled again, a new handle could be
// created on server side, and some decoder doesn't recognize the handle
// even if it's the same buffer.
if (!storingMetadataInDecodedBuffers() || !info->mNewGraphicBuffer) {
err = mOMXNode->fillBuffer(
info->mBufferID, OMXBuffer::sPreset, info->mFenceFd);
} else {
err = mOMXNode->fillBuffer(
info->mBufferID, info->mGraphicBuffer, info->mFenceFd);
}
info->mNewGraphicBuffer = false;
info->mFenceFd = -1;
if (err == OK) {
info->mStatus = BufferInfo::OWNED_BY_COMPONENT;
}
return err;
}
不管是视频还是音频数据填充，都会走入到OMXNodeInstance中：
status_t OMXNodeInstance::fillBuffer(
IOMX::buffer_id buffer, const OMXBuffer &omxBuffer, int fenceFd) {
Mutex::Autolock autoLock(mLock);
OMX_BUFFERHEADERTYPE *header = findBufferHeader(buffer, kPortIndexOutput);
if (header == NULL) {
ALOGE("b/25884056");
return BAD_VALUE;
}
...
OMX_ERRORTYPE err = OMX_FillThisBuffer(mHandle, header);
}
最终，在这里就会调入到OMX的标准接口OMX_FillThisBuffer中去了。再看一下芯片平台对这个接口的实现：
OMX_ERRORTYPE MS_OMX_FillThisBuffer(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_BUFFERHEADERTYPE *pBuffer)
{
...
/* 1.上面传下来的pBuffer确认是否匹配底层已申请的buffer */
for (i = 0; i < pMSPort->portDefinition.nBufferCountActual; i++) {
if (pBuffer == pMSPort->bufferHeader[i]) {
findBuffer = OMX_TRUE;
break;
}
}
...
/* 将buffer消息写入到msg中并入列 */
message = MS_OSAL_Malloc(sizeof(MS_OMX_MESSAGE));
if (message == NULL) {
ret = OMX_ErrorInsufficientResources;
goto EXIT;
}
message->messageType = MS_OMX_CommandFillBuffer;
message->messageParam = (OMX_U32) i;
message->pCmdData = (OMX_PTR)pBuffer;
if (pMSComponent->u8DebugLevel) MS_OSAL_Log(MS_LOG_DEBUG, "%s FTB: %lld", pMSComponent->componentName, pBuffer->nTimeStamp);
MS_OSAL_Queue(&pMSPort->bufferQ, (void *)message);
MS_OSAL_SemaphorePost(pMSPort->bufferSemID);
}
芯片平台底层是通过消息机制来进行buffer传递的，IL层中只需要将可用buffer的信息写入msg中，并推入到IL层自己实现的消息队列即可，至于buffer出现线程那边，会不停地读取可用msg，然后从msg中读取出可用的buffer进行新一轮的数据填充即可。
六、总结：
一张图剖析OMX_FillThisBuffer和FillBufferDone机制：
