在学习Android Camera API 2使用OpenGL ES 2.0和GLSurfaceView对预览进行实时二次处理（黑白滤镜），发现是通过SurfaceTexture接收相机帧数据并最终通过一个GL_TEXTURE_EXTERNAL_OES纹理将帧数据返回给相机应用的，文本主要分析下
SurfaceTexture是如何创建的SurfaceTexture如何获取相机帧数据SurfaceTexture如何将帧数据绑定到GL_TEXTURE_EXTERNAL_OES纹理上
1.SurfaceTexture是如何创建的
首先，我们需要分析下SurfaceTexture是如何创建的，都做了哪些工作
- 应用层SurfaceTexture创建代码如下：
//创建纹理id
int[] tex = new int[1];
GLES20.glGenTextures(1, tex, 0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, tex[0]);
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST);
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE);
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_WRAP_T, GL10.GL_CLAMP_TO_EDGE);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
//创建SurfaceTexture并传入tex[0]
mSurfaceTexture = new SurfaceTexture(tex[0]);
- Framework 层SurfaceTexture创建代码如下：
//frameworks\base\graphics\java\android\graphics
//构造函数
public SurfaceTexture(int texName) {
this(texName, false);
}
//构造函数
//singleBufferMode是否是单buffer，默认为false
public SurfaceTexture(int texName, boolean singleBufferMode) {
mCreatorLooper = Looper.myLooper();
mIsSingleBuffered = singleBufferMode;
//native方法nativeInit
nativeInit(false, texName, singleBufferMode, new WeakReference<SurfaceTexture>(this));
}
发现SurfaceTexture构造函数只是简单的调用了下nativeInit方，接着分析下nativeInit方法，SurfaceTexture Native 方法注册表如下：
//frameworks\base\core\jni\SurfaceTexture.cpp
static const JNINativeMethod gSurfaceTextureMethods[] = {
{
"nativeInit", "(ZIZLjava/lang/ref/WeakReference;)V", (void*)SurfaceTexture_init },
...
{
"nativeUpdateTexImage", "()V", (void*)SurfaceTexture_updateTexImage },
....
};
发现nativeInit对应的Native方法为SurfaceTexture_init，
接着分析下SurfaceTexture_init
//frameworks\base\core\jni\SurfaceTexture.cpp
//isDetached为false
//texName为应用创建的GL_TEXTURE_EXTERNAL_OES型texture
//singleBufferMode为false
//weakThiz为SurfaceTexture对象弱引用
static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached,
jint texName, jboolean singleBufferMode, jobject weakThiz)
{
sp<IGraphicBufferProducer> producer;
sp<IGraphicBufferConsumer> consumer;
//创建IGraphicBufferProducer及IGraphicBufferConsumer
BufferQueue::createBufferQueue(&producer, &consumer);
if (singleBufferMode) {
consumer->setMaxBufferCount(1);
}
sp<GLConsumer> surfaceTexture;
//isDetached为false
if (isDetached) {
....
} else {
//将consumer和texName封装为GLConsumer类对象surfaceTexture
surfaceTexture = new GLConsumer(consumer, texName,
GL_TEXTURE_EXTERNAL_OES, true, !singleBufferMode);
}
....
//设置surfaceTexture名字
surfaceTexture->setName(String8::format("SurfaceTexture-%d-%d-%d",
(isDetached ? 0 : texName),
getpid(),
createProcessUniqueId()));
// If the current context is protected, inform the producer.
consumer->setConsumerIsProtected(isProtectedContext());
//将surfaceTexture保存到env中
SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
//将producer保存到env中
SurfaceTexture_setProducer(env, thiz, producer);
jclass clazz = env->GetObjectClass(thiz);
//JNISurfaceTextureContext继承了GLConsumer::FrameAvailableListener
sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz,
clazz));
//surfaceTexture设置帧回调对象ctx，
//收到帧数据是会触发ctx->onFrameAvailable方法
surfaceTexture->setFrameAvailableListener(ctx);
SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
}
通过上边的分析可见，创建SurfaceTexture时会通过BufferQueue分别创建一对IGraphicBufferProducer和IGraphicBufferConsumer类的生产者、消费者对象：producer、consumer。然后给消费者consumer注册一个继承了GLConsumer::FrameAvailableListener的帧回调对象 JNISurfaceTextureContext。
在分析Android Camera学习总结时，分析过Android Camera 在createCaptureSession时需要向Camera系统设置至少一个surface。而创建SurfaceTexture时创建了一对IGraphicBufferProducer和IGraphicBufferConsumer型的生产者、消费者对象：producer、consumer,生产者producer就可以封装为一个surface对象供Camera系统使用，实现代码如下：
///frameworks/base/core/jni/android_view_Surface.cpp
static jlong nativeCreateFromSurfaceTexture(JNIEnv* env, jclass clazz,
jobject surfaceTextureObj) {
sp<IGraphicBufferProducer> producer(SurfaceTexture_getProducer(env, surfaceTextureObj));
....
sp<Surface> surface(new Surface(producer, true));
......
surface->incStrong(&sRefBaseOwner);
return jlong(surface.get());
}
在Camera 系统收到帧数据后，通过queueBuffer归还GraphicBuffer给surface时，会触发消费者consumer注册的帧回调JNISurfaceTextureContext::onFrameAvailable，该方法又会调用Java层注册的SurfaceTexture.OnFrameAvailableListener的onFrameAvailable方法，代码如下：
void JNISurfaceTextureContext::onFrameAvailable(const BufferItem& /* item */)
{
....
//调用Java层SurfaceTexture.OnFrameAvailableListener的onFrameAvailable方法
env->CallStaticVoidMethod(mClazz, fields.postEvent, mWeakThiz);
.....
}
至此，可以了解到：
创建SurfaceTexture时，在Native层会创建一对IGraphicBufferProducer和IGraphicBufferConsumer的生产者和消费者对象：producer、consumer。生产者producer可以封装为surface供Camera系统使用，消费者consumer在收到相机帧数据时会触发消费者consumer注册的帧回调函数JNISurfaceTextureContext::onFrameAvailable。
2. SurfaceTexture如何获取相机帧数据
SurfaceTexture在收到帧回调onFrameAvailable时，可以通过调用SurfaceTexture.updateTexImage() 来更新SurfaceTexture中的GL_TEXTURE_EXTERNAL_OES型纹理内容。更新纹理内容首先需要获取帧数据，我们分析下其代码
//frameworks\base\graphics\java\android\graphics\SurfaceTexture.java
public void updateTexImage() {
nativeUpdateTexImage();
}
接着分析下Native方法nativeUpdateTexImage，在jni中对用的函数是SurfaceTexture_updateTexImage
//frameworks\base\core\jni\SurfaceTexture.cpp
static void SurfaceTexture_updateTexImage(JNIEnv* env, jobject thiz)
{
//获取env中保存的surfaceTexture
sp<GLConsumer> surfaceTexture(SurfaceTexture_getSurfaceTexture(env, thiz));
//更新纹理,surfaceTexture为GLConsumer类型
status_t err = surfaceTexture->updateTexImage();
....
}
接着分析下GLConsumer::updateTexImage()方法
扫描二维码关注公众号，回复：
11830153 查看本文章
//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::updateTexImage() {
....
//更新EGL状态，如：
//更新mEglDisplay为eglGetCurrentDisplay
//mEglContext为eglGetCurrentContext
// Make sure the EGL state is the same as in previous calls.
status_t err = checkAndUpdateEglStateLocked();
....
BufferItem item;
//获取帧数据buffer
// Acquire the next buffer.
// In asynchronous mode the list is guaranteed to be one buffer
// deep, while in synchronous mode we use the oldest buffer.
err = acquireBufferLocked(&item, 0);
.....
//Update the Current GLConsumer state.
//Release the previous buffer.
err = updateAndReleaseLocked(item);
.....
//将新buffer绑定到纹理
// Bind the new buffer to the GL texture, and wait until it's ready.
return bindTextureImageLocked();
}
分析下GLConsumer::acquireBufferLocked
//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::acquireBufferLocked(BufferItem *item,
nsecs_t presentWhen, uint64_t maxFrameNumber) {
//获取Consumer当前的显示内容BufferItem
//既获取相机预览帧数据
status_t err = ConsumerBase::acquireBufferLocked(item, presentWhen,
maxFrameNumber);
...
// If item->mGraphicBuffer is not null, this buffer has not been acquired
// before, so any prior EglImage created is using a stale buffer. This
// replaces any old EglImage with a new one (using the new buffer).
if (item->mGraphicBuffer != NULL) {
int slot = item->mSlot;
//由获取的item->mGraphicBuffer生成一个EglImage，并赋值给mEglSlots[slot].mEglImage
mEglSlots[slot].mEglImage = new EglImage(item->mGraphicBuffer);
}
return NO_ERROR;
}
至此，可以了解到surfaceTexture通过 GLConsumer::acquireBufferLocked就可以获取了当前的相机帧数据，相机帧数据存放在了item->mGraphicBuffer中。
3. SurfaceTexture如何将帧数据绑定到GL_TEXTURE_EXTERNAL_OES纹理上
我们接着分析下GLConsumer::updateAndReleaseLocked
status_t GLConsumer::updateAndReleaseLocked(const BufferItem& item,
PendingRelease* pendingRelease)
{
status_t err = NO_ERROR;
//acquireBufferLocked获取的BufferItem
int slot = item.mSlot;
.....
//重新检查EGL状态
// Confirm state.
err = checkAndUpdateEglStateLocked();
....
// Ensure we have a valid EglImageKHR for the slot, creating an EglImage
// if nessessary, for the gralloc buffer currently in the slot in
// ConsumerBase.
// We may have to do this even when item.mGraphicBuffer == NULL (which
// means the buffer was previously acquired).
err = mEglSlots[slot].mEglImage->createIfNeeded(mEglDisplay, item.mCrop);
.....
// Do whatever sync ops we need to do before releasing the old slot.
if (slot != mCurrentTexture) {
err = syncForReleaseLocked(mEglDisplay);
....
}
GLC_LOGV("updateAndRelease: (slot=%d buf=%p) -> (slot=%d buf=%p)",
mCurrentTexture, mCurrentTextureImage != NULL ?
mCurrentTextureImage->graphicBufferHandle() : 0,
slot, mSlots[slot].mGraphicBuffer->handle);
// Hang onto the pointer so that it isn't freed in the call to
// releaseBufferLocked() if we're in shared buffer mode and both buffers are
// the same.
//将 mEglSlots[slot].mEglImage赋值到nextTextureImage
sp<EglImage> nextTextureImage = mEglSlots[slot].mEglImage;
// release old buffer
if (mCurrentTexture != BufferQueue::INVALID_BUFFER_SLOT) {
if (pendingRelease == nullptr) {
status_t status = releaseBufferLocked(
mCurrentTexture, mCurrentTextureImage->graphicBuffer(),
mEglDisplay, mEglSlots[mCurrentTexture].mEglFence);
}
....
}
// Update the GLConsumer state.
mCurrentTexture = slot;
mCurrentTextureImage = nextTextureImage;
mCurrentCrop = item.mCrop;
mCurrentTransform = item.mTransform;
mCurrentScalingMode = item.mScalingMode;
mCurrentTimestamp = item.mTimestamp;
mCurrentDataSpace = item.mDataSpace;
mCurrentFence = item.mFence;
mCurrentFenceTime = item.mFenceTime;
mCurrentFrameNumber = item.mFrameNumber;
//跟新转换矩阵
computeCurrentTransformMatrixLocked();
return err;
}
该方法主要是更新下mCurrentTextureImage和mCurrentTexture并释放就得buffer。
接着分析下如何将mCurrentTextureImage绑定到GL_TEXTURE_EXTERNAL_OES纹理上，方法是GLConsumer::bindTextureImageLocked()
status_t GLConsumer::bindTextureImageLocked() {
....
GLenum error;
....
//mTexTarget为应用创建的GL_TEXTURE_EXTERNAL_OES型texture
glBindTexture(mTexTarget, mTexName);
...
//由mGraphicBuffer生成EGLImageKHR
status_t err = mCurrentTextureImage->createIfNeeded(mEglDisplay,
mCurrentCrop);
.....
//将mCurrentTextureImage内容绑定到mTexTarget上
mCurrentTextureImage->bindToTextureTarget(mTexTarget);
.....
// Wait for the new buffer to be ready.
return doGLFenceWaitLocked();
}
我们先看一下GLConsumer::EglImage::createIfNeeded是如何通过mGraphicBuffer生成EGLImageKHR图像得
status_t GLConsumer::EglImage::createIfNeeded(EGLDisplay eglDisplay,
const Rect& cropRect,
bool forceCreation) {
.....
// If there's no image, create one.
if (mEglImage == EGL_NO_IMAGE_KHR) {
mEglDisplay = eglDisplay;
mCropRect = cropRect;
//创建EGLImageKHR对象
mEglImage = createImage(mEglDisplay, mGraphicBuffer, mCropRect);
}
....
return OK;
}
接着分析下GLConsumer::EglImage::createImage
EGLImageKHR GLConsumer::EglImage::createImage(EGLDisplay dpy,
const sp<GraphicBuffer>& graphicBuffer, const Rect& crop) {
EGLClientBuffer cbuf =
static_cast<EGLClientBuffer>(graphicBuffer->getNativeBuffer());
const bool createProtectedImage =
(graphicBuffer->getUsage() & GRALLOC_USAGE_PROTECTED) &&
hasEglProtectedContent();
EGLint attrs[] = {
EGL_IMAGE_PRESERVED_KHR, EGL_TRUE,
EGL_IMAGE_CROP_LEFT_ANDROID, crop.left,
EGL_IMAGE_CROP_TOP_ANDROID, crop.top,
EGL_IMAGE_CROP_RIGHT_ANDROID, crop.right,
EGL_IMAGE_CROP_BOTTOM_ANDROID, crop.bottom,
createProtectedImage ? EGL_PROTECTED_CONTENT_EXT : EGL_NONE,
createProtectedImage ? EGL_TRUE : EGL_NONE,
EGL_NONE,
};
.....
eglInitialize(dpy, 0, 0);
//调用eglCreateImageKHR创建EGLImageKHR 对象
EGLImageKHR image = eglCreateImageKHR(dpy, EGL_NO_CONTEXT,
EGL_NATIVE_BUFFER_ANDROID, cbuf, attrs);
...
return image;
}
通过上边的分析发现，createIfNeeded 就是通过eglCreateImageKHR(dpy, EGL_NO_CONTEXT, EGL_NATIVE_BUFFER_ANDROID, cbuf, attrs)将一个graphicbuffer生成了一个EGLImageKHR图像。
现在已经生成了EGLImageKHR图像，接下来分析下 EGLImageKHR 图像是如何绑定的GL_TEXTURE_EXTERNAL_OES纹理上的
//frameworks\native\libs\gui\GLConsumer.cpp
void GLConsumer::EglImage::bindToTextureTarget(uint32_t texTarget) {
//更新纹理texTarget的数据为mEglImage
//相当于glTexImage2D或者glTexSubImage2D
glEGLImageTargetTexture2DOES(texTarget,
static_cast<GLeglImageOES>(mEglImage));
}
从上述分析看，整个更新纹理的过程大致分为两步
- 生成纹理内容EGLImageKHR 对象
通过ConsumerBase::acquireBufferLocked获取当前的显示内容GraphicBuffer（在这里就是相机帧数据），然后由此GraphicBuffer生成一个EGLImageKHR图像 - 将
EGLImageKHR图像通过glEGLImageTargetTexture2DOES绑定到GL_TEXTURE_EXTERNAL_OES型纹理上。
其中关于将graphic buffer直接作为纹理的文章，请参考