SurfaceTexture of Android graphics system

introduction

SurfaceTexture is the core of off-screen rendering and TextureView. It contains a BufferQueue internally, which can convert the image stream generated by Surface into texture for further processing by the business side. The entire architecture is shown in the figure below:

1. First, generate an image stream through Canvas, OpenGL, Camera or Video Decoder.

2. Then, the image stream is queued into the BufferQueue through the Surface and notified to the GLConsumer.

3. Then, GLConsumer gets the image stream GraphicBuffer from the BufferQueue and converts it to a texture.

4. Finally, the business party can further process the texture, such as adding special effects or adding it to the screen.

Let's look at the initialization of SurfaceTexture and the flow of image data within SurfaceTexture.

SurfaceTexture initialization

new SurfaceTexture(textureId)Start SurfaceTexture initialization, the core logic is as follows:

SurfaceTexture initialization

SurfaceTexture_initIt is the core code for SurfaceTexture initialization, as shown below:

static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached, jint texName, jboolean singleBufferMode, jobject weakThiz)
{
    // 创建BufferQueueCore、BufferQueueProducer、BufferQueueConsumer
    sp<IGraphicBufferProducer> producer;
    sp<IGraphicBufferConsumer> consumer;
    BufferQueue::createBufferQueue(&producer, &consumer);

    if (singleBufferMode) { // 单缓冲
        consumer->setMaxBufferCount(1); // 双缓冲、三缓冲就是指这里
    }

    // Java层的SurfaceTexture,实际对应Native层GLConsumer
    sp<GLConsumer> surfaceTexture;
    if (isDetached) {
        surfaceTexture = new GLConsumer(consumer,GL_TEXTURE_EXTERNAL_OES,true,!singleBufferMode);
    } else {
        surfaceTexture = new GLConsumer(consumer,texName,GL_TEXTURE_EXTERNAL_OES,true,!singleBufferMode);
    }

    // If the current context is protected, inform the producer.
    if (isProtectedContext()) {
        consumer->setConsumerUsageBits(GRALLOC_USAGE_PROTECTED);
    }
    // 为Java层SurfaceTexture.mSurfaceTexture设置GLConsumer对象地址
    SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
    // 为Java层SurfaceTexture.mProducer设置producer对象地址
    SurfaceTexture_setProducer(env, thiz, producer);
    // SurfaceTexture jclass
    jclass clazz = env->GetObjectClass(thiz);

    // weakThiz表示Java层SurfaceTexture对象的弱引用，JNISurfaceTextureContext是JNI封装类，负责回调Java层SurfaceTexture.postEventFromNative方法
    sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz, clazz));
    // 为GLConsumer设置回调（回调到java层）
    surfaceTexture->setFrameAvailableListener(ctx); 
    // 为Java层SurfaceTexture.mFrameAvailableListener设置ctx的对象地址
    SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
}

After SurfaceTexture is initialized, a listener is set to GLConsumer, which will call back to the Java layer Method, further callback to the OnFrameAvailableListener listener registered in SurfaceTexture, used to notify the business layer that a new has joined the team. If the business layer is interested in the latest , call to update to the texture, otherwise do nothing and ignore it Some graphical data. JNISurfaceTextureContextSurfaceTexture.postEventFromNativeGraphicBufferGraphicBufferupdateTexImageGraphicBuffer

GLConsumer is the direct consumer of BufferQueue and is responsible for converting GraphicBuffer into texture. Then notify the indirect consumer to consume the texture through the listening classwp<FrameAvailableListener> mFrameAvailableListener. When the indirect consumer is SurfaceFlinger, the listening class is Layer, and Layer further notifies SurfaceFlinger to synthesize all Layers and then display them on the screen. When the indirect consumer is SurfaceTexture, the listening class is JNISurfaceTextureContext, which is used to notify the Java layer that new image data is available.

SurfaceTexture image data flow

This section mainly looks at the creation of the producerSurface, the process of the business layer receiving the frame availability notification and updating the target texture.

Create producer Surface based on SurfaceTexture

After creating based on texture IDSurfaceTexture, Surface is generally created. At this time, Surface is the producer and SurfaceTexture (Native layer corresponding to GLConsumer) is the consumer. The consumer is responsible for converting the GraphicBuffer obtained from the BufferQueue into a texture, and then the business layer can further process the texture, such as applying special effects or putting it on the screen.

As a producer, Surface writes GraphicBuffer to BufferQueue through BufferQueueProducer; as a consumer, GLConsumer reads GraphicBuffer from BufferQueue through BufferQueueConsumer.

The core logic of creating based onSurfaceTexture is in the Native layer: create a Surface object based on the BufferqueueProducer held by SurfaceTexture, and bind the address of the object Set to the Java layer variable. The core code is as follows:SurfaceSurface.mNativeObject

// 基于SurfaceTexture创建Surface
public Surface(SurfaceTexture surfaceTexture) {
    synchronized (mLock) {
        mName = surfaceTexture.toString();
        // 保存Native层Surface对象地址
        setNativeObjectLocked(nativeCreateFromSurfaceTexture(surfaceTexture));
    }
}
// 根据SurfaceTexture持有的BufferqueueProducer创建Surface，并返回对象地址    
static jlong nativeCreateFromSurfaceTexture(JNIEnv* env, jclass clazz, jobject surfaceTextureObj) {
    // 从Java层surfaceTexture.mProducer中获取BufferqueueProducer的对象地址，并创建BufferqueueProducer。
    sp<IGraphicBufferProducer> producer(SurfaceTexture_getProducer(env, surfaceTextureObj));
   // 基于BufferqueueProducer，创建Native Surface对象
    sp<Surface> surface(new Surface(producer, true));
   // 返回Surface对象地址
    surface->incStrong(&sRefBaseOwner);
    return jlong(surface.get());
}

Visible, create the Native layerSurface object, the BufferqueueProducer parameter is required, it is responsible for dequeuing from BufferQueue and enqueueGraphicBuffer.

After creatingSurface, you can draw image data to Surface in a variety of ways, such as: Canvas drawing, Camera output, video decoder rendering and OpenGL rendering.

Next, let's look at how the graphics data is updated to the target texture in two steps?

Business layer receives notification of frame availability

Here, we take Canvas drawing as an example to see the callback process available when the business layer receives frames, as shown below:

SurfaceTexture frame callback

After the Java layer Surface calls the unlockCanvasAndPost method, the Native layer Surface queues the GraphicBuffer into the BufferQueue through the BufferqueueProducer, and at the same time notifies the listener through BufferQueueCore'ssp<IConsumerListener> mConsumerListener (the implementation class is BufferQueue::ProxyConsumerListener) The consumer then notifies the OnFrameAvailableListener listener set by the Java layer for SurfaceTexture step by step.

The business layer actively updates the target texture

After the Java layer OnFrameAvailableListener listener receives the callback, it needs to call updateTexImage from the OpenGL thread to update GraphicBuffer to the texture. The texture here is the texture ID passed in when we created the SurfaceTexture. The entire update process is as follows:

SurfaceTexture-updateTexImage

OnFrameAvailableListener.onFrameAvailable callback can occur in any thread, so updateTexImage cannot be called directly in the callback, but must be switched to the OpenGL thread call. Because the updateTexImage call chain involves OpenGL operations, it must be in the GL thread.

The core code isGLConsumer::updateTexImage:

• First, obtain the available from the BufferQueue through BufferQueueConsumer, which contains . BufferItemGraphicBuffer

• Then, create EglImage and EGLImageKHR based onGraphicBuffer.

• Finally, update the texture content based onEGLImageKHR.

To put it simply, throughupdateTexImage, we update the latest graphics data to the texture. As for how to use the texture, it is a matter of the business layer.

Update OES texture based on GraphicBuffer

As mentioned above, the updateTexImage method will eventually update GraphicBuffer to the target texture, which is actually done through EglImage and < /span> on the Android platform: EGLImageKHR is implemented. Here we look at how to use EGLImageKHR

1. The texture target needs to be replaced fromGL_TEXTURE_2D to GL_TEXTURE_EXTERNAL_OES, for example: glBindTexture, glTexParameteri and other functions.

2. In the fragment shader, declare the OES extension:#extension GL_OES_EGL_image_external : require. Also, usesamplerExternalOESinstead ofsampler2Dtexture types.

3. BasicGraphicBufferFigure number, communicationeglCreateImageKHRBuildingEGLImageKHR.

GraphicBufferIt can be obtained from BufferQueue, or you can obtain the memory address after locking and write graphics data. For details, please refer to GraphicBuffer.cpp.

1. Fill into texture data throughglEGLImageTargetTexture2DOESEGLImageKHR

glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, static_cast(EGLImageKHR));

1. Last, useeglDestroyImageKHR销毁EGLImageKHR.

The function prototype of creation and destructionEGLImageKHR is as follows:

// 创建EGLImageKHR，在Android平台上，ctx可以是EGL_NO_CONTEXT，target是EGL_NATIVE_BUFFER_ANDROID，buffer是由GraphicBuffer创建来的。
EGLImageKHR eglCreateImageKHR(EGLDisplay dpy, EGLContext ctx, EGLenum target, EGLClientBuffer buffer, const EGLint *attrib_list)

// 销毁EGLImageKHR
EGLBoolean eglDestroyImageKHR(EGLDisplay dpy, EGLImageKHR image)

For specific usage, please refer to GLConsumer::EglImage::createImage.

GLConsumer internally encapsulates the EglImage class, which is responsible for the processing logic of GraphicBuffer, EGLImageKHR and OES textures. The core code is as follows Show:

// EglImage根据GraphicBuffer创建EGLImageKHR，然后使用EGLImageKHR更新纹理，是GLConsumer中负责把从BufferQueue获取的GraphicBuffer，转换为纹理的工具类。
class EglImage : public LightRefBase<EglImage>{
public:
    // 唯一的构造函数，接收一个GraphicBuffer参数
    EglImage(sp<GraphicBuffer> graphicBuffer);

    // 如果参数发生变更，则调用createImage创建内部的EGLImageKHR
    status_t createIfNeeded(EGLDisplay display, const Rect& cropRect, bool forceCreate = false);

    // 把EGLImageKHR绑定的GraphicBuffer图形数据上传到目标纹理
    void bindToTextureTarget(uint32_t texTarget){
        glEGLImageTargetTexture2DOES(texTarget, static_cast<GLeglImageOES>(mEglImage));
    }

private:
    // 创建内部的EGLImageKHR
    EGLImageKHR createImage(EGLDisplay dpy, const sp<GraphicBuffer>& graphicBuffer, const Rect& crop);

    // 用于创建EGLImageKHR的GraphicBuffer
    sp<GraphicBuffer> mGraphicBuffer;
    // 根据GraphicBuffer创建的EGLImageKHR
    EGLImageKHR mEglImage;
    // 创建EGLImageKHR需要的参数
    EGLDisplay mEglDisplay;
    // 创建EGLImageKHR时，使用的裁剪区域
    Rect mCropRect;
}

Summarize

SurfaceTexture is the core of off-screen rendering. For example, we can set SurfaceTexture to Camera to receive camera image data and convert it into OES texture. Then we can use OpenGL to perform further special effects processing on the OES texture, and finally display it on the screen or record it. into video. Therefore, understandingSurfaceTexturethe underlying principles will help with business layer development and troubleshooting. I hope this article will be helpful to those who are interested.