I have been studying Android GraphicBuffer for some time, so what are the advantages of Android GraphicBuffer?
I searched on the Internet and reprinted a good article introducing this issue.
[Reprinted]
The introduction on the EGL extension in ANDROID is as follows:
Because in OpenGL ES, uploading textures (glTexImage2D(), glSubTexImage2D()) is an extremely time-consuming process , and uploading a full screen with a screen size of 1080×1920 The texture needs 20～60ms. In this case, SurfaceFlinger cannot run at 60fps. Therefore, Android uses the image native buffer, and operates the graphic buffer directly as a direct texture .
[转载]using direct textures on android
using direct textures on android
16 dec 2011
I’ve been working at Mozilla on Firefox Mobile for a few months now. One of the goals of the new native UI is to have liquid smooth scrolling and panning at all times. Unsurprisingly, we do this by drawing into an OpenGL texture and moving it around on the screen. This is pretty fast until you run out of content in the texture and need to update it. Gecko runs in a separate thread and can draw to a buffer there without blocking us, but uploading that data into the texture is where problems arise. Right now we use just one very large texture (usually 2048x2048), and glTexSubImage2D can take anywhere from 25ms to 60ms. Given that our target is 60fps, we have about 16ms to draw a frame. This means we’re guaranteed to miss at least one frame every time we upload, but likely more than that. What we need is a way of uploading texture data asynchronously (and preferably quicker). This is where direct textures can help.
If you haven’t read Dianne Hackborn’s recent posts on the Android graphics stack, you’re missing out (part 1, part 2). The window compositing system she describes (called SurfaceFlinger) is particularly interesting because it is close to the problem we have in Firefox. One of the pieces Android uses to to draw windows is the gralloc module. As you may have guessed, gralloc is short for ‘graphics alloc’. You can see the short and simple API for it here. Android has a wrapper class that encapsulates access to this called GraphicBuffer. It has an even nicer API, found here. Usage is very straightforward. Simply create the GraphicBuffer with whatever size and pixel format you need, lock it, write your bits, and unlock. One of the major wins here is that you can use the GraphicBuffer instance from any thread. So not only does this reduce a copy of your image, but it also means you can upload it without blocking the rendering loop!
To get it on the screen using OpenGL, you can create an EGLImageKHR from the GraphicBuffer and bind it to a texture:
#define EGL_NATIVE_BUFFER_ANDROID 0x3140
#define EGL_IMAGE_PRESERVED_KHR 0x30D2
GraphicBuffer* buffer = new GraphicBuffer(1024, 1024, PIXEL_FORMAT_RGB_565,
GraphicBuffer::USAGE_SW_WRITE_OFTEN |
GraphicBuffer::USAGE_HW_TEXTURE);
unsigned char* bits = NULL;
buffer->lock(GraphicBuffer::USAGE_SW_WRITE_OFTEN, (void**)&bits);
// Write bitmap data into 'bits' here
buffer->unlock();
// Create the EGLImageKHR from the native buffer
EGLint eglImgAttrs[] = {
EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE };
EGLImageKHR img = eglCreateImageKHR(eglGetDisplay(EGL_DEFAULT_DISPLAY), EGL_NO_CONTEXT,
EGL_NATIVE_BUFFER_ANDROID,
(EGLClientBuffer)buffer->getNativeBuffer(),
eglImgAttrs);
// Create GL texture, bind to GL_TEXTURE_2D, etc.
// Attach the EGLImage to whatever texture is bound to GL_TEXTURE_2D
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, img);
The resulting texture can be used as a regular one, with one caveat. Whenever you manipulate pixel data, the changes will be reflected on the screen immediately after unlock. You probably want to double buffer in order to avoid problems here.
If you’ve ever used the Android NDK, it won’t be surprising that GraphicBuffer (or anything similar) doesn’t exist there. In order to use any of this in your app you’ll need to resort to dlopen hacks. It’s a pretty depressing situation. Google uses this all over the OS, but doesn’t seem to think that apps need a high performance API. But wait, it gets worse. Even after jumping through these hoops, some gralloc drivers don’t allow regular apps to play ball. So far, testing indicates that this is the case on Adreno and Mali GPUs. Thankfully, PowerVR and Tegra allow it, which covers a fair number of devices.
With any luck, I’ll land the patches that use this in Firefox Mobile today. The result should be a much smoother panning and zooming experience on devices where gralloc is allowed to work.
[Reprint] EGL extension in ANDROID
Original: http://tangzm.com/blog/?p=167
Google has made some extensions to egl in Android to make the entire display rendering software system run more efficiently. In our analysis and modification of the SurfaceFlinger code, we can often see the code related to these egl extensions, such as android native fence, KHR image, etc. Although skipping these content has little effect on understanding SurfaceFlinger itself, when reading the code, every time I see these "little stones", I feel uncomfortable. Therefore, it took a little time to find some KHRONOS documents, combined with the SurfaceFlinger source code, to have a basic understanding of these EGL extensions.
The producer/consumer model of BufferQueue
Before entering the discussion of these extensions, let's briefly review the operating mechanism of Andriod BufferQueue.
In Android (after 3.0), up to application, down to surfaceflinger, all drawing processes are done using OpenGL ES. For each painter (producer, content producer), the steps are roughly the same.
(1) Obtain a Surface (usually through SurfaceControl)
(2) Take this Surface as a parameter to get egl draw surface and read surface. Usually these two are the same object
(3) Configure egl config, and then call eglMakeCurrent() to configure the current drawing context.
(4) Start to draw content, generally through glDrawArray() or glDrawElemens() to draw
(5)调用eglSwapBuffers() swap back buffer和front buffer。
(6) Go back to step 4 and draw the next frame.
We know that the final result of all drawing is nothing more than a piece of pixel memory, which stores RGB or YUV values; this memory is the backend of the Surface. In Android, in order to allow content producers and consumers to work in parallel, three times the memory buffer (MTK is four times the buffer) is used behind each Surface, so that when the painter is drawing, it will not affect the current The screen is displayed, and after the painter finishes drawing a frame, he can usually get a new Buffer to draw the next frame without waiting.
In the figure, the buffer in BufferQueue is marked into 4 states:
(1) FREE means that the Buffer is not used and its content is empty
(2) DEQUEUED indicates that the Buffer is being drawn by the content producer
(3) QUEUED means that the Buffer has been drawn and put into the BufferQueue, waiting to be displayed (or the next step)
(4) ACQUIRED means that the Buffer is being used by consumers for display (or next step processing)
The state of migration is always FREE=>DEQUEUED=>QUEUED=>ACQUIRED=>FREE.
When the producer does eglMakeCurrent(), it will find an item in the FREE from the BufferQueue, mark it as DEQUEUED, and use it as the current drawing target. After drawing, when the producer calls eglSwapBuffers(), the current DEQUEUED Buffer is set to QUEUED, marking it can be displayed (or the next step), and at the same time looking for another Buffer Item whose status is FREE from the BufferQueue, and setting it to DEQUEUED, continue drawing.
At the other end, consumers of BufferQueue (usually SurfaceFlinger, or some ImageProcessor) look for items marked as QUEUED from BufferQueue for display or next processing. The relationship between producers and consumers can refer to the following figure:
EGL_ANDROID_image_native_buffer
After talking about the mechanism of BufferQueue, you can enter the topic. The first extension introduced by Andriod is EGL_ANDROID_image_native_buffer. In OpenGL ES, uploading textures (glTexImage2D(), glSubTexImage2D()) is an extremely time-consuming process, and it takes 20-60ms to upload a full-screen texture under a screen size of 1080×1920. In this case, SurfaceFlinger cannot run at 60fps. Therefore, Android uses the image native buffer, and operates the graphic buffer directly as a texture ( direct texture ).
The use of image native buffer on Android is quite convenient:
First create an EGLImageKHR with Graphic Buffer
Then bind the egl image to texture2D OES
If you need to reference the texture in OpenGL ES, you must change the type in glsl from sampler2D to samplerExternalOES, for example
EGL_ANDROID_native_fence_sync
Before discussing Android native fence sync, let's first consider how to synchronize and coordinate the CPU and GPU. We know that the API call of OpenGL ES is actually a series of commands. When the user calls these APIs, these commands will be cached by the OpenGL lib library (or the underlying driver?). Manually calling glFlush() will force all commands in the current context to be sent to the GPU for execution, but before these commands are executed, glFlush() will return. glFinish() sends all commands to the GPU and waits for these commands to be executed before returning.
It seems that glFinish() is a feasible way to synchronize CPU and GPU work. But the flaw of glFinish() is also obvious. During GPU work, the current thread has been waiting. If no other threads have work to do (scheduled by the CPU), the CPU will be wasted during this time. In addition, if other threads are waiting for the completion of the GPU task, they must also be notified manually through Condition after glFinish(). Therefore, KHRONOS added EGL_KHR_fence_sync (fence synchronization object) on this basis. Through eglCreateSyncKHR(), a fence object of type EGLSyncKHR will be created in the current position of the current GL context (after all previously called commands). After that, the current thread or any other thread can wait on this fence synchronization object through eglClientWaitSyncKHR(), just like waiting for a Condition. Similar to Condition, the wait function can also accept a timeout parameter to specify the timeout condition. The figure below demonstrates how egl fence synchronizes GPU and multiple threads.
Android native fence goes one step further in KHR fence. You can get a fd (file descriptor) from the fence object, or generate a sync object from fd. With this feature, Android extends the scope of synchronization from multiple threads to multiple processes! This is too important for Android, because we know that most of the producers and consumers of BufferQueue are not in the same process. With android native fence, you can synchronize the use of Buffer in a multi-process environment.
Someone may ask, why is it so troublesome? Isn't BufferQueue Item stateful? Isn't it enough to judge the usage of the graphic buffer by the state? In fact, in order to make the CPU and GPU work in parallel in Android, the state of the BufferQueue Item did not wait until the operation of the graphic buffer (on the GPU side) was completely finished. That is to say, when the producer calls eglSwapBuffers(), BufferQueue The status of the Item is immediately rewritten (DEQUEUED=>QUEUED, FREE=>DEQUEUED) without using glFinish() to wait for all operations to end before setting. In this way, both producers and consumers can proceed to the next step as soon as possible. In this way, a problem arises. How does the consumer know that the graphic buffer has been "really" written, and how does the producer know that the graphic buffer has been actually used by the consumer and released. The answer is of course android native fence.
On the one hand, the producer inserts an acquire fence after eglSwapBuffers(), and the consumer knows that the graphic buffer is actually written by waiting for this fence, and then consumes (show...). On the other hand, when the consumer processing is completed, a relative fence will be inserted. When the producer is in dequeueBuffer(), it will wait for this fence, but then it will actually enter a state where it can be drawn.
Let us look at the specific locations where these fences are generated and waiting from the code
Acquire Fence
- EGL driver (lib) will be passed into acquire fence when queueBuffrer. ( See
code ) - Android system calls BufferQueue::queueBuffer() on SurfaceFlinger through Binder, and
passes fence as a parameter. BufferQueue records the fence in the slot ( see
code ) - When SurfaceTexture (ConsumerBase) acquireBuffer, acquire
fence is stored in slot ( see
code ) - If this Surface will be displayed by SurfaceFlinger, then there are two situations. Add using GLES for synthesis, then when Layer::onDraw(), it will wait on the fence of SurfaceTexture ( see
code ),
if it is synthesized by HWC, SurfaceFlinger will pass the fence to HWC ( see
code ). Although we cannot see the code implementation of HWC, the correctly implemented HWC will wait for all fences to be signaled internally before being synthesized - If this Surface is used as a texture
for drawing in another GL context. Because SurfaceTexture cannot know when the texture is actually used, Android will wait for the fence to be triggered in SurfaceTexture::updateTexImage() before completing the update.( see
code )
Release Fence
In turn, the Release Fence is used to notify the producer when the consumer has finished using the graphic buffer.
- If Surface is used as a Layer composite display by SurfaceFlinger, then in SurfaceFlinger::postComposition(), SurfaceFlinger will set a release
fence for each Surface . The generation of this release fence is done by HWC or FrameBuffer GLES.
Both HWC and FB GLES should be generated in the appropriate location on the release fence - If Surface is used as a texture, there
are two fate of the graphic buffer. One is that the old graphic
buffer is replaced by a new one in SurfaceTexture::updateTexImage()
; the other is that the graphic
buffer is detached from the current GL context in SurfaceTexture::detachFromContext() . In either case, SurfaceTexture will call syncForReleaseLocked to insert the release
fene - Finally, in BufferQueue::dequeueBuffer(), BufferQueue uses eglClientWaitSyncKHR() to wait for the release
fence to be signaled ; after this, the producer can really dequeue a graphic buffer and start drawing.
EGL_ANDROID_framebuffer_target
In addition to HWC, SurfaceFlinger will also use the GLES method (FrameBuffer method) to synthesize Surface. In order to distinguish between ordinary EGL context (rendering results are used as HWC input or GLES texture) and FrameBuffer dedicated EGL context (rendering results are output to FrameBuffer), Android uses the EGL_FRAMEBUFFER_TARGET_ANDROID CONFIG attribute to mark the EGL context that requires a FrameBuffer. Specific information can refer to KHRONOS standard
EGL_ANDROID_recordable
Refer to KHRONOS standard
EGL_ANDROID_blob_cache
Refer to KHRONOS standard