Android OpenGL ES Learning (1) - Basic Concepts
Android OpenGL ES Learning (2) - Graphics Rendering Pipeline and GLSL
Android OpenGL ES Learning (3) – Drawing Plane Graphics
Android OpenGL ES Learning (4) – Orthogonal projection
Android OpenGL ES Learning (5) – Gradient Color
Android OpenGL ES Learning (6) – Using VBO, VAO and EBO/IBO to optimize the program
Android OpenGL ES Learning (7) - Texture
Android OpenGL ES Learning (8) – Matrix Transformation
Android OpenGL ES learning (10) – GLSurfaceView source code analysis GL thread and custom EGL
Android OpenGL ES Learning (11) – Rendering YUV Video and Video Vibrato Effects
Android OpenGL ES learning (twelve) - MediaCodec + OpenGL parsing H264 video + filter
Android OpenGL ES learning (thirteen) - off-screen rendering FBO (screenshot) RBO, OES to FBOCode engineering address: https://github.com/LillteZheng/OpenGLDemo.git
For more audio and video, refer to: Android Audio and Video Getting Started/Advanced Tutorial
I recently encountered a function that requires screenshots, and found that using GLES30.glReadPixels directly has reached more than 2s, and it will block the freeze.
So it is necessary to learn about FBO. The effect to be achieved this time is as follows:
1. Basic concepts
OpenGL regards the framebuffer (frame buffer) as the rendering window by default. In our previous programs, the default frame buffer is used, which is generated and configured when your program starts.
But OpenGL also allows us to define our own frame buffer FBO, which can store the results of our several different rendering samples without affecting the default frame buffer, and display them on the window after processing.
For example, after the camera receives the stream, it performs a beautification effect and then displays it.
1.1 Advantages
The benefits of doing this are:
- Improve rendering efficiency (background drawing)
- Avoid lagging and splashing screens
- Implement texture sharing (texture id of fbo)
2. FBO concept
2.1 Create FBO
In OpenGL, the way to create FBO is very simple:
val frameBuffers = IntArray(1)
GLES30.glGenFramebuffers(1, frameBuffers, 0)
then use
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, frameBuffers[0])
Activates the framebuffer object, but it is not complete; a complete framebuffer needs to meet the following conditions:
- Attach at least one buffer (color, depth or stencil buffer).
- At least one color attachment (Attachment).
- All attachments must be complete (memory preserved).
- Each buffer should have the same number of samples.
From the above conditions, we know that the frame buffer needs to be attached with at least one color attachment to work properly;
2.2 Accessories
An attachment needs to be attached to the framebuffer. An attachment is a memory location that acts as a buffer for the framebuffer, think of it like an image. When creating an attachment we have two options: Texture or Renderbuffer Object.
All subsequent rendering operations will render to the attachment to the currently bound framebuffer. Since our framebuffer is not the default framebuffer, render commands will not have any effect on the window's visual output. For this reason, rendering to a different framebuffer is called off-screen rendering .
2.3 Texture attachments
When attaching a texture to a framebuffer, all rendering commands will write to the texture as if it were a normal color/depth or stencil buffer. The advantage of using textures is that the results of all rendering operations will be stored in a texture image, which we can easily use later in the shader.
In this way, when we finish rendering with fbo, we can get the id of this texture to do special effects or display.
Three use FBO for texture display
We copy the code of the texture in Chapter 7 Android OpenGL ES Learning (7) - Texture Based on it, we realize the function of using FBO and make a cognition of FBO. The effect is as follows:
Creating a texture for a framebuffer is similar to creating a normal texture:
val textures = IntArray(1)
GLES30.glGenTextures(1, textures, 0)
val textureId = textures[0]
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, textureId)
//纹理过滤
GLES30.glTexParameteri(
GLES30.GL_TEXTURE_2D,
GLES30.GL_TEXTURE_MIN_FILTER,
GLES30.GL_NEAREST
)
GLES30.glTexParameteri(GLES30.GL_TEXTURE_2D, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_LINEAR)
GLES30.glTexImage2D(
GLES30.GL_TEXTURE_2D,
0,
GLES30.GL_RGB,
width,
height,
0,
GLES30.GL_RGB,
GLES30.GL_UNSIGNED_SHORT_5_6_5,
null
)
Note that we set null in the data of glTexImage2D, because we only need the memory address, we don't need to fill it temporarily, and we don't need to pay too much attention to the wrapping method, the size is also.
Finally, it needs to be attached to the framebuffer:
GLES30.glFramebufferTexture2D(
GLES30.GL_FRAMEBUFFER,
GLES30.GL_COLOR_ATTACHMENT0,
GLES30.GL_TEXTURE_2D,
textureId,
0
)
glFrameBufferTexture2D has the following parameters:
- target: the target of the framebuffer (draw, read, or both)
- attachment: The type of attachment we want to attach. Currently we are attaching a color attachment. Note the 0 at the end means we can attach multiple color attachments. We will refer to it later in the tutorial.
- texttarget: the type of texture you wish to attach
- texture: the texture itself to attach
- level: The level of the multi-level gradual distance texture. We leave it at 0.
Of course, in addition to the color attachment, there are also depth and stencil buffers, which do not need to be paid attention to here for the time being. The complete code is as follows:
private var fboBean: FboBean? = null
private fun useFbo(width: Int, height: Int) {
val frameBuffers = IntArray(1)
GLES30.glGenFramebuffers(1, frameBuffers, 0)
val textures = IntArray(1)
GLES30.glGenTextures(1, textures, 0)
val textureId = textures[0]
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, textureId)
//纹理过滤
GLES30.glTexParameteri(
GLES30.GL_TEXTURE_2D,
GLES30.GL_TEXTURE_MIN_FILTER,
GLES30.GL_NEAREST
)
GLES30.glTexParameteri(GLES30.GL_TEXTURE_2D, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_LINEAR)
GLES30.glTexImage2D(
GLES30.GL_TEXTURE_2D,
0,
GLES30.GL_RGB,
width,
height,
0,
GLES30.GL_RGB,
GLES30.GL_UNSIGNED_SHORT_5_6_5,
null
)
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, frameBuffers[0])
GLES30.glFramebufferTexture2D(
GLES30.GL_FRAMEBUFFER,
GLES30.GL_COLOR_ATTACHMENT0,
GLES30.GL_TEXTURE_2D,
textureId,
0
)
val status = GLES30.glCheckFramebufferStatus(GLES30.GL_FRAMEBUFFER)
if (status != GLES30.GL_FRAMEBUFFER_COMPLETE) {
throw RuntimeException("Failed to create texture.")
}
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, 0)
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, 0)
fboBean = FboBean(frameBuffers[0], textureId, width, height)
}
3.1 How to use it?
In Android OpenGL ES Learning (7) - Texture , we first get the texture id of the original image by loading the image, and we only need to display the size of the image in the screenshot, so when we load the original image, we can use fbo:
texture = loadTexture(TAG, MainApplication.context, R.mipmap.wuliuqi)?.apply {
Log.d(TAG, "useVaoVboAndEbo() called width = $width, height = $height")
useFbo(width, height)
Log.d(TAG, "bind frame buffer succeeded")
}
3.2 Rendering
So how to render it? As mentioned earlier, when we use GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, fboId), we have switched from the default framebuffer to our own framebuffer, and then all rendering is on your fbo texture.
Then it's very simple, follow these steps:
- Bind FBO GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, fboId)
- Render texture data GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, id)
- Unbind FBO GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER,0)
- Render the original image or the texture of the FBO
The complete code is as follows:
override fun onDrawFrame(gl: GL10?) {
resetMatrix()
// 步骤1：使用 glClearColor 设置的颜色，刷新 Surface
fboBean?.apply {
GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT )
// 绑定 FBO
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER,fboId)
GLES30.glViewport(0, 0, width, height)
//解决倒影
//rotate(180f)
texture?.apply {
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, id)
}
GLES30.glBindVertexArray(vao[0])
GLES30.glDrawElements(GLES30.GL_TRIANGLE_STRIP, 6, GLES30.GL_UNSIGNED_INT, 0)
val startTime = System.currentTimeMillis()
val bmp = readBufferPixelToBitmap(width, height)
image?.post {
image?.setImageBitmap(bmp)
}
val endTime = System.currentTimeMillis()
Log.d(TAG, "zsr onDrawFrame: ${
endTime - startTime}")
// 解绑 FBO
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, 0)
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER,0)
}
GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT )
GLES30.glViewport(0, 0, screenWidth, screenHeight)
resetMatrix()
// 正交投影
if (aspectRatio > 1) {
Matrix.orthoM(UnitMatrix, 0, -aspectRatio, aspectRatio, -1f, 1f, -1f, 1f)
} else {
Matrix.orthoM(UnitMatrix, 0, -1f, 1f, -aspectRatio, aspectRatio, -1f, 1f)
}
GLES30.glUniformMatrix4fv(uMatrix, 1, false, UnitMatrix, 0)
//显示 fbo 的图片
fboBean?.let {
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, it.textureId)
}?: GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, texture!!.id)
//使用原图
// GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, texture!!.id)
GLES30.glBindVertexArray(vao[0])
GLES30.glDrawElements(GLES30.GL_TRIANGLE_STRIP, 6, GLES30.GL_UNSIGNED_INT, 0)
}
Then you will find out, how is it reflected?
That is because the coordinates of FBO are frame buffer coordinates, which are opposite to texture coordinates (picture source (https://blog.csdn.net/yu540135101/article/details/102912319)) :
Quad RBO (Buffer Object Attachment)
Similar to VBO and VAO, RBO (Renderbuffer Object) is the buffer object of the texture attachment, which stores the data in OpenGL's native rendering format, which is optimized for off-screen rendering to the frame buffer.
Using RBO is a very good choice if you have multiple render objects or have depth and stencil attachments.
The renderbuffer object stores all rendering data directly into its buffer without doing any texture format conversion, making it a faster writable storage medium. However, renderbuffer objects are usually write-only, so you cannot read them (eg using texture access). Of course you can still read it with glReadPixels, which will return the pixels of a specific region from the currently bound framebuffer, not the attachment itself.
Because its data is already in native format, it is very fast when writing or copying its data to other buffers. So operations like swapping buffers are very fast when using renderbuffer objects.
Creating an rbo and framebuffer is very similar:
//创建rbo
val rbos = IntArray(1)
GLES30.glGenFramebuffers(1,rbos,0)
Then bind this buffer object, so that all subsequent rendering buffer operations are on rbo:
GLES30.glBindRenderbuffer(GLES30.GL_RENDERBUFFER,rbos[0])
Since renderbuffer objects are usually write-only, they are often used for depth and stencil attachments, because most of the time we don't need to read values from the depth and stencil buffers, only care about depth and stencil testing.
When it comes to depth testing, we naturally think of our ninth chapter Android OpenGL ES Learning (9) - coordinate system and. 3D effect is achieved.
Create a depth and stencil renderbuffer object by calling glRenderbufferStorage
GLES30.glRenderbufferStorage(GLES30.GL_RENDERBUFFER,GLES30.GL_DEPTH24_STENCIL8,width,height)
This object is specifically designed to be used as a framebuffer attachment, rather than a general purpose data buffer (General Purpose Data Buffer) like a texture. Here we choose GL_DEPTH24_STENCIL8 as the internal format, which encapsulates 24-bit depth and 8-bit stencil buffer.
The last thing to do is to attach this renderbuffer object:
GLES30.glFramebufferRenderbuffer(GLES30.GL_FRAMEBUFFER,GLES30.GL_DEPTH_STENCIL_ATTACHMENT,GLES30.GL_RENDERBUFFER,rbos[0])
4.1 Rendering
As before, the 3D effect is first drawn on the FBO. At this time, the textureId of the FBO has the entire effect, so we can actually draw the texture, and there is no need to draw each surface with a for loop. :
override fun onDrawFrame(gl: GL10?) {
// notUseDeepTest()
resetMatrix()
fboBean?.apply {
// 绑定 FBO
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER,fboId)
GLES30.glBindRenderbuffer(GLES30.GL_RENDERBUFFER,rboId)
GLES30.glViewport(0, 0, width, height)
useDeepTest()
val startTime = System.currentTimeMillis()
val bmp = readBufferPixelToBitmap(width, height)
image?.post {
image?.setImageBitmap(bmp)
}
val endTime = System.currentTimeMillis()
Log.d(TAG, "zsr onDrawFrame: ${
endTime - startTime}")
// 解绑 FBO
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, 0)
GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER,0)
GLES30.glBindRenderbuffer(GLES30.GL_RENDERBUFFER,0)
}
// useDeepTest()
GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT or GLES30.GL_DEPTH_BUFFER_BIT)
resetMatrix()
GLES30.glBindVertexArray(vao[0])
//直接用 fbo 的纹理绘制即可
fboBean?.let {
GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, it.textureId)
}
GLES30.glDrawArrays(GLES30.GL_TRIANGLES, 0, 36)
}
The effect is as follows:
In this way, we have almost learned about FBO; basically we have mastered the principle of using FBO, thinking about the effect at the beginning, installing FBO in OES, basically we can write it ourselves.