The scene view area of Overload has a picking mouse function. After clicking the picked object, it will be displayed in the Inspector panel. This article analyzes the principle behind the mouse pickup function.
1. OpenGL FrameBuffer
There are two common ways to implement mouse picking: rendering ID to texture and ray casting intersection. Overload uses rendering id to texture, and its implementation requires the use of OpenGL's frame buffer, so you must first understand OpenGL's frame buffer.
The cache we generally discuss refers to the window cache by default, which is displayed directly in the window. We can also create a custom cache to let the GPU pipeline render to the texture, and then use this texture elsewhere. And the data in the texture is only a binary value, not necessarily a color, and any meaningful data can be written.
If we want to create a framebuffer object, we need to call glGenFramebuffers() and get an unused identifier. When using the frame buffer, you need to call glBindFramebuffer(GL_FRAMEBUFFER, bufferID) binding first. If you want to render to a texture map, you need to call glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENTi, textureId, level) to associate the level level of the texture map to the frame buffer attachment. If rendering also requires depth cache and template cache, then a rendering cache is also needed.
The rendering cache is also an efficient memory area managed by OpenGL. It can store data in a specific format, and it only makes sense if it is associated with a frame buffer. Calling glGenRenderbuffers can create a rendering buffer, and binding operations are also required when operating it. Use glBindRenderbuffer when binding.
After seeing this, do you think the frame buffer is too complicated to use? In fact, the settings of the frame buffer are all fixed-format codes, and the routines are basically the same. Let’s string them together with pseudocode first. Assuming that our program is designed for process, we first call the init function for initialization, and then the main loop continuously calls the display function for rendering. The approximate pseudo code is as follows:
init() {
glGenFramebuffers(1, &m_bufferID); // 生成帧缓存
glGenTextures(1, &m_renderTexture) // 生成纹理对象
// 设置纹理格式
glBindTexture(GL_TEXTURE_2D, m_renderTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glBindTexture(GL_TEXTURE_2D, 0);
// 将纹理作为颜色附件绑定到帧缓存上
glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, m_renderTexture, 0);
glGenRenderbuffers(1, &m_depthStencilBuffer); // 生成渲染对象
// 设置渲染对象数据格式
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_STENCIL, p_width, p_height);
// 配置成帧缓存的深度缓冲与模板缓冲附件
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, m_depthStencilBuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, m_depthStencilBuffer);
}
display() {
// 1. 绑定帧缓存
glBindFramebuffer(GL_FRAMEBUFFER, m_bufferID);
// 2. 渲染物体到帧缓存
glClearColor();
glClear();
draw();
// 3. 解绑帧缓存
glBindFramebuffer(GL_FRAMEBUFFER, 0);
// 4. 使用帧缓存渲染出来的纹理
...
glActiveTexture();
glBindTexture(GL_TEXTURE_2D, id);
}The code in the init function remains essentially the same.
2. Overload’s encapsulation of FrameBuffer
Overload encapsulates FrameBuffer into a class Framebuffer, and the code is located in Framebuffer.h and Framebuffer.cpp. First look at the Framebuffer.h file. The Framebuffer class is defined as follows. If you are not familiar with the terms in the comments, you need to learn OpenGL.
class Framebuffer
{
public:
/**
* 构造函数，会直接创建一个帧缓冲
* @param p_width 帧缓冲的宽
* @param p_height 帧缓存的高
*/
Framebuffer(uint16_t p_width = 0, uint16_t p_height = 0);
/**
* 析构函数
*/
~Framebuffer();
/**
* 绑定帧缓冲，对其进行操作
*/
void Bind();
/**
* 解除绑定
*/
void Unbind();
/**
* 对帧缓冲的大小进行改变
* @param p_width 新的帧缓冲宽度
* @param p_height 新的帧缓冲高度
*/
void Resize(uint16_t p_width, uint16_t p_height);
/**
* 帧缓冲的id
*/
uint32_t GetID();
/**
* 返回OpenGL纹理附件的id
*/
uint32_t GetTextureID();
/**
* 返回渲染缓存的id，这个方法在Overload中其他地方没有使用
*/
uint32_t GetRenderBufferID();
private:
uint32_t m_bufferID = 0; // 帧缓冲的id
uint32_t m_renderTexture = 0; // 纹理附件的id
uint32_t m_depthStencilBuffer = 0; // 渲染缓存的id
};Let’s first look at the implementation of its constructor:
OvRendering::Buffers::Framebuffer::Framebuffer(uint16_t p_width, uint16_t p_height)
{
/* Generate OpenGL objects */
glGenFramebuffers(1, &m_bufferID); // 生成帧缓冲id
glGenTextures(1, &m_renderTexture); // 生成颜色缓冲纹理
glGenRenderbuffers(1, &m_depthStencilBuffer); // 生成渲染缓存
// 设置m_renderTexture纹理参数
glBindTexture(GL_TEXTURE_2D, m_renderTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glBindTexture(GL_TEXTURE_2D, 0);
/* Setup framebuffer */
Bind();
// 将纹理设置为渲染目标
glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, m_renderTexture, 0);
Unbind();
Resize(p_width, p_height);
}The frame buffer, texture, and rendering cache objects are directly generated in the construction, and the texture is associated with the frame buffer as a color attachment. Let’s look at the resize method again.
void OvRendering::Buffers::Framebuffer::Resize(uint16_t p_width, uint16_t p_height)
{
/* Resize texture */
// 设置纹理的大小
glBindTexture(GL_TEXTURE_2D, m_renderTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, p_width, p_height, 0, GL_RGB, GL_UNSIGNED_BYTE, 0);
glBindTexture(GL_TEXTURE_2D, 0);
/* Setup depth-stencil buffer (24 + 8 bits) */
glBindRenderbuffer(GL_RENDERBUFFER, m_depthStencilBuffer);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_STENCIL, p_width, p_height);
glBindRenderbuffer(GL_RENDERBUFFER, 0);
/* Attach depth and stencil buffer to the framebuffer */
Bind();
// 配置深度缓冲与模板缓冲
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, m_depthStencilBuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, m_depthStencilBuffer);
Unbind();
}The sum of these two methods is basically the same as the previous pseudo-code init function, but it is encapsulated in an object-oriented way.
3. Mouse Picking Principle
Mouse picking in Overload first renders the object's ID into the texture, reads the corresponding pixel value on the image based on the mouse position, and then decodes it to obtain the object's ID. In the red box in the figure below are the three key steps of this function:
Let's first look at the RenderSceneForActorPicking function. This function renders the objects, cameras, and lights in the scene. The rendering methods of the three of them are very similar. Taking the rendering camera as an example, the code is as follows:
/* Render cameras */
for (auto camera : m_context.sceneManager.GetCurrentScene()->GetFastAccessComponents().cameras)
{
auto& actor = camera->owner;
if (actor.IsActive())
{
// 对摄像机的id进行编码，设置到Shader的unfiorm中
PreparePickingMaterial(actor, m_actorPickingMaterial);
auto& model = *m_context.editorResources->GetModel("Camera");
auto modelMatrix = CalculateCameraModelMatrix(actor);
// 绘制摄像机，其覆盖区域的像素全部是其id
m_context.renderer->DrawModelWithSingleMaterial(model, m_actorPickingMaterial, &modelMatrix);
}
}There is a special function PreparePickingMaterial, which turns the three bytes of the ID into colors and saves them in the u_Diffuse variable. This variable will be used in the Shader. The core code is shown in the red box in the picture below. This encoding method is a commonly used method for writing information into images and can be used directly for reference.
Shader is definitely needed to draw in FrameBuffer. The Shader of Overload is encapsulated into a material. Special materials are required for picking. The variable m_actorPickingMaterial in the RenderSceneForActorPicking function stores this material. We trace the code and look for the initialization of this variable, and we can find the following code:
/* Picking Material */
auto unlit = m_context.shaderManager[":Shaders\\Unlit.glsl"];
m_actorPickingMaterial.SetShader(unlit);
m_actorPickingMaterial.Set("u_Diffuse", FVector4(1.f, 1.f, 1.f, 1.0f));
m_actorPickingMaterial.Set<OvRendering::Resources::Texture*>("u_DiffuseMap", nullptr);
m_actorPickingMaterial.SetFrontfaceCulling(false);
m_actorPickingMaterial.SetBackfaceCulling(false);It seems that this Shader is saved in the file Unlit.glsl. Also note that u_DiffuseMap is set to null. Remember this, this is intentional, and the devil is hidden in these details.
We open this file and analyze this Shader:
#shader vertex
#version 430 core
layout (location = 0) in vec3 geo_Pos;
layout (location = 1) in vec2 geo_TexCoords;
layout (location = 2) in vec3 geo_Normal;
layout (std140) uniform EngineUBO
{
mat4 ubo_Model;
mat4 ubo_View;
mat4 ubo_Projection;
vec3 ubo_ViewPos;
float ubo_Time;
};
out VS_OUT
{
vec2 TexCoords;
} vs_out;
void main()
{
vs_out.TexCoords = geo_TexCoords;
gl_Position = ubo_Projection * ubo_View * ubo_Model * vec4(geo_Pos, 1.0);
}
#shader fragment
#version 430 core
out vec4 FRAGMENT_COLOR;
in VS_OUT
{
vec2 TexCoords;
} fs_in;
uniform vec4 u_Diffuse = vec4(1.0, 1.0, 1.0, 1.0);
uniform sampler2D u_DiffuseMap;
uniform vec2 u_TextureTiling = vec2(1.0, 1.0);
uniform vec2 u_TextureOffset = vec2(0.0, 0.0);
void main()
{
FRAGMENT_COLOR = texture(u_DiffuseMap, u_TextureOffset + vec2(mod(fs_in.TexCoords.x * u_TextureTiling.x, 1), mod(fs_in.TexCoords.y * u_TextureTiling.y, 1))) * u_Diffuse;
}There’s not much to say about the Vertex Shader of this GPU program. Just calculate the NDC coordinates of the grid and you’re done. What is puzzling is the last line of code of Fragment Shader. Let me talk about the conclusion first. This line of code is equivalent to FRAGMENT_COLOR = u_Diffuse. As for why, to put it simply, u_DiffuseMap is set to null in the application, but when it is passed to the CPU, the texture with a null value is set to an empty texture. The size of this empty texture is one pixel, and its value is pure white, so its sampling results are all 1.0.
See the following code for empty context initialization:
See if there is only one pixel and the value is 1.0.
Having said this, the core details of texture rendering required for picking are basically finished. Let's take a look at how to read this texture.
First get the following mouse position. Since it is drawn with imgui, the absolute position of the mouse needs to be transformed into the relative position of the image. First bind the FrameBuffer and use glReadPixels to read the pixels. Note that the image format is RGB, which is consistent with the settings for initializing the FrameBuffer. You must pay attention to these details, and there are many mysteries. Finally, the pixel is decoded to obtain the ID of the scene object.
Reading code means understanding these details so that we can imitate them and use them in our own projects!