Getting started with OpenGL/OpenGL ES: graphics API and terminology analysis

When I first came into contact with OpenGL, I also paid attention to the issue of Apple's father's abandonment. I thought to myself, Apple's father has abandoned OpenGL/Open LG ES, so what's the use of learning this thing?

It is worth noting that: 1. It took 4 years for Apple's own system to migrate to Metal. 2. Before Metal was launched, Apple was highly integrated with OpenGL ES and cooperated with corresponding layers and GLKit to assist developers to quickly use OpenGL. The deprecation of ES 3 and OpenGL ES is only for the underlying API dependence of Apple's internal system, and it does not mean that iOS developers will never use OpenGL ES. It’s just that the role has become a third party. After all, its cross-platform and stability make it difficult for existing development to give up, and these points are currently difficult for Metal. The project team is already very large, and will not be migrated to Metal for the time being, so learning Metal alone is not enough. 5. Therefore, learning needs to be done step by step through OpenGL -> OpenGL ES -> Metal. This is also a personal experience, and I have tried it before. I want to learn GPUImage or OpenGL ES, but I only know how to use it, but I don’t know the principle, so I need to lay a solid foundation first

Purpose of this article

• A Quick Look at the Graphics API

• Quickly understand the professional terms under OpenGL

1. Graphics API

1.1 Introduction to Graphics API

• OpenGL (Open Graphics Library): It is a cross-programming language and cross-platform programming graphics program interface. It abstracts computer resources as OpenGL objects, and abstracts operations on these resources as OpenGL instructions.

• OpenGL ES (OpenGL for Embedded Systems): It is a subset of OpenGL 3D graphics API, designed for embedded devices such as mobile phones, PDAs and game consoles, removing many unnecessary and low-performance API interfaces.

• DirectX: It is composed of many APIs, and DirectX is not a pure graphics API. The most important thing is that DirectX is a multimedia processing API on Windows. It does not support platforms other than Windows, so it is not a cross-platform framework. Classified according to the nature, it can be divided into four parts, display part, sound part, input part and network part.

• Metal: Apple unveiled a new platform technology for game developers that delivers 10x faster rendering performance for 3D graphics. Metal is a framework introduced by Apple to solve 3D rendering

1.2 What is the purpose of the graphics API to solve?

Simply put, it is to realize the underlying rendering of graphics

• For example, in game development, the rendering of game scenes/game tasks

• For example, in audio and video development, for data rendering after video decoding

• For example, in the map engine, for data rendering on the map

• For example, in animation, to achieve animation drawing

• For example, in video processing, adding a filter effect to the video

Essence: It is to use the GPU chip to efficiently render graphics images. The graphics API is the only way for iOS developers to get close to the GPU.

2. Analysis of professional terms under OpenGL

2.1 OpenGL context context

• Before the application calls any OpenGL instructions, it needs to create an OpenGL context context. This context is a very large state machine, which saves various states in OpenGL, which is also the basis of OpenGL instruction execution

• No matter which language the OpenGL function is in, it is a process-oriented function similar to C language. In essence, it operates on a certain state or object in the huge state machine of the OpenGL context. Of course, you must first put this object Set as the current object. Therefore, by encapsulating OpenGL instructions, it is possible to encapsulate OpenGL related calls into an object-oriented graphics API

• Since the OpenGL context is a huge state machine, switching contexts often has a large overhead, but different drawing modules may need to use completely independent state management. Therefore, multiple different contexts can be created in the application program, different contexts can be used in different threads, and resources such as textures and buffers can be shared among the contexts. Such a solution will be more reasonable and efficient than repeatedly switching contexts or modifying the rendering state in large numbers

2.2 OpenGL state machine

• A state machine is theoretically a machine that describes the various states an object goes through during its life cycle, the transitions between states, the causes and conditions that send transitions, and the activities performed during transitions. In other words, a state machine is a behavior, the sequence of states an object goes through in response to events during its life cycle and the responses to those state events. Has the following characteristics:

  • With memory function, can remember the current state

  • It can receive input, modify its current state according to the input content and its original state, and can have corresponding output

  • When entering a special state (stop state), it will no longer receive input, that is, stop working

• A computer can be said to be a typical state machine

  • The computer's memory (memory, hard disk, etc.) can remember the current state of the computer itself (the software currently stored in the computer, the data stored in the computer, the current settings of the computer, etc., are actually binary values ​​and belong to current status)

  • The computer's input device receives input (keyboard, mouse, file input), modifies its own state (modifies the value in memory) according to the input content and its own state (mainly refers to the program code that can be run), and can get output ( display the result to the screen)

  • When it enters a special state (shutdown state), it no longer receives input and stops working

• By analogy to OpenGL, it can be understood like this

  • OpenGL can record its own state (such as the color currently used, whether the blending function is turned on, etc.)

  • OpenGL can receive input (when calling the OpenGL function, it can be understood that OpenGL is receiving input), such as calling glColor3f, that is, after OpenGL receives this input, it will modify its "current color" state

  • OpenGL can enter a stopped state and no longer accept input. OpenGL will always stop working before the program exits

OpenGL is a state machine, it maintains its own state unless the user enters a command to make it change state

[Learning address]: FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media advanced development

[Article Benefits]: Receive more audio and video learning packages, Dachang interview questions, technical videos and learning roadmaps for free. The materials include (C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) Click 1079654574 to join the group to receive it~

3. Rendering

The operation of converting graphics/image data into a 3D space image is called rendering ( Rendering)

4. Vertex array VertexArray and vertex buffer VertexBuffer

• Vertex: refers to its vertex position data when drawing a graph, and this data can be directly stored in an array or cached in GPU memory

• Images in OpenGL are composed of primitives. In OpenGL ES, there are three types of primitives: points, lines, and triangles. Developers can choose to set the function pointer, and when calling the drawing method, the vertex data is directly transferred from the memory, that is to say, this part of the data was stored in the memory before, and is called the vertex array. The method with higher performance is to allocate a piece of video memory in advance, and transfer the vertex data to the video memory in advance. This part of the video memory is called the vertex buffer.

5. Pipeline

• Pipeline: It can be understood as a rendering pipeline. In fact, it refers to the process in which a bunch of original graphic data passes through a pipeline, undergoes various changes and processes, and finally appears on the screen.

• Rendering graphics under OpenGL will go through nodes one by one. And such operations can be understood as pipelines. It can be imagined as a pipeline, each task is executed like a pipeline, and there is a sequence between tasks. The reason why it is called a pipeline is because the graphics card comes in a fixed order when processing data, and strictly follows this order. Just like water stays from one end of a pipe to the other, this sequence cannot be broken.

• GLShaderManagerFixed pipeline: Simply understood as the process of rendering images, we can only implement a series of shader processing by calling the fixed pipeline effect of the class.

• Programmable pipeline: Simply understood as the process of processing graphics, we can use custom vertex shaders and fragment shaders. Since the usage scenarios of OpenGL are very rich, fixed pipelines or storage shaders cannot complete every task. At this time, the relevant parts are opened to be programmable.

6. Shader program Shader

• Turn fixed rendering pipeline architecture into programmable rendering pipeline. Therefore, OpenGL also needs to specify a shader program compiled by shader before actually calling the drawing function. Common shaders mainly include Vertex Shader, Fragment Shader/Pixel Shader, Geometry Shader, and Tessellation Shader. Until OpenGL ES3.0, only the two most basic shaders, vertex shader and fragment shader, are supported

• OpenGL handles shaders the same as other compilers. Through steps such as compiling and linking, a shader program (glProgram) is generated, and the shader program includes the operation logic of both the vertex shader and the fragment shader. When drawing in OpenGL, the vertex shader first operates on the incoming vertex data. In Assembly by Primitives, vertices are converted to primitives. Then rasterization is performed to convert vector graphics, such as primitives, into rasterized data. Finally, pass the rasterized data into the fragment shader for calculation. Fragment shaders operate on each pixel in the rasterized data and determine the pixel's color.

6.1 Vertex Shader Vertex Shader

• Generally used to process each vertex transformation of graphics (rotation/translation/projection, etc.)

• A vertex shader is a program used in OpenGL to calculate the attributes of vertices. A vertex shader is a vertex-by-vertex program, which means that each vertex data is executed once. Of course this is parallel, and other vertex data cannot be accessed during the vertex shader operation

• Generally speaking, the typical vertex attributes that need to be calculated mainly include vertex coordinate transformation, vertex-by-vertex lighting operations, and so on. The operation of converting the vertex coordinates from its own coordinate system to the normalized coordinate system happens here

6.2 Fragment Shader Fragment Shader

• Generally used to process the color calculation and filling of each pixel in the graphics

• A fragment shader is a program used to calculate the color of a fragment (pixel) in OpenGL. A fragment shader is a program that operates pixel by pixel, that is to say, each pixel will execute a fragment shader, and of course it is also parallel.

7、GLSL （OpenGL Shading Language）

The OpenGL shading language is the language used for shading coding in OpenGL, that is, short custom programs written by developers. They are executed on the GPU (Graphic Processor Unit) instead of part of the fixed rendering pipeline. , enabling programmability at different levels in the rendering pipeline. GLSL's shader code is divided into two parts: VertexShader (vertex shader) and Fragment Shader (fragment shader)

8. Rasterization

• Rasterization is the process of converting vertex data into fragments. It has the function of converting the graph into an image composed of rasters. Each element in the fragment corresponds to a pixel of the framebuffer

• Rasterization is actually a process of turning geometric primitives into two-dimensional images. This process consists of two parts. The first part of the work: determine which integer grid areas in the window coordinates are occupied by primitives; the second part of the work: assign a color value and a depth value to each area. The rasterization process produces fragments

• The process of converting the mathematical description of an object and the color information related to the object into pixels for corresponding positions on the screen and colors for filling pixels is called rasterization, which is a process of converting an analog signal into a discrete signal

9. Texture

Textures can be understood as pictures. When rendering graphics, it is necessary to encode and fill pictures. In order to make the scene more realistic, the pictures used here are often called textures. In OpenGL, it is more customary to call textures rather than pictures.

10. Blending

• After the test phase, if the pixel is still not culled, the pixel's color will be blended with the color attached to the color in the framebuffer. The blending algorithm can be specified by the function of OpenGL. However, the blending algorithm provided by OpenGL is limited. If a more complex blending algorithm is required, it can generally be implemented through a pixel shader. Of course, the performance will be worse than the original sound blending algorithm.

• Blending is mixing two colors together. Specifically, it is to mix the original color of a certain pixel position with the color to be painted in a certain way, so as to achieve a special effect

11. Transformation matrix Transformation

If you want to translate, scale, and rotate the graphics, you need to use the transformation matrix

12. Projection matrix Projection

It is used to convert 3D coordinates to 2D screen coordinates, and the actual lines will also be drawn in 2D coordinates

13. Rendering on the screen/swapping buffer SwapBuffer

Render buffers generally map system resources, such as windows. If you render the image directly to the buffer corresponding to the window, you can display the image to the screen. However, it is worth noting that if there is only one buffer per window, then during the drawing process, the screen is refreshed, and the window may display an incomplete image.

A regular OpenGL program will have at least two buffers. What is displayed on the screen is called the screen buffer, and what is not displayed is called the off-screen buffer. After a buffer is rendered, the image is displayed on the screen by exchanging the on-screen buffer with the off-screen buffer.

Because the refresh of the display is generally carried out line by line, in order to prevent the swapping of buffers, the images in the upper and lower areas of the screen belong to two different frames, so the exchange generally waits for the signal that the display is refreshed, and the display is refreshed twice. Interchange in the interval, this signal is called vertical synchronization signal, this technology is called vertical synchronization

After using the double buffer and vertical synchronization technology, the frame rate cannot fully reach the highest level allowed by the hardware because it always waits for the buffer exchange before rendering the next frame. In order to solve this problem, three buffers are introduced. Technology, while waiting for vertical synchronization, alternately render two off-screen buffers back and forth, and when vertical synchronization occurs, the screen buffer and the recently rendered off-screen buffer are exchanged to achieve the purpose of fully utilizing hardware performance

Original link: Getting Started with OpenGL/OpenGL ES: Graphics API and Analysis of Professional Terms - Nuggets