In-depth review of front-end technology, today I also want to learn a lot~
1. Introduction
When playing 3D games, have you ever wondered how these 3D objects are rendered? How is the animation done? Why is there a situation where the model is worn, the shadow is wrong, and the protagonist cannot be reflected in the mirror? Answering these questions requires an understanding of real-time rendering. The most basic of these is the rendering pipeline.
"Rendering pipeline" , also known as graphics pipeline, is a conceptual model for a computer to render a 3D model to a 2D screen.
The rendering pipeline generally only refers to the rendering process of 3D polygon rendering, which is different from ray tracing (ray tracing). Ray tracing is based on the reversible principle of light path, emitting light from the viewpoint, when it hits the surface of the object, calculates the corresponding color and light intensity according to the surface material, and continues to calculate reflection and refraction, etc., and finally traces back to the light source or non-contributing point. In 3D polygonal rendering, rays are emitted from the object and eventually fall to the viewpoint.
The rendering pipeline is generally divided into 4 parts, application (Application), geometry processing (Geometry Processing), rasterization (Rasterization) and pixel processing (Pixel Processing). Different materials and different implementations may have different division methods, but the overall process is similar.
The rendering pipeline of "Real-time Rendering, 4th"
The organization of this article is based on the rendering pipeline mentioned in "Real-time Rendering, 4th" . In order to solve specific problems, the rendering process in OpenGL and WebGPU and the application in three.js are used as examples.
2. Application stage
Just imagine the 3D games we play, with all kinds of models, textures, motion, lighting, collision detection, creature AI, animations, and more. But for the GPU, it only cares about "primitives" .
Primitives are basic drawable units, generally referring to "points, line segments and triangles" , which are essentially a collection of vertices. For example, a line segment has two vertices and a triangle has three vertices. In general, primitives are at most triangles, since they always have the same number of vertices, and three vertices define a plane, which can then be conveniently treated as a two-dimensional plane. If there are four points, additional methods are required to ensure that they are on the same plane and do not produce concave polygons.
That's right, even a sphere can be represented by a triangle
Therefore, before entering GPU rendering, it is necessary to complete the corresponding calculations such as motion, animation, collision, AI, etc., and convert the content to be rendered into primitives. Finally, a series of primitives, instructions, textures, and various parameters are uploaded to the GPU.
Example: https://threejs.org/docs/index.html?q=Geometry#api/en/geometries/BoxGeometry
2.1 Primitive topology
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/ClickedPoints
When the mouse is clicked on the canvas, 1 pixel will be drawn on the corresponding position of the canvas (because 1 pixel is hard to see, in the example, the canvas is scaled 10 times, so it will look blurry).
Every time the mouse is clicked, a vertex is added to the primitive array. After the entire rendering process is completed, a white point is drawn on the canvas.
So how to draw lines and triangles? In WebGPU, through the pipeline primitive.topology, you can specify the topology of the vertices. Currently, there are the following five types. Other implementations have similar configuration methods.
type GPUPrimitiveTopology =
| "point-list" // 点
| "line-list" // 线
| "line-strip" // 线条带
| "triangle-list" // 三角形
| "triangle-strip"; // 三角形条带
// 如果是 -strip，还需要指定 stripIndexFormat。What is a triangle strip?
Generally speaking, our models are composed of continuous triangles, and there will be cases where multiple triangles share vertices. If vertices are used to represent triangles (v1, v3, v2) (v2, v3, v4) (v3, v5, v4)... drawing n triangles requires 3nvertices If these vertices are shared, drawing n triangles requires n + 2only vertices. Therefore, when we describe vertices, we omit these repeated vertices, which are strips.
But it should be noted that the vertices of the triangle are in order. Whether the order of the vertices of the triangle is clockwise (cw) or counterclockwise (ccw) determines whether the entire triangle face is facing the camera or facing away from the camera. This information is very important, and subsequent steps can remove faces facing away from the camera. For example, in the above strip, if (v1, v2, v3) is clockwise as the front in the subsequent steps, then the drawing of the next triangle should be (v2, v4, v3), (v3, v4, v5), (v4 , v6, v5)...
To draw a triangle strip, the vertex order should be like this
In WebGPU, the default is counterclockwise, and it can also be GPUFrontFaceconfigured to be clockwise or counterclockwise.
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/HelloTriangle
The triangles in this example are counterclockwise, and backface culling is not turned on. You could try turning on backface culling and adjusting it clockwise to see if it still renders.
// 实践解答
const pipeline = device.createRenderPipeline({
vertex: {
// ...
},
fragment: {
// ...
},
primitive:{
topology: 'triangle-list',
cullMode: 'back', // 'back' | 'front' | 'none'
frontFace: 'cw', // 'cw' | 'ccw'
},
});After the primitives and other information are uploaded to the GPU, the next step is to be processed by the GPU.
3. Geometry processing stage
At this point, we have at least some primitives. The geometry processing stage is divided into the following four functional stages, which process the primitives and finally obtain their coordinates in the screen space.
3.1 Vertex Shading - Vertex Shading
In the case of disregarding the topological way, that is, some vertices. At this stage, we process the vertices.
Vertex shading is to attach some attributes (such as color, material, normal) or make some modifications (such as adjusting position, discarding) to these vertices through the existing information. The most important thing at this stage is to determine the position of the vertex on the canvas, and the position is the only necessary output of the vertex shader.
Let's continue to look at the WebGPU example just now. The only thing the vertex shader does is to convert the 2-dimensional array [x, y] into the simplest vertex with only the position (x, y, z=0.0, w=1.0) vector.
const vs = `
// ...
[[stage(vertex)]]
fn main([[location(0)]] a_position : vec2<f32>) -> VertexOutput {
var output : VertexOutput;
output.position = vec4<f32>(a_position, 0.0, 1.0);
return output;
}
`;Vertex coordinates are in a homogeneous coordinate system, so although the position is a 3-dimensional vector, it needs to be represented by a 4-dimensional vector. It will be explained in detail later in the projection.
3.1.1 Coordinate Transform
Drawing a 2D triangle, determining the position of the vertices is easy. But in the actual scene, the object is 3D. In the 3D scene, we need to perform a series of coordinate transformations to determine the position of the vertex on the screen.
Pre-knowledge: For any point in two-dimensional or three-dimensional space, we can apply matrix transformation to perform affine transformation, such as translation, scaling, stretching and rotation. Therefore, at this stage, we will finally determine the position of the vertex on the screen through a step-by-step matrix coordinate transformation.
MVP(Model-View-Projection) matrix coordinate transformation process
Although usually all three transformations are applied simultaneously, the projection matrix differs from the other two in that perspective projection is not affine, and strictly speaking it is " almost" not representable by an orthogonal matrix transformation.
3.1.1.1 Model matrix
Taking the most basic cube model as an example, the relative coordinates of each vertex of the cube must first be obtained. We call these model coordinates. The origin of the three coordinate axes is at the center of the model, in other words, if a vertex is at the center of the model, its coordinates should be (0, 0, 0).
In 3D Canvas, the coordinates are usually right-handed, and the directions of the coordinate axes are shown in the figure
There may be multiple identical models in a scene, and these models can have different rotation, translation, and scaling transformations, so it is necessary to apply a model matrix to them, transform their coordinates into world coordinates, and place In the scene (world space, world space).
These are all 1x1x1 cubes, transformed by the model matrix to make them behave differently in world space
3.1.1.2 View matrix
When we are in different positions and our eyes look at different angles, the objects in front of us are different, which shows that the position and orientation of "our" are also very important. In a 3D scene, "we" is "camera" . Therefore, it is necessary to transform the model coordinates with the world as the origin into camera coordinates with the camera as the origin by applying the view matrix.
3.1.1.3 Projection matrix
There are two types of projection matrices. The objects we see with the human eye are always near and far, and the position of the vertex should also be related to the distance of the camera. Parallel lines even intersect at infinity. This is called perspective projection. It can still be implemented with a 4x4 matrix. Unlike perspective projection, under orthographic projection, the size of an object on the projection plane has nothing to do with its relative distance. Orthographic projections are often used in architectural blueprinting and design to ensure that objects do not change in size and relative to each other.
3.1.2 Lighting, animation and texture coordinate (UV) transformation
This part is an optional output of vertex shading, which will be explained after the main flow of the rendering pipeline.
Because the number of vertices is generally much smaller than the number of pixels, in order to improve performance, lighting calculations, shading, etc. can be performed in the vertex stage, but the accuracy is usually low. These are typically processed on a per-pixel basis in subsequent stages as GPU computing power increases. Vertex shaders tend to only output vertex related information.
3.1.3 Tessellation*
The more vertices, the more triangles, the more different planes can be expressed, and the more details can be supported. Tessellation uses a series of algorithms to add more vertices to the original primitives to form a finer model.
At the same time, because it adds more vertices, it also provides more room for subsequent displacement maps.
Example: https://threejs.org/docs/index.html#api/en/geometries/SphereGeometry
This example simulates tessellation by adjusting the corresponding parameters, and you can see that when the value is low, it looks less spherical (in WebGL, tessellation is not a programmable stage).
3.1.4 Geometry Shading*
Unlike tessellation, which adds vertices inside the primitive, it can transform the original primitive into zero or more new primitives by discarding vertices or adding extra vertices outside the primitive.
Turn triangles into more triangles, or line segments into polylines
There is a saying that it is often used to achieve the rendering of a large number of particles. For example, each particle only uses one vertex. At this stage, it is expanded into polygons of different shapes or discarded, and a large number of particles are rendered by means of texture maps.
In practice, however, this shader is often poorly performing, and most people, even most GPU vendors, would argue that it should be avoided in practice. Geometry shaders are not available in WebGL and WebGPU.
3.2 Projection - Projection
There are two types of projection: parallel projection and perspective projection. Orthographic and perspective projections are generally used in 3D rendering.
Perspective projection, orthographic projection, isometric projection, oblique projection
Example: https://threejs.org/examples/?q=camera#webgl_camera
See the difference between perspective and orthographic projections by switching between different cameras.
3.2.1 Orthographic Projection
Orthographic projection is a kind of parallel projection. The biggest feature of this type of projection is that there is no near big and far small, and parallel lines are still parallel after projection.
It is also subdivided into orthographic projection, a projection that draws three views of an object, and axonometric projection, a projection in which multiple surfaces can be seen at the same time. Axonometric projection is further divided into equivalent axonometric projection, orthographic projection and orthographic projection, which respectively indicate whether the scaling ratio of the three coordinate axes is the same (or whether the angles are equal). The isometric projections are the same, and there are 2 axes in the orthographic The same, positive and three tests are different.
In 3D rendering, there are 6 configurable parameters for the orthographic projection, which are the left, right, up, down, and near distances of the hexahedron. The final projection matrix is as follows:
3.2.2 Perspective projection
The characteristic of perspective projection is that objects that are farther away appear smaller, and parallel lines eventually meet at a point.
In 3D rendering, there are 4 configurable parameters for perspective projection, namely fov, aspect ratio of viewing cone, near plane and far plane. The final projection matrix is as follows (OpenGL):
in
The w of (x, y, z, w) after the operation may not be 1, and the hardware will automatically handle it.
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/HelloTriangle
In this example, if the output vertex coordinate w is not 1.0, modify the value of w slightly, what will happen to the triangle? Why? Does this explain how perspective projections can be represented by matrix transformations even though they are not affine?
const vs = `
// ...
[[stage(vertex)]]
fn main([[location(0)]] a_position : vec2<f32>) -> VertexOutput {
var output : VertexOutput;
output.position = vec4<f32>(a_position, 0.0, 1.0); // 这里是 w
return output;
}
`;| w = 3.0 | w = 0.9 |
|---|---|
 |
 |
3.3 Clipping - Clipping
Clipping is to clip polygons that do not need to be displayed on the screen, so as to reduce the data that needs to be processed later and improve performance. There are 2 types of clipping: 2D clipping and 3D clipping.
2D clipping removes polygons that are not in the viewing plane or viewport. For polygons that are half in and half out, vertices are added.
There are several types of 3D clipping, and some clipping can be enabled or disabled individually during the rendering process.
View frustum clipping: Removes polygons that are not within the view frustum and in the near clipping plane and outside the far clipping plane.
Backface culling: Removes polygons whose back (or front) faces are facing us, based on vertex order.
Occlusion Culling: Culls a polygon if it is fully occluded by another polygon.
(In a previous practice we tried enabling backface culling)
3.4 Screen Mapping - Screen Mapping
After the above series of processing, we will get a very square unit cube, where (x, y, z) are all [-1, 1]. Now, we need to convert the normalized device coordinates (Normalized device coordinate) to screen coordinates, such as x from [-1, 1] => [0, 1024), y from [-1, 1] => [0, 768 ), and some implementations map z to [0, 1].
3.5 Geometry Phase Review
Looking back at the entire geometry processing stage, its input is a series of primitives, and then after vertex coloring (required, at least the position of the output vertices), surface subdivision and geometric coloring are performed to make the primitives more refined. Finally, through Clipping and screen mapping, get the window coordinates of all the vertices that need to be drawn, and other information added to the vertices by the vertex shader (such as color, normal vector, texture UV coordinates, etc.).
Fourth, the rasterization stage
Rasterization is also known as scan conversion. Although our vertex connections and triangles are continuous, the screen is composed of pixels, so we need to discretize our primitives into fragments (fragment, a collection of covered pixels), so that subsequent pixels processing and display. After rasterization, we can determine which pixels belong to which primitives and get the corresponding fragments.
 |
 |
This stage mainly includes two processes: primitive assembly and triangle traversal.
4.1 Triangle assembly (primitive assembly) - triangle setup (primitive assembly)
We said before that the primitive is a triangle, but it is actually just the position and topology data of some vertices. At this stage, we will actually connect the coordinates of these points into triangles or line segments (through the difference, find the corresponding position, color, normal equation, etc.), and provide it to the next step.
For the difference between color and normal, please refer to the following polygon coloring
4.2 Triangle traversal - triangle traversal
In this part, through various algorithms, determine which pixels these primitives will cover, and ensure that no pixel is covered by multiple triangles (saving rendering resources, and avoid rendering order affecting rendering results), and get the slice corresponding to the primitive Yuan.
4.2.1 Drawing boundaries
"Digital Differential Analyzer (DDA)"
Let the coordinates of two points on the line segment be (x1, y1), (x2, y2), then:
Assuming that the slope m <= 1, then set x = x1, every time the x coordinate increases by 1, the y coordinate increases by m, since m may be a decimal, draw after rounding y.
Assuming slope m > 1, swap x and y.
If m > 1, there will be a large number of grids that cannot be drawn, which can be solved by exchanging xy
However, this algorithm involves floating-point arithmetic, and its performance is relatively poor.
"Bresenham Algorithm"
This algorithm only needs to perform integer addition and subtraction and bit operations, avoiding floating-point operations. As an optimization of DDA, you can learn more about it here: https://zhuanlan.zhihu.com/p/74990578
"Anti-aliasing"
One way of thinking: split a pixel into more areas, and perform color mixing according to the scanned pixels.
4.2.2 Filled triangles
The most common here is to use the scan line fill method.
Every time a side is scanned, it will be +1. If it is an odd number, the scanned pixel will be filled. It should be noted that if a vertex is scanned, it is necessary to use whether the adjacent vertices are on both sides of the scan line to determine whether to enter or leave the polygon. This algorithm can also be optimized.
Five, pixel processing stage - Pixel Processing
So far, we have obtained all fragment information. Next, we need to process these fragments.
5.1 Pixel Shading (Fragment Shading)* - Pixel Shading(Fragment Shading)
In this part, we need to calculate the color of each pixel.
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/HelloTriangle
In the example, the fragment shader only outputs a
vec4representing color. What should I do if I change this triangle to green? What is the 4th value used to control? Why did the modification not work?
const fs = `
[[stage(fragment)]]
fn main() -> [[location(0)]] vec4<f32> {
return vec4<f32>(1.0, 0.0, 0.0, 1.0);
}
`;In real scenes, fragment shading is very complicated, including the material, highlights, shadows, textures, reflections of objects... It is a big pit. We will introduce it later.
5.2 Pixel Merging - Pixel Merging
At this point, we have obtained the pixel color corresponding to each fragment, and then we need to combine the colors of all fragments. At this point, there are likely to be some triangles occluding each other, so some algorithm is needed to decide how to draw. Most of the hardware is implemented by the depth buffer (z-buffer) algorithm. When drawing, store the depth of the pixel to be drawn. When preparing to cover it, first test whether the depth of the pixel to be drawn is less than the drawn depth. If it is less than that, it will cover and update the depth information, otherwise it will remain unchanged.
However, the depth buffer algorithm only has two results: rendering and non-rendering, and it cannot render translucent objects.
Our rendering pipeline is complete here.
6. Summary
etc! How do you sum it up? After learning this rendering process, it seems that there are no answers to any questions! How to light, how to reflect light, how to map, how to animate bones? It seems that I still don’t know anything... I only remember that the vertex coloring stage outputs the vertex position, the fragment coloring stage outputs the color, and then there is no more...
It doesn't matter, let's review first:
Vertex shading: Determine the position of vertices through a series of coordinate transformations, and can also provide some additional information.
Surface subdivision: Add vertices inside the primitive to make the primitive more refined, look more delicate, and facilitate displacement maps.
Geometry Shading: Primitives can be added or removed, generally not.
Clipping: Removes parts that will not be rendered to improve performance and render fineness.
Screen Mapping: Convert coordinates from unit cube to screen coordinates.
Primitive assembly and traversal: determine the pixel corresponding to the triangle.
Pixel Shading: Determine the color of each pixel.
Binning: Merge the pixels of all fragments.
After these steps are completed, after a series of tests and mixing, it is finally ready to be displayed on the screen.
Next, we will try to answer some more questions.
Seven, rendering practice
7.1 Animation and simple shading
When it comes to animation, it should be to change the position of the vertices, to change the position, that is to apply the transformation matrix. Therefore, we can provide additional parameters to the vertex shader and change its value through requestAnimationFrame.
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/RotatingTriangle
example, if not used
Hilo``3d.Matrix3(), is it possible to create an animation matrix that can be rotated? Tip:mat3x3It is a data structure aligned to 16 bytes, so there are 12 elements in the array, but the 4th, 8th, and 12th elements are meaningless.
// 实践的解
let rotateAngle = 1;
function getModelMatrix(){
let matrix = CSSStyleValue.parse('transform', `rotate(${rotateAngle++}deg)`).toMatrix();
const { a, b, c, d, e, f } = matrix;
modelMatrixData.set([a, c, e, b, d, f, 0, 0, 1]);
modelMatrixData.copyWithin(8, 6, 9);
modelMatrixData.copyWithin(4, 3, 6);
return modelMatrixData;
}Speaking of shading, it should be to modify the output of the fragment shader. What should I do if I want the triangles to have different colors?
Example: https://06wj.github.io/WebGPU-Playground/#/Samples/MultiAttributeColor
In the example, we can provide color information in the vertex shader stage and use it directly in the fragment shader stage.
Eh? We only provide three colors of red, green, and blue for the three vertices, why is the final output in color? This is because, by default, the output of the vertex shader is subtracted and sent to the fragment shader. If you don't want to be deltaed, we can set the delta interpolatemethod to flat.
const vs = `
struct VertexOutput {
[[builtin(position)]] position : vec4<f32>;
[[location(0), interpolate(flat)]] v_color : vec3<f32>;
};
// ...
`;
const fs = `
[[stage(fragment)]]
fn main([[location(0), interpolate(flat)]] v_color: vec3<f32>) -> [[location(0)]] vec4<f32> {
return vec4<f32>(v_color, 1.0);
}
`;When set, the entire triangle is red, since the color of this face will depend on the representative vertex.
Next, we'll introduce light into the scene.
7.2 light
The most complex part of rendering a 3D scene is simulating light and light interactions, on the one hand, as realistically as possible, and on the other hand, the performance is as good as possible. This part is divided into light source, light interaction, lighting model, shading and light effect.
7.2.1 Light source
Let's take the ThreeJS light source as an example to introduce several common light sources:
https://threejs.org/docs/index.html?q=Light#api/en/lights/AmbientLight
AmbientLight: It will evenly illuminate all objects in the scene.
Parallel light source DirectionalLight: emits directional parallel rays, generally used to simulate sunlight.
HemisphereLight: the color of the sky and the ground is different (blue sky, green ground)
Point light source PointLight: Emit light uniformly from a point to all around. Generally used to simulate light bulbs.
Rectangular light source RectAreaLight: emit light from a rectangle to the surroundings, used to simulate windows or blue sky lights.
Spotlight SpotLight: Conical light source, the center of the cone weakens to the surroundings, simulating a flashlight.
7.2.2 Light Interaction
"Ambient light (ambient)"
Some lights are difficult to find a direct light source, such as in a room without lights, which is usually not completely dark. Ambient light is used to simulate the lighting effect produced by this indirect light source. It illuminates all surfaces uniformly without direction, and the lighting effect is only related to the intensity of the light source and the characteristics of the model surface itself.
"Diffuse"
This interaction is a major factor in determining the brightness and color of an object. After the light hits the object, it is evenly reflected in all directions. Therefore, regardless of the angle of the observer, the lighting effect of the same position of the object is the same. The lighting effect is related to the light intensity, the diffuse reflectance of the object, and the angle between the light angle and the normal line of the object surface.
"Specular"
But the surface of most objects in life is not completely rough, so it will not be uniformly reflective. Smooth objects reflect more strongly in the direction corresponding to the light, producing specular highlights until specular reflection occurs.
Diffuse smooth specular
In 3D scene rendering, specular highlights depend on the intensity of the specular light and the specular reflectance of the object's surface.
Thinking: Why can't the mirror reflect the main character in previous 3D games?
Because this lighting model is calculated based on the surface of a single object, only the object itself and the light source affect the surface color of the object, and there is no reflection from other objects. For specular reflections, the last calculated result can only be the surface highlight. Therefore, there is no way to make a real specular reflection effect.
How to solve this problem? Two paths can be taken: better lighting models and environment maps.
7.2.3 Lighting model
Illumination models can be divided into local illumination models and global illumination models. In the local illumination model, the reflection of the object (that is, the color of the object) depends only on the "properties of the object surface itself" and "the color of the direct light source" , including the Phong illumination model and its improved version, the Blinn-Phong illumination model. In the global illumination model, the interaction between light and various objects in the entire scene and the surface of the object will be considered additionally, such as multiple reflections, transmissions, refractions, etc., which require a considerable amount of calculations, including ray tracing, radiation coloring, and photon mapping. .
In real-time 3D rendering, generally only the local illumination model is discussed.
The Phong lighting model takes into account ambient, diffuse, and specular highlights in light interactions and adds them together to form the final color. An improved version of the Phong lighting model improves the accuracy and efficiency of calculating specular highlights.
7.3 Polygonal Shading - Polygonal Shading
With the lighting model in place, we need to determine the color of the polygon faces. There are several methods of coloring:
7.3.1 Flat Shading - Flat Shading
A triangle has three vertices, we select a representative vertex (the first vertex, or the normal and color mean of the triangle face), when coloring the triangle, calculate the lighting effect for the color and normal of this vertex, and calculate the result Applies to the entire polygon.
This works well because only 1 lighting calculation per polygon is required. However, it usually results in all triangles of the model being clearly visible. Generally, this is not what we want unless we are doing low poly art.
7.3.2 Gouraud Shading - Gouraud Shading
Golaud shading is a difference shading method. First, calculate the normal of each vertex, then calculate the lighting for each vertex, and then calculate the lighting of each edge through bilinear interpolation, and then calculate the lighting of each edge. Value out the lighting of each pixel.
| Get vertex normal - neighbor polygon mean | Obtain the lighting of edges and pixels - bilinear difference |
|---|---|
 |
 |
This shading method can render the surface of the object smoothly, but some highlight information will be lost. Imagine a huge triangular surface, if only the middle part of the triangular surface produces specular highlights, and the vertices are not illuminated, then the entire plane will have no lighting effect
7.3.3 Phong Shading - Phong Shading
Feng's shading is similar to Golaud shading, but it does not make a difference to the light, but to make a difference to the normal, get the normal of each pixel, and calculate the light for each pixel.
| Get vertex normal - neighbor polygon mean (same as above) | Obtain the normal of the edge and the pixel - bilinear difference |
|---|---|
 |
 |
Vertex normals, plane shading, Golaud shading, Phong shading
Compare the ThreeJS example to understand the difference between the three coloring methods:
MeshPhongMaterial：https://threejs.org/docs/index.html?q=mesh#api/en/materials/MeshPhongMaterial
打开 flat shading 的 MeshPhongMaterial：https://threejs.org/docs/index.html?q=mesh#api/en/materials/MeshPhongMaterial
MeshLambertMaterial：https://threejs.org/docs/index.html?q=lam#api/en/materials/MeshLambertMaterial
| Flat Shading MeshPhongMaterial* | Gouraud Shading MeshLambertMaterial | Phong Shading MeshPhongMaterial |
|---|---|---|
 |
 |
 |
*Difference between Feng's lighting model and Feng's shading model
7.4 Texture Mapping - Texture Mapping
According to the previous theory, the vertex color is the color of the light and the material itself. However, if I want to achieve a brick wall, no matter how many vertices I add, no amount of lighting, no matter how good the shading method is, I can't get this effect...
Texture maps come in handy at this time. It provides more drawing details without changing the geometry itself.
Texture maps originally generally referred to diffuse maps (diffuse mapping). It maps pixels on a 2D texture directly onto a 3D surface. With the development of multi-pass rendering, there are now more and more kinds of textures. Such as bump maps, normal maps, displacement maps, reflection maps, specular maps, and ambient occlusion maps... Let's briefly introduce some of them.
7.4.1 Diffuse map - diffuse mapping
Imagine that we put wallpaper on a white wall. This should be a relatively simple matter, as long as it is aligned and pasted... Pasting a rectangular material on a rectangular surface is similar to pasting wallpaper. It only needs to be in a 2-dimensional space. Just map the coordinates.
Like pasting wallpaper, this kind of texture does not have smooth reflection, specular highlight and other lighting effects after pasting it. It only affects the background color of diffuse reflection.
Practice: In ThreeJS Editor, add a PointLight, a Plane, and add a texture map for the Plane. If you did everything right, you should see something like the following. Alternatively, you can import the JSON below.
But sometimes, we need to complete some 3D mapping. For example, we are making a globe and need to paste the flat world map outside the sphere. Or take a panoramic photo, and when previewing, you need to stick it inside the sphere.
At this time, we need a more complex mapping relationship. Find the correspondence between the coordinates (x, y, z) on the geometry and the coordinates (u, v) of the 2D texture, which is generally called uv mapping. The texture coordinate transformation we mentioned in the vertex shading stage before refers to this process.
There are also some complex geometries where it is difficult to find the correspondence between the points on it and the 2D material plane. For this kind of geometry, we can wrap it with a simple geometry (such as a ball or a cube), apply a texture on the simple geometry, and when we need to draw a point on the complex geometry, project from the center to the simple geometry, and take the simple geometry texture information.
Practice: Learn about ThreeJS panoramas attached to a sphere and panoramas attached to a cube. Is there distortion around them from different angles? Are there other visual differences?
A panorama attached to a sphere: https://threejs.org/examples/?q=panorama#webgl_panorama_equirectangular
Panoramas attached to cube faces: https://threejs.org/examples/?q=panorama#webgl_panorama_cube
But the diffuse reflection map can only affect the pixels to be drawn. No matter how 3D the map is, once it is placed in a 3D scene with complex lighting, the whole object will still look flat. To make an object feel 3D without violating harmony, it needs to be able to interact with light.
7.4.2 Bump map - bump mapping
To solve this problem, we can provide more information when calculating lighting. According to the previous conclusions, in addition to color and material, there are normals that affect lighting interaction and shading. Therefore, in actual use, in order to make the object more 3D, the more common method is to use the normal map (normal mapping, 3-channel bump map) in the bump map.
The normal map is a 24-bit RGB image, and (r, g, b) represent the normal (x, y, z) values after mapping from [-1, 1] to [0, 255]. The special one is z, because the normal z direction of the object facing the camera is always negative. We stipulate that the sign of z is reversed to participate in the mapping, which also leads to the value of b always >= 128, so that the normal map looks Always bluer.
X: -1 to +1 : Red: 0 to 255
Y: -1 to +1 : Green: 0 to 255
Z: 0 to -1 : Blue: 128 to 255
Get the normal map, and the effect of pasting the normal map on the plane Pane, you can see that the plane can interact with light
Eh? Where did this sticker come from? It seems to be rendered from the scene on the left... There are corresponding 3D objects to render this texture, it would be better to use real objects directly, why stick it on the plane? Thinking questions, follow-up answers.
7.4.3 Displacement map - displacement mapping
The normal still looks flat after it's pasted...because it only affects the normal, not the real geometry height, so it's flat at flat angles (closer to 180 degrees).
To fix this, we can instead overlay a displacement map to affect its geometric height. The displacement map is black and white because only the height information needs to be manipulated. It adds a lot of extra geometry while providing more accurate 3D rendering, and is the most expensive of its kind. In the past, it was generally only used for offline rendering for a long time.
Practice: In ThreeJS Editor, add DirectionalLight, AmbientLight, Plane and Box, and add displacement map for Box. Adjust the number of surfaces in the Box Geometry and observe the effect of the displacement map. You can also import the JSON below.
Effect of Seg = 1, 5, 20
7.4.4 Environment map - environment mapping
If the object is very smooth and will reflect specularly, or is very transparent and will refract, what should we do? To review, our light is a local illumination model, which only depends on the properties of the object surface itself and the color of the direct light source, and the specular highlight is just a highlight... I really want to make a mirror. If I can't use ray tracing, what should I do? ?
This is where the environment map comes in.
The environment map is similar to the 2D texture, which is to surround a ball or cube on the outside of the object and paste the corresponding texture. When the object needs to draw reflection or refraction, the material information corresponding to the cube is searched according to the reflection or refraction light path.
Practice: Learn about ThreeJS environment maps: https://threejs.org/examples/?q=env#webgl_materials_envmaps
| reflection | refraction |
|---|---|
 |
 |
"7.4.5 Shadow map - shadow mapping / shadowing projection"
The shadow map is the shadow information in the environment.
If the light source is replaced with a camera, the depth image of the scene can be obtained from the perspective of the light source. When drawing the scene, if the depth of the corresponding position is deeper, it means that this position must not be illuminated by the corresponding light source, and the rendering of light can be ignored...
Do this for each light source, and you'll be able to draw out the effect of the shadows.
7.4.6 The meaning of texture
Sometimes when we are playing 3D games, some objects clearly exist, but we cannot see them on the water surface or in the mirror, or other objects in the sun have shadows, but they have no shadows. Why?
Since they are expensive to render in real time in partially rendered models, they are usually textures! If there is no corresponding element on the pre-rendered environment map or shadow map, then naturally they cannot be seen in reflections or shadows. In order to improve performance, some content in the scene needs to be pre-rendered as textures offline. This part is also called texture baking.
7.5 Multi-pass Rendering - Multiple-pass Rendering
There are so many things to do, it is difficult to do it all at once, so we can render through multi-passes, and after completing different tasks each time, combine them through some algorithm. This way, each step can be cached separately. When the scene changes, some pass renderings that have already been done can remain unchanged.
There seems to be no conclusion on what *beauty pass refers to. Here it refers to the default rendering that does not consider the relationship between scenes.
Practice: Learn about ThreeJS multi-pass rendering: https://threejs.org/examples/?q=ao#webgl_postprocessing_sao
Adjust the output via the controls in the upper right corner, and observe the output for Default (Beauty), Ambient Occlusion (AO), Shadows (Depth), and Normal (Normal), as well as the combined output.
| Beauty | TO THE | Depth | Normal |
|---|---|---|---|
 |
 |
 |
 |
7.6 Post Processing - Post Processing
After the entire rendering process is over, we have all the outputs of each stage, as well as the final information of all pixels. At this point, we also have the opportunity to transform the entire scene and apply some effects. This stage is post-processing.
Some common post-processing stages to do: edge detection, object glows, volumetric lights (god beams), signal glitch effects.
Practice: Learn about some common Post Processing: https://vanruesc.github.io/postprocessing/public/demo/#antialiasing
| edge detection | shine | volume light |
|---|---|---|
 |
 |
 |
appendix
References:
Reference to how the article is organized: https://www.uni-weimar.de/fileadmin/user/fak/medien/professuren/Computer_Graphics/course_material/Graphics___Animation/5-GFX-pipeline.pdf
For details, please refer to: https://www.logiconsole.com/real-time-rendering-1/
Rendering pipeline Chinese PDF: https://positiveczp.github.io/%E7%BB%86%E8%AF%B4%E5%9B%BE%E5%BD%A2%E5%AD%A6%E6%B8% B2%E6%9F%93%E7%AE%A1%E7%BA%BF.pdf
Process reference: https://zhuanlan.zhihu.com/p/61949898
resource:
Shaders：https://thebookofshaders.com/
ShaderToy：https://www.shadertoy.com/view/llj3zV
Free or paid 3D models: https://sketchfab.com/3d-models?features=downloadable&sort_by=-likeCount