Unity rendering pipeline process

Unity rendering pipeline process:

1. CPU packs data, materials, normals
   a. Culling objects (objects outside the camera will be culled)
   b. Rendering sorting (sorting objects with a given order and depth)
   c. Sending data (packing all data and sending it to the GPU )
   d. Call Shader: SetPassCall, DrawCall
2. GPU rendering pipeline
   a. Vertex shader, which will be executed vertex by vertex. Convert model space to clip space. Can handle vertex offsets and vertex lighting.
   b. After vertex shader: rasterization stage
   1. Clipping: Clip the model outside the clipping space
   2. Convert to NDC (normalized device coordinates, using perspective division)
   3. Culling (back side culling, front culling, etc.), culling vertices in a specific direction
   4. Viewport transformation (conversion to screen coordinates)
   5. primitive assembly
   6. Rasterization (will produce aliasing)
      c. Fragment shader, performed pixel by pixel.
      d. After the fragment shader: the output merge stage
   7. Alpha testing
   8. template test
   9. depth test
   10. Color mixing
       e. Frame buffer
       f. Post-processing stage (CPU calls GPU rendering pipeline)
       g. GPU rendering to screen

Multi-camera rendering

1. Each camera will render the vertices of the objects in the camera.

2. Each camera will run a complete rendering pipeline process

3. By default, objects other than the skybox are cleared, and clear conditions and rendering order are set through ClearFlags and Depth.

4. Render overlays will occur if objects from other cameras are unclear.

With one camera clear, one camera reserved seems to be able to achieve the left and right eye effect of 3D video. (though takes twice as long to render)

CPU rendering pipeline

Object culling (objects outside the camera will be culled)
frustum culling

That is: use the area formed by the frustum of the visual range of the camera to perform collision detection with the object.
Optimization: However, complex object collision detection is very performance-consuming, so a simple collider Box Colider will be used for wrapping, called AABB bounding box.

Hierarchical culling

The culling of specific objects is performed through the camera. Eliminate objects at a specific level (objects that are not checked are excluded).
It feels like this can achieve fast hiding of objects. (Reduce rendering by setting objects to hidden layers without destroying them).

Occlusion Culling

Objects that are completely occluded by opaque objects are eliminated by judging the object position and occlusion relationship.
Rendering sorting (sorting objects with a given order and depth)
is sorted by the Render Queue value, and sorted by depth when they are equal.

1. Opaque objects default to 2000
2. Transparent objects default 3000
3. Values ​​greater than 2500 are understood as semi-transparent queues, which will be sorted from back to front by depth
4. Less than 2500 is understood as an opaque object, sorted by depth from front to back

Sorting opaque objects from front to back is helpful for optimization, and occluded objects will not be rendered.

Translucent objects are rendered from the back to the front to ensure that the rendering color is correct. The mixing order of translucent colors is different, and the resulting colors are different.
Translucent object interiors are not guaranteed to be rendered exactly from back to front. The resulting display effect: 1. There is no thickness like a patch,

Send data (pack all data and send to GPU)

Model data
format: obj format, fbx format (recommended)

obj data:

1. Vertex coordinates (3D)
2. Normal information (normal vector)
3. UV coordinates (two-dimensional data, the last bit is useless)
4. Index list, one row of data represents a triangular surface (find the vertex, normal, UV of each vertex through the index). Recording through the index helps to compress the data,

Call Shader: SetPassCall, DrawCall

GPU rendering pipeline

Vertex Shader
Vertex Shader in Unity

The most important task: transform the vertex coordinates from the model space to the clipping space:
transform the coordinates of the vertices through the MVP matrix.

1. Change from model space to world space:
   the reference system changes from the object itself to the unified coordinates of the world.
   In model space, the coordinate space of the modeling software is the coordinate generated by the model.
2. World space changes to camera space
   The reference system changes from world coordinates to camera-centered
3. Changing the camera space to the clipping space
   to flatten the camera's frustum into a 2x2x1 (CVV matrix)

Primitive assembly and rasterization

perform cropping

Clips excess pixels and images in clipping coordinate space.
Shapes that exceed the CVV will be clipped.

Spatial range: [w, w, 0] ~ [-w, w, w]
NDC - Normalized Device Coordinates

Prepare for conversion to screen coordinates Space range: The z value in the
[-1, -1, 1]~[1, 1, 0] space is the depth value, which will be retained in the following operations, and then depth culled
Come in handy.

Clip space conversion to NDC

P.xyz / Pw = NDC
At the same time, pay attention to different devices NDC and different
DX platforms: upper left corner [0, 0], lower right corner [1, 1]
OpenGL: lower left corner [0, 0], upper right corner [1, 1]

// Adapt to ndc coordinates under different platforms
o.screen_pos.y = o.screen_pos.y * _ProjectionParams.x;

Calculate NDC

The farthest we can get directly is only the coordinates in the clipping space. If you want to get the NDC coordinates, you need to calculate it yourself. But very simple.
The coordinate space in the clipping space is: [-w,-w,w] -> [w, w, 0]
The NDC coordinates are: [-1, -1,1] -> [1, 1, 0] The
clipping space coordinates Divide by w to get =>[-1, -1, 0] -> [1, 1, 0]

Method 1:
// Vertex shader

o.screen_pos = o.vertex;
// Adapt to ndc coordinates under different platforms
o.screen_pos.y = o.screen_pos.y * _ProjectionParams.x;

// Fragment shader
// Calculate screen space NDC
coordinates // Perspective division xyz range [-w,-w,w],[w,w,0] => [-1, -1,1]=>[1 ,1,0]
half2 screen_ndc = i.screen_pos.xy / (i.screen_pos.w + 0.000001);
half2 screen_uv = (screen_ndc + 1.) * .5; // [x, y] range: [-1, 1] => [0, 1]

Method 2:
// Vertex shader
o.screen_pos = ComputeScreenPos(o.vertex);

// Fragment shader
// Calculate screen space NDC
coordinates // [x, y] range: [-1, 1] => [0, 1]
half2 screen_uv = i.screen_pos.xy / (i.screen_pos.w + 0.000001);

backface culling

Objects facing away from the camera are culled.

Arranged by the index list of the triangle faces, if the arrangement order is clockwise, it is the front face, and if it is counterclockwise, it is the back face.

Screen Coordinate Space - Viewport Transform
Convert xy coordinates to screen space width and height.
Downconvert NDC space to coordinates in screen space

primitive assembly

Connect the vertices in the plane to form a closed triangle.

rasterization

Generates pixels for each fragment in each primitive in the plane.

Depth Value Z Normal
Vertex Color Tangent Position All Custom Data

Fragment shader Shader

Texture technology
Texture sampling
Sampling [0, 0] ~ [1, 1] in the texture coordinate UV. By UV mapping to the pixel of the texture, the pixel at that point is taken out.
Texture filtering mechanism
Small images are mapped to large areas: equivalent to zooming in on images.

1. Rounding - produces sharper edges
2. bilinear difference

Distortion caused by mapping a large image to a small area:
Mipmap, which is mapped by giving a set of images of different sizes. Use large images when displaying large areas, and small images for small areas.
Mipamp will only increase the storage footprint by 1/3. Each image size is sequentially decreased by 1/2.

texture addressing mode

What should be the value when the index exceeds UV.

1. Repeat mode - take a decimal (will form the effect of taking UV in a loop)
2. Clamp interception - Exceeding will take the maximum and minimum values.

texture compression format

RGBA 32bit
ASTC 4x4 block, ASTC 6x6 - good texture detail preservation, obvious compression
ETC2 8bits
PVRTC 4bits

lighting calculation

Light composition:

1. direct light
2. indirect lighting

lighting model

BRDF's physically based rendering model is too complex.

Empirical Lighting Model

The basic framework of the lighting model = direct light diffuse reflection + direct light specular reflection + indirect light diffuse reflection + indirect light specular reflection + ... (subsurface scattering, PDR, etc.)

1. Lambert diffuse lighting model
2. The Phong lighting model
   calculates the direction of the reflected light from the direction of the incident light and the normal vector. Knowing the reflected light, you can compare the angle between the line of sight and the reflected light to get how much light it sees.
3. The Blinn lighting model
   calculates its half-angle vector through the angle between the incident light direction and the normal vector. Approximates the direction of reflected light in half-angle directions (half-angle calculations are simpler).
4. Ground lighting model
5. Flat lighting model

ambient light

Can be simulated equivalently to indirect light.
Indirect lighting implementation:
Lightmap
Reflection Probe
Light Probe

output merge

The most important tasks: dealing with occlusion relations, dealing with translucent mixing.
Frame buffer FrameBuffer:

1. ColorBuffer ColorBuffer
2. Depth buffer DepthBuffer, Z-buffer
3. Template buffer StencilBuffer

Alpha testing

Fragments that are fully transparent, or that do not meet the given transparency will be discarded.

template test

Select an area for comparison, and it will be discarded if it is not in the area.

depth test

Tested with depth, failing will be discarded.
The depth of fragments that pass the test will be stored in the Z-buffer. (It can also be controlled not to write).
In Shader, it can be controlled by ZWrite and ZTest.
ZWrite: Whether to write to the Z-buffer
ZTest: Depth test rules

Depth testing technology in advance:
Early-Z, which occurs after the vertex Shader and before the primitive assembly.
It is an advance optimization technology that avoids unnecessary primitive assembly and fragment calculation through early testing.

Blending

Color blends semi-transparent pixels.

1. back to front
2. In general, ZWrite will be closed

Written in Shader such as:
Blend SrcAlpha OneMinusSrcAlpha
Blend SrcAlpha One

Post-processing

After the post-processing, the CPU can call the GPU to adjust the overall screen image after the shader work is completed.

---

Content source: "1. Overview of rendering pipeline"