foreword
The time to do this research and analysis should be a few days in early December last year (2022), mainly through UE's official live broadcast explanation and SIGGRAPH 2021 ( Radiance Caching for Real-Time Global Illumination ) and 2022 ( Lumen: Real-time Global Illumination in Unreal Engine 5 ) two sharings for learning and understanding. I didn't look at the specific source code, so the content of the article is all for personal understanding. At the same time, after Lumen has been updated, it seems that some methods have also been replaced.
What is Lumen
Lumen is UE5's real-time global illumination (Realtime Global Illumination) solution. The core is a real-time soft ray tracing system, but it also supports hardware ray tracing. The focus of this article is also on its soft ray tracing part.
To put it simply, the main way Lumen implements real-time GI is to use the directed distance field (SDF) to accelerate the intersection of light and the scene on the basis of screen space global illumination (SSGI), and store material lighting information through Surface Cache, that is, the basic process as follows:
- Use SSGI for ray tracing to get the global illumination information of a part of the pixels
- For pixels not hit by SSGI, global illumination information is obtained by interpolating using screen space probes (screen space probes emit multiple rays to trace the scene). Sparse world space probes are also used here for high beams to reduce noise
- The ray tracing part uses the distance field of the scene to intersect, and obtains lighting information through Surface Cache (Surface Cache will be selectively updated at runtime)
SSGI
The method of global illumination in screen space is to use the depth, normal and other information of screen space for ray tracing. Lumen uses the structure of hierarchical Z buffer (Hierarchical Z Buffer) to speed up the intersection. Of course, the biggest problem with SSGI is that it is impossible to obtain off-screen information (just like the water surface realized by SSR, the reflection of lowering the head is also cut off along with the top of the screen).
screen space probe
It's just that SSGI can't meet the needs of global illumination, so ray tracing needs to be performed in the scene. The traditional method is to emit one or more rays to intersect with the scene for each pixel on the screen, but Lumen believes that this method is relatively noisy, so it uses the method of placing probes at certain distances on the surface of the object in the screen space. way, and trace more rays at the probe, and cache the sampled Radiance. Probes are placed sparsely and evenly at first, but if a pixel cannot be interpolated using surrounding probes, a probe is placed at that pixel location (as shown in the image below).
Finally, the probe will perform a 3x3 convolution noise reduction, and finally interpolate the lighting result to the pixel. At the same time, BRDF and the lighting information of the previous frame are also used for importance sampling, thereby improving the sampling effect.
World Space Probe
It's just that the screen space probe still can't solve the high beam problem very well, so lumen introduces the world space probe, which places the probe more sparsely in the world space, but samples more directional light. At this time, lumen sets the tracking distance of the screen space probe to a shorter range, and uses the world space probe for a longer distance, but in order to prevent light leakage, the direction of the world space probe tracking may be different from the direction of the screen space probe. Although this also introduces a certain bias.
Mesh SDF与Surface Cache
As mentioned earlier, ray tracing needs to be performed in the scene. Lumen uses the directed distance field SDF to get the shortest distance from a point in space to the surface of the object (when it is inside the object, the distance is negative), and uses RayMarching to quickly obtain the intersection point of the ray and the scene. Because the model is usually represented by a triangular surface, the distance field corresponding to the model needs additional calculation, and the representation accuracy will be lost to a certain extent. Lumen generates Mesh SDF separately for each static mesh, which is generated when the model is imported.
SDF only provides intersections and normals, and cannot provide information such as material lighting at light intersections. At the same time, it also needs to cache some expensive calculations, so Lumen uses Surface Cache to store material lighting information of objects. Simply put, a mesh is captured and stored by multiple axis-aligned Mesh Cards.
Surface Cache is constantly updated during runtime, but not all of them are updated every frame, but a part of them is selected according to the priority related to the last use time and last update time. For the calculation of Surface Cache direct lighting, shadow maps will be reused, and the part that cannot be solved will be traced to the light source to determine whether it is in the shadow. Indirect lighting is similar to the screen space probes mentioned above. Lumen will place probes on the surface at a certain density and trace rays, interpolating the results to surrounding texels.
Global SDF与Voxel Lighting
The above content can actually run through in theory, but there are usually a large number of meshes in the scene, and it is too expensive to traverse all Mesh SDFs for RayMarching, so Lumen generates a Global SDF for the entire scene (dynamic objects and static objects respectively cache, only the changed parts are updated at runtime).
Compared with Mesh SDF, Global SDF has lower accuracy. Lumen chooses to use Mesh SDF for close-range tracking (the objects that need to be intersected starting from this cell are maintained in the scene cell, so only intersecting Part of the object), Global SDF is used for long-distance tracking. However, there is another problem here... Surface Cache is based on the object. When the ray intersects with the Global SDF, the intersection point is found but it is not known which mesh is hit, and the surface cache of the object cannot be sampled. . Therefore, Lumen uses Voxel Lighting to voxelize the surroundings of the camera and store the lighting information in Surface Cache in it, so that after obtaining the intersection with GDF, it can correspond to Voxel to obtain lighting information.
Summarize
So the core idea of Lumen is to perform faster ray tracing on the scene expressed by SDF on the basis of SSGI, and at the same time selectively update the lighting information in the scene to reduce the pressure, and use screen probes and world probes to further reduce noise. Among them, the Mesh Card is used to capture the surface of the object and stored in the Surface Cache. The Surface Cache is used to provide the surface material and lighting information of the object that cannot be provided by the SDF. The voxelized scene is used to provide lighting information in a larger scene.
(Again: This is an old Lumen scheme that has been personally understood, and there may be errors in it, see the preface for reference)