Large-scale 3D interactive development and optimization practice

development background

Thanks to the explosion of the "Metaverse" concept some time ago, various companies have launched activities or channels using 3D scenes.

Compared with the traditional 2D page, the 3D scene has one more dimension, and more content can be displayed on the same screen, which can completely display the information of objects and products.

The corresponding disadvantage is that the user's usage method changes, and the user needs additional learning costs. In addition, the amount of development, art resources and equipment for generating 3D models required in the initial stage are also increased costs.

In this context, our team received a development request for an interactive project from the Food Channel. We hope that through the display and interaction of 3D scenes, as an attempt and exploration of future shopping, we can meet users' needs for a beautiful and novel future. . Make shopping scenarios and entertainment, and bring users a wonderful shopping experience.

Front-end framework selection

Compared with the previous 2D projects, the 3D project has changed mainly in the performance of the client. In the hope of not relying on app client support and being able to run in as many environments as possible, our first solution is to realize 3D project realization on the Web side.

development kit

First of all, we consider mature development kits, such as unity/egret, etc., but these development kits have some problems that we cannot bypass, such as:

Commercial use requires a fee
It needs to be developed in other languages (such as C#), which will cost a lot for the team to learn
The file size of the packaged output is too large
The official documents are not detailed enough, and the learning curve is shaky

Engine Name/Comparison Dimensions	Use price (weight 50%	Getting started with scripting (weight 30%	Scene construction (weight 20%	Support model format (weight 10%	The richness of community information (weight 30%	Support web publishing (one vote veto
Unity 3d	3	7	10	8	10	Y
Laya	4	9	7	7	7	Y
Egret	10	8	7	7	6	Y
Cocos2d-js						N
Godot	10	7	7	8	7	Y

Due to the above reasons, there is no satisfactory choice for the team in the development kit, and we look for development tools in other directions.

Open Source Rendering Library

In addition, two 3D rendering libraries that are widely used by the Web front end are compared:

◦The components provided by three.js are small in granularity and relatively basic, and can do a high degree of customized secondary development, but if you need to develop an interactive project, you need to develop more components

◦babylon.js not only provides small-grained basic components, but also encapsulates close-to-out-of-the-box components. And comes with performance measurement tools, providing convenient debugging methods and optimization strategies

After trialing various development kits/rendering libraries in the team, babylon.js was finally selected as the rendering layer library of the project, and the business logic was re-developed on the components provided by it.

Project scene construction

render hierarchy

The project rendering level is generally divided into two layers: 3D scene layer and HUD layer

1.3D scene layer, as the name suggests, renders 3D scenes, consisting of interactive models such as character models, building models and treasure chests

2. The HUD layer renders 2D UI content such as interactive buttons, pop-up windows, and product lists required by the business

Originally, babylonjs supports mixed rendering of 3D and 2D content, but if you use babylonjs for both rendering, you need to use a uniform resolution when setting the two content, but on current mobile devices, it can support pixel resolution (such as iPhone 14's The pixel resolution is 1170x2532) only a small part of the rendering is not stuttered. On most devices, it can only support smooth running at a logical resolution (such as the logical resolution of iPhone 14 is 390x844), but setting such a resolution will make the 2D layer rendering blurred, so use a layered method for rendering.

The 3D scene layer is rendered by babylonjs, while the HUD layer is rendered using the traditional DOM method through the react framework.

Second 3D rendering layer

The rendering layer is divided into 3D scene layer and HUD layer, which brings a problem. When it is necessary to render 3D content on the HUD layer, such as displaying 3D models, it is necessary to add another layer of 3D rendering layer, and the 3D rendering layer does not stop The rendering method is called to respond to user operations and play animations, which consumes a lot of computing resources of the CPU and GPU, and also occupies the memory space for storing model vertex information and map textures. Therefore, when multiple 3D rendering layers coexist, certain management is required to optimize performance. We employ the following strategy to manage multiple 3D rendering layers:

◦Reinstancing when displaying another 3D rendering layer, and suspending the rendering of the original 3D rendering layer

◦Destroy when no display is needed, restore the rendering method call of the original 3D rendering layer

In order to minimize the occupation of resources and improve the rendering performance of the project.

Interactive component development

Impact checking

babylonjs comes with a method for detecting collisions between models, but if you use the high-precision model provided by the designer to directly call the collision detection method, the amount of calculation will be very large. Although there is no serious lag phenomenon on the test equipment, it has been Make the device hot.

Therefore, it is necessary to use an invisible, thin-faced "air wall" model that surrounds the model for collision detection. In the early stage of the project, the "air wall" model needs to be provided by the designer, and a low-precision enclosing model is made in the modeling software based on the original model. In the subsequent iterative development, our team developed a "one-click air wall generation" tool, which automatically generates low-precision models, reduces the number of resources delivered by designers, and reduces the chance of errors when updating models.

Lens obstacle avoidance

Because the project uses a third-person lens, the lens is a certain distance away from the character model. When the character walks or the user controls the angle, the lens may collide with the architectural model or scene model, resulting in the phenomenon of "lens passing through the model".

The built-in lens of babylonjs does not have the function of avoiding the model. When the product has no processing experience, we made the following two solutions:

The periphery of the lens is surrounded by an invisible model, and the collision detection is performed with the building and scene models just like the characters, so that the lens will not enter the model.

The advantage of this approach is that the built-in collision detection method can be used and no additional development is required. But the shortcomings are also obvious. The user has no expectation of the collision between the lens and the model, and the user always feels that the lens is unnatural and out of control.

The camera and the character are connected by a stick model, and the collision detection with the building and scene models is also called on the stick model. When a collision occurs at a certain position of the stick model, the camera will move to the position between the character and the collision point. While avoiding the lens entering the model, it also prevents the model from being interspersed between the character and the lens, causing the problem that the user cannot find the character.

The effect achieved by this method is in line with the lens movement logic of some 3D games that are also third-person perspectives. The user feels more natural and will not lose control. The amount of additional development introduced is also within a controllable range.

Handover of resources with the design team

model format

Among the many 3D model formats, we chose the .gltf format. Compared to other model formats, .gltf can reduce redundant data in 3D formats that is not related to rendering, thus ensuring a smaller file size.

At present, the 3D materials are relatively large, which is undoubtedly fatal for the loading experience of the mobile terminal. Therefore, a format with a smaller size also has a higher preference weight.

In addition, .gltf is a summary of various 3D formats in the past two decades. It uses the optimal data structure to ensure maximum compatibility and scalability, and supports more expansion while having a large capacity. , such as support for multiple textures, multiple animations, etc.

So .gltf became the only material format we agreed with the visual.

Model output process

Originally, the modeling software used by the designer workflow was C4D, but in the process of resource handover, we found several problems:

1. Lack of export gltf file function. Some versions of C4D cannot export models in gltf format; some versions can export, but there is a problem with the export. And because of some renderer support problems used by the designer, the C4D version cannot be easily updated.

2. The size of the exported model is not uniform. Maybe because of some version of C4D export problems, or some settings in C4D failed to export to the gltf file, the size of the model exported by the designer several times is not uniform, for example, the character model is several times larger than the architectural model.

3. Export material information is lost. When the designer is modeling, because the model may be used in multiple channels, such as rendering promotional images, most of the cases will use a third-party renderer for rendering. At this time, the unique materials of these renderers may be used in the model. When these materials are exported to gltf files, the information of these unique materials will be lost. When importing it into the scene on the page, designers will find that the displayed effect is far from what they saw in the modeling software.

After many discussions with designers, we have established a workflow for exporting models:

After the C4D modeling is completed, export the file in FBX format, and then import it into the blender software that supports gltf better. Designers can preview whether their materials are lost during the transfer process. The model in the gltf file exported by blender Can also maintain a consistent size.

preset lighting

In the default rendering settings, we put the model output from the design side into the scene, and with the light source, there are only changes in light and shade, no shadows, and some three-dimensional effects are missing.

During our attempts to incorporate shadows, performance was found to be severely impacted. After consulting the rendering principle, I found that every time a shadow is added on a plane, it is equivalent to rendering the scene one more time, and the rendering pressure doubles.

After communicating with the design side, it was decided to pre-add the projection of the building on the texture of the floor. This method can have a better effect on most scenes with fixed models, and the shadows of characters can be simulated by moving with the model using static pictures.

rendering optimization

compressed texture

During the development period, it was found that flashbacks are easy to occur on older iPhone devices. It should be that the memory used by the page exceeds the upper limit.

The largest resource used in the project is the model gltf file. Check the content of the file. The texture map accounts for a large part of the volume. After analyzing the resources, it is found that the size of many textures is 3K (3072x3072 pictures). According to the WebGL rendering principle, no matter What is the original format of the texture resource, and finally it needs to be decompressed before rendering, which means that a texture needs to occupy 3072 x 3072 x 3Byte = 27MB in memory, and it needs to be transmitted to the GPU after decompression, which is very likely when multiple textures are rendered at the same time Takes up a lot of memory.

After communicating with the design side, it was agreed to replace the lower-resolution textures on some models whose display distance cannot be very close.

In addition, usually the png/jpg format images used in 2D projects are not suitable for 3D rendering. They need to go through the above-mentioned decompression process before they can be read by the GPU.

In the field of 3D rendering, there are other formats suitable for GPU reading, such as ETC supported by Android, PVRTC supported by iOS, and a new generation of standard compressed texture format ASTC. They can be read by GPU without decompression, which can greatly reduce the intermediate The memory capacity occupied by decompression.

In the project, we use the gltf-transform tool to reduce the texture resolution and convert the format.

model reduction

The process of rendering a model in WebGL is to first use the vertex information of the model to determine the triangular faces, and then calculate the color to be displayed on each triangular face. Therefore, if the number of model faces can be reduced, the amount of calculation for each rendering can be reduced, and the rendering time required for each frame can be reduced.

As mentioned above, when the designer is modeling, he may be faced with the need to output renderings, without optimizing real-time rendering, so too many surfaces may be used in some places.

Referring to the optimization experience 1 of other students in the team, the gltf-transform tool is used to automatically reduce the surface of the model. After repeated communication with the design tester, we determined the parameters ratio = 0, error = 0.0001

batch rendering

There is a concept of draw call in 3D rendering. A draw call is a drawing instruction from the CPU to the GPU. In one instruction, the CPU will transmit the triangle information to be drawn to the GPU, and how to calculate the color on the triangle. This method is called a material in a language that humans understand. So a draw call can only draw surfaces of the same material.

Because each draw call has these preparation actions, usually two draw calls will take more time than one.

In the model file, surfaces of the same material may not be defined in the same model, so the CPU will split these surfaces into different drawing instructions, increasing the number of draw calls.

There is an optimization method for this situation called batching, which can merge these faces of the same material so that they can be drawn in one draw call.

There is no tool to help us process model files in this work, but when loading model files at the front end, we can traverse the mesh mesh in the model and merge those using the same material.

It should be noted that the mesh with animation cannot be handled in this way, because the center of the merged object will change, for example, after two self-rotating balls are merged, they will revolve around the midpoint of the two balls.

Subsequent iterations

Model lazy loading and hierarchical loading

Although the scene displayed by the temporary project is not very large, and the simultaneous loading and rendering does not put too much pressure on the device, but when the scene grows to a certain extent, it is necessary to introduce lazy loading and hierarchical loading of the model.

◦Lazy loading strategy: load and insert the model into the scene when the camera moves close enough, and destroy the model far enough away from the camera.

◦Graded loading strategy: When the camera is far away, load a lower-precision model, and then switch to a higher-precision model when it is closer.

The above two strategies are the loading strategies used by larger 3D games now, which can reduce the number of faces drawn on the same screen and reduce the rendering pressure.

Graded rendering

At present, there is a huge gap in the performance of devices for accessing 3D projects. Some can run smoothly with special effects, and some can only basically run at the device resolution.

babylonjs comes with a hierarchical rendering function, which can detect the running frame rate in real time to determine whether to downgrade. In subsequent iterations, a hierarchical rendering strategy from pixel resolution plus special effects to device resolution basic rendering can be added.

real-time lighting

After using the above hierarchical rendering strategy, you can add real-time light and shadow effects on devices with better performance, and dynamically replace pre-baked textures

scene builder

In the previous project development process, designers, products, and operations all needed to output demos through the front-end to roughly experience the effect of the 3D scene and decide how to adjust in the next step. In order to solve this pain point, our team has developed a 3D scene building tool. Users can build 3D scenes by uploading gltf files and preview the rendering effect in real time.

And added interactive components deposited in the project to quickly generate 3D scene projects.

Reference source:

Talk about glTF file compression https://jelly.jd.com/article/61057292df18aa019e8a2967

Author: JD Retail Hu Junwen

Source: JD Cloud Developer Community