//
Editor's note: The development of cloud computing and network infrastructure provides better development opportunities for cloud rendering. Huawei Cloud has long been exploring and developing cloud rendering in the three directions of self-developed graphics rendering engine, industrial field rendering and AI accelerated rendering. Research. This LiveVideoStackCon 2023 Shanghai station invited Chen Pu from HUAWEI CLOUD to share some applications of cloud rendering in vertical scenarios.
Text/Chen Pu
Organize/LiveVideoStack
Good afternoon, everyone. It is a great honor to share and communicate with you today. The theme of this sharing is Huawei Cloud rendering practice - from 2D graphics to 3D space.
I joined Huawei in 2005. Since 2007, I have been responsible for the research and development of Huawei cloud computing, self-developed desktop cloud and artificial intelligence technologies. I am currently working on the research and development of Huawei cloud rendering services.
This sharing is divided into the following parts: Firstly, it introduces the scope of rendering technology and the problems that cloud rendering hopes to solve; secondly, it introduces some practices of Huawei Cloud Rendering in 2D, 3D and cloud-native rendering; then it introduces industrial images and AI accelerated rendering practice.
-01-
range of rendering techniques
When it comes to cloud rendering, you may directly think of cloud games, but in fact the two are not the same. Huawei Cloud is committed to applying rendering technology to various industries. Currently, it mainly includes two branches: real-time rendering and offline rendering. The landing scenes of real-time rendering include virtual exhibitions, car shows, virtual spaces, and digital humans. Effects are mainly used in scenes such as design renderings and film and television.
-02-
Problems Cloud Rendering Hopes to Solve
Some people tend to directly equate rendering with 3D graphics, so first explain the concept of rendering. Baidu Encyclopedia defines rendering as the last process of CG, and this explanation is biased towards offline rendering. Wikipedia and Baidu Encyclopedia have also mentioned rendering in browser engines and OpenGL. You can see that the rendered objects include 2D and 3D graphics. Wikipedia's definition of rendering is more accurate, it actually refers to the process of software using binary and formatted data to generate images.
We define cloud rendering as: realize the customer's graphics rendering business on the cloud, and the whole process goes from no graphics to 2D graphics to 3D graphics.
Our existing image rendering architecture is shown above, from the GPU at the lowest layer of 3D technology to the graphics card driver, DX, OpenGL, Vulkan runtime, 3D rendering engine, and then to 3D applications. It is difficult to directly use the underlying API to develop graphics rendering programs.
Another 2D application branch mainly includes common PC 2D applications such as Office, from GPU, graphics card driver, GDI runtime library defined by Microsoft, GUI framework to 2D applications.
-03-
2D rendering practice
Next, we will introduce our cloud 2D practice. The standard 2D graphics display process is shown in the figure above. First, the application program calls the GUI program library, and the GUI library is converted into the underlying GDI runtime library call. At this time, the runtime library is still in the user state and does not call the hardware. Then the GDI runtime calls the graphics card driver, and the graphics card driver calls the graphics card to complete image rendering and output to the display.
So to realize the remote display of 2D applications, does it only need to encode and transmit the rendered picture to the client? In fact, this method has disadvantages such as low encoding efficiency, large video bandwidth occupation, and color distortion caused by lossy compression, and is not suitable for 2D rendering on the cloud.
The rendering process we designed is to convert the image rendering call sequence into data after the graphics card driver obtains the 2D image rendering call, use the 2D rendering command processing program to process the data, encode the image and compress it losslessly, and finally transmit it to the client through the network Decoded display. Taking text input as an example, the input string is converted into pixel-level text by font drivers, and only needs to be displayed in a small update area on the screen, the amount of transmitted data is small, and lossless compression can be achieved. There are implementation plans for the above process on both Windows and Linux platforms.
-04-
3D rendering application cloud practice
3D cloud rendering architecture is similar to 2D rendering. Due to technical constraints in the early days, we chose to use GPU pass-through technology to bind the GPU to the virtual machine. The virtual machine can directly access the GPU to obtain high-performance graphics rendering capabilities, and then transmit the screen to the client through remote application technology.
The advantage of this method is that the virtual machine can directly install the standard graphics driver of the equipment manufacturer, which ensures compatibility and makes the performance of the graphics card close to that of the physical GPU. The disadvantage is that the GPU is bound to the virtual machine, which loses flexibility and costs more. But it is still the mainstream way to use GPUs on the cloud.
In 2012, we tried to transfer the rendering from the cloud to the client for remote completion. Essentially, we need to obtain the 3D instruction stream of the user program.
According to the Microsoft WDDM graphics card driver, the user application 3D image rendering call can be captured. Under this architecture, the driver program is divided into user mode and kernel mode. To implement 3D programs, you need to rely on the Direct runtime library, which calls the driver provided by the device manufacturer and interacts with the GPU through the kernel mode driver. Similar programs also exist for OpenGL.
So we processed all 3D application instruction streams and transmitted them to the client, and re-rendered on the client. After more than 50 game tests, the rendering function was basically realized. However, the bandwidth is unstable, and the implementation difficulty and cost are relatively high.
In 2012 we started developing software GPU virtualization. Its basic principle is realized through two kinds of virtual machines, one is a virtual GPU virtual machine serving customers, and the other is the aforementioned physical GPU virtual machine using pass-through technology.
We have built memory communication between the two virtual machines on the virtualization platform, installed the virtual graphics card driver in the virtual GPU virtual machine, used the driver to capture the 3D image driver call, and used the shared memory to transmit it to the physical GPU virtual machine. In the physical GPU virtual machine Use the pass-through physical graphics card for rendering, and return the screen after rendering.
There are two challenges faced by this method. One is that it is difficult to implement driver compatibility due to the multiple versions of DirectX. Second, the internal shared memory transmission needs to be further accelerated. The disadvantage is that the solution itself belongs to the resource preemption model, and the isolation ability is poor. Secondly, the research and development of virtual graphics card driver compatibility is very manpower.
In 2013, we cooperated with Nvidia to support hardware GPU virtualization on the virtualization platform. This method can obtain better isolation capability and driver compatibility. The disadvantage is that Nvidia charges a separate license fee for the virtual machine, which is relatively expensive.
-05-
Cloud Native Rendering Practice
The solution introduced above only focuses on the underlying graphics rendering capabilities supported by the operating system level, and does not penetrate enough into specific business scenarios. From the perspective of actual business scenarios, the goal of HUAWEI CLOUD Rendering is to help customers migrate their graphic rendering scenarios to the cloud. Offline rendering scenarios mainly serve film and television and high-definition design drawings. The derived application-level rendering technologies include AI rendering acceleration and HUAWEI CLOUD native rendering. Offline ray tracing rendering supported by the engine.
The real-time rendering scene serves 3D space, digital twins, real-time design and games, etc. It is mainly divided into two sets of solutions: the existing 3D application directly goes to the cloud without modification and the cloud after cloud-native optimization, including real-time rendering of 3D applications and cloud application And remote rendering, 3D space acceleration.
Next, we will introduce the cloud rendering technology in real-time rendering scenarios. Huawei Cloud provides cloud application products to help existing applications directly migrate to the cloud, mainly including 3D design software that requires high CPU/GPU computing power, which solves the problem of software downloads. Installation is time-consuming and labor-intensive and expensive to sell, such as CAD, CAE, CAM and other design, modeling, and simulation software.
Its principle is to use Windows multi-session technology to build a lightweight isolation technology, so that Windows Server can serve multiple clients at the same time. Its advantage is that it has a certain isolation capability, and because of the multi-session implementation at the virtual machine level, each session can directly access the physical GPU. It is a standard Windows technology and has good interoperability with the internal network of the enterprise. The disadvantage is that GPU computing and video memory are in multi-user preemptive mode and cannot be isolated. And each session needs to pay for Windows RDS License independently.
In order to optimize the license cost problem mentioned above, we combine sandbox and Hook technology to allow Windows to run multiple applications without using multi-session technology and serve multiple access clients. This is also the current mainstream technology of cloud rendering plan. Its advantage is that it does not rely on Windows multi-session technology, which reduces the license cost. The disadvantage is that compared with GPU virtualization, resource isolation capabilities are limited.
The aforementioned hardware GPU virtualization is also a major implementation method, but a remote interaction module needs to be added to manage 3D applications. Its advantage is good isolation and compatibility, but the disadvantage is that Nvidia charges license fees separately for virtual machines.
Next, we will introduce the applicable scenarios of several solutions. Choosing a cloud application solution generally involves strong intranet intercommunication (accounts, permissions) and application compatibility requirements (2D, 3D various types); choosing 3D application real-time rendering generally focuses on 3D rendering programs, such as 3D display based on DirectX development Applications, games, etc.; GPU hardware virtualization/physical pass-through solutions are generally used in conjunction with cloud applications and 3D applications for real-time rendering.
What the above technologies have in common is that 3D applications are directly moved to the cloud for calculation, but can pure cloud rendering realize that the application does not go to the cloud, but only the rendering goes to the cloud?
Therefore, we have implemented a cloud-based solution based on cloud-native optimization, that is, remote rendering. It is suitable for web, PC and mobile 3D applications at the same time. The application actually runs on the client side, but the rendering calculation is offloaded, and the rendering is completed on the cloud through the remote rendering SDK, and the final picture is sent back to the client side through video encoding.
Its advantage is that the cloud only undertakes the rendering function, and the computing power on the terminal side can complete other logical operations. Combined with the actual situation of the client, local, cloud or hybrid rendering can be flexibly selected. The disadvantage is that the application needs to be modified based on the remote rendering SDK, and it cannot meet the requirements of very high interactive implementation scenarios (such as some games).
The figure above shows the specific process for users to use the remote rendering service. After the service is activated and a cloud rendering session is created, the remote SDK controls the cloud rendering, and the rendered image is sent back to the client for display through the Huawei RTC service.
Next, demonstrate the effect of remote rendering. In the traditional mode, designers need to obtain renderings through offline rendering before viewing the design effect. Using HUAWEI CLOUD Rendering, designers can view the design effect in real time, and cloud ray tracing rendering can more realistically restore the on-site lighting environment.
So what are the specific advantages of combining rendering and cloud? Compared with consumer-grade graphics cards, the hardware used in cloud data centers is very expensive. What benefits does the high cost actually bring? As shown in the figure above, if you regard the cloud as a large machine, you can find that cloud rendering supports multiple users of different types (mobile phones, tablets, hosts, etc.) at the same time, and can provide an application that can be used by multiple users. Scenes. Its hardware resources are not bound to the user, but determined by the user's usage time. The software also does not need to be installed locally.
The aforementioned cloud solutions for 3D applications are essentially just deploying local programs to the cloud. The cloud only provides a substitute for local resources. Computing resources are scheduled according to connections, and there is no sharing among different applications.
HUAWEI CLOUD has realized some new attempts by transforming the self-developed rendering engine for multi-user access, so that a single engine can access multiple users at the same time in the same scene. This method saves the memory and video memory loaded in some scenes in disguise. And the calculation shared between users can save CPU and GPU. As a cloud-native solution that gives full play to the advantages of the cloud, it is fundamentally different from the aforementioned solutions. After actual testing, it can increase the efficiency by more than 0.5 times.
-06-
Industrial Graphics Rendering Practice
Next, we will introduce some of our practices in the field of industrial graphics rendering. The industrial graphics rendering engine may be different from the rendering work that everyone contacts in daily life. It is a computer-aided design (CAD), computer-aided manufacturing (CAM), computer-aided engineering (CAE), building information modeling (BIM) and other industrial design software. The basic technical components determine the real presentation of 3D design objects and ensure the correctness of the front-end design.
At present, there is a lack of self-developed high-quality industrial graphics rendering engines in China, and the self-developed engines of major companies cannot support high-quality and large-scene graphics rendering. The relatively well-known GPU renderers in the industry are basically foreign products (such as KeyShot and HOOP, etc.).
With the support of the National Technology Innovation Center, HUAWEI CLOUD launched an industrial graphics rendering engine research and development project. The goal is to produce a rendering engine that reaches the world's advanced level after 2 to 3 years of development, and meet the needs of CAD, CAE, CAM and EDA scenarios. Graphics rendering needs.
The picture above shows the recent real-time rendering effect of our self-developed industrial rendering pipeline. At present, it has completed the integration and docking with some Chinese 3D CAD manufacturers. In 2024, it will gradually realize the docking with other enterprise design software.
Next, we will demonstrate the real-time rendering of cloud ray tracing, which focuses on the presentation of materials, light and shadow effects.
-07-
AI Accelerated Rendering Practice
Finally, we will introduce HUAWEI CLOUD's AI-accelerated offline rendering. Our original old idea was to replace similar products from competitors through self-developed rendering engines. Due to the high ecological barriers, this idea was not feasible.
The existing offline rendering process is that the client application submits graphics data through the farm service and the farm fetcher, determines the computing node and completes the graphics rendering. Due to the high consumption of ray tracing calculations in the process, the overall time required is also very long. Therefore, we hope to accelerate the rendering by introducing AI technology, reduce the light-tracing calculation of rendering by adjusting the parameters of the farm service, thereby reducing the calculation of the pure CG side, and then use the proprietary AI model to perform AI inference calculation on the output CG data of the rendering process. The output effect can reach the same quality.
The figure above shows the output effect of HUAWEI CLOUD AI accelerated rendering. It can be seen that the quality of the AI output is basically the same as that of the native rendering.
The picture above shows an actual case of AI-accelerated rendering of film and television dramas. It can be seen that details such as hair are basically the same as the original rendering, and the higher the quality requirements, the better the speed-up effect.
A summary of our technical timeline follows. Since 2011, our development has progressed from 2D rendering to self-developed 3D space cloud rendering, from GPU virtualization to ray tracing engine and AI rendering acceleration. The industries involved include corporate office, digital human, film and television, cloud games, industrial design, etc. .
Our R&D team has a global layout, relying on world-renowned universities and industrial bases in the graphics field, and continues to cultivate cloud-native graphics technology to support industrial upgrading.
Next, we will demonstrate our self-developed offline rendering video, which is mainly for home decoration design scenes. The reality of light and shadow is stronger than real-time rendering, and each frame rendering only takes 5 to 10 seconds.
Finally, we will give a video demonstration of our self-developed real-time rendering, which is based on Huawei's self-developed rendering engine.
I think that rendering is both a technology and a tool, which can help users achieve the desired effect. I also hope to have in-depth exchanges with all the experts present here. That's all for my sharing, thank you all!
▲Scan the QR code in the picture or click " Read the original text " ▲
Direct access to LiveVideoStackCon 2023 Shenzhen Station 20% off ticket purchase channel