This article is reproduced from OpenHarmony TSC official WeChat public account " Summit Review Issue 10 | Use and Implementation of OpenHarmony Graphics Drivers on OpenHarmony "
Speaker | Huang Ran
Review and arrangement | Liao Tao
Typesetting proofreading | Li Pingping
Guest profile
Huang Ran, senior graphics technology expert of Huawei's terminal BG software department, founder of Huawei's terminal game standards, tools and analysis, core member of GPU Turbo black technology, and leader of the open source graphics driver SIG, game SIG, and compatibility working group in the OpenHarmony community and other duties.
content source
The First Open Atom Open Source Foundation OpenHarmony Technology Summit - OS Kernel and Windows Sub-Forum
video review
video link
Contents
Graphics driver is also a software program, which connects the operating system and application programs in series to communicate and interact with computer graphics hardware, and is a key link to provide high-quality graphics display for the operating system by utilizing hardware performance. What work has OpenHarmony done on the use and implementation of open source graphics drivers? Huang Ran, leader of the OpenHarmony game SIG group and graphics driver SIG group, and senior technical expert of Huawei terminal graphics, shared some points with you at the first OpenHarmony Technology Summit.
01 ► Challenges faced by OpenHarmony graphics driver
The evolution of graphics driver technology is always related to the development of GPU hardware. Since 1975, with the development of GPU hardware from early high-end graphics workstations in the professional field to desktop GPU graphics cards, and then to today’s mobile terminals, cloud and server GPU graphics cards, the graphics driver API has also evolved from OpenGL to DirectX.
At present, mainstream manufacturers in the field of graphics drivers have closed source of their own core codes, and open source graphics drivers such as Arm Mali, Qualcomm Adreno, and Nvidia are not particularly "open".
With the rise and success of the open source movement, companies such as AMD and Intel have established a good ecosystem for open source graphics drivers, and achieved good results. For a completely open source operating system like OpenHarmony, the graphics open source driver has good reference and learning significance, but of course there are many challenges. How difficult is it to master the open source graphics driver? First of all, the development and research of graphics drivers requires solid software and hardware development skills, and because the development of open source graphics drivers in China is very slow, few developers specialize in this work, and there is a lack of technical exchanges and practical experience sharing. The picture below shows the notes made by Mr. Huang Ran in the early study and research in the field of open source drive:
In addition, for OpenHarmony, most of the current small manufacturers cannot obtain the support of closed-source GPU manufacturers, resulting in a poor visual smooth experience, which limits the commercial use of many OpenHarmony products, and also hinders the promotion of the OpenHarmony ecosystem to a certain extent.
02 ►Introduction to Open Source Graphics Driver Architecture
From the perspective of drivers, the kernel of OpenHarmony rich devices is based on Linux, so first introduce the overall architecture of Linux open source drivers. The entire driver architecture can be divided into two parts: 2D and 3D. The older framework of the 2D part is based on X11, while the newer framework is based on Wayland.
Part of the 3D driver converts OpenGLES or Vulkan API and shader into hardware ISA through mesa. The 2D DDX driver can also go to the mesa layer through glamor, which avoids the bifurcated driving route of 2D and 3D (it used to be bifurcated, and 2D went to DDX).
The overall driver is UMS+KMS structure, UMS is responsible for the analysis of the user layer driver, and KMS is used for display and hardware rendering, and the transmission from UMS to KMS is formed through libdrm and DRM.
There are several key concepts in the graphics driver:
One is LLVM, TGSI and Gallium. TGSI is an intermediate language used to describe shaders. It is the only intermediate representation used by all drivers, and all Shaders will be converted into intermediate IR. Gallium is the backend of LLVM, which can perform ISA rendering of different hardware based on different hardware. The radeonsi in the figure is the backend rendering of AMD's radeon.
The second is ISA. ISA consists of control flow (CF) instructions, ALU instructions, instructions fetched through texture cache, and instructions fetched through vertex cache, where the control flow program directs program subroutines by using control flow instructions (conditional jumps, loops, and subroutines) The stream of statements, including memory allocation instructions and other instructions, which can specify when the vertex and geometry programs complete related operations, similar to the assembly language of the CPU.
The third is Fence. Fence can make the GPU and CPU work together to improve the speed of image display. The GPU events generated by the Fence mechanism can ensure that the rendering commands issued by the user-mode program are executed sequentially, thereby ensuring the consistency of the upper-layer application rendering-related data.
03 ►Transplantation of open source graphics driver on OpenHarmony
The OpenHarmony driver framework supports multiple access modes, enabling rapid deployment of southbound hardware. Among them, the display framework supports Display_Gralloc, Display_Gfx, and three types of southbound interfaces of Device HDI. Among them, Display_Gralloc is responsible for memory allocation; Display_Gfx is responsible for graphics hardware 2D drawing, which can be used for offline synthesis; Device HDI is responsible for display device feature management, including screen display, Online and offline hardware synthesis, hardware Vsync, display device color management, etc. In terms of development board capability support, RK3568 and HI3516dv300 support DRM memory allocation, DRM display delivery, and hardware offline synthesis. HI3751V350 supports FbDev and DmaBuf-Heap, supports FbDev display, and does not support hardware offline synthesis.
In view of the overall situation of the above-mentioned OpenHarmony driver framework, the adaptation work of the open source GPU driver is mainly divided into the following three stages: (1) Verify that the kernel panfrost driver and the user mode panfrost driver can work normally; (2) Adapt the open source GPU driver to OpenHarmony (Flutter+weston) old framework; (3) The open source GPU driver is adapted to the new OpenHarmony (RenderService) framework. At present, more and more interested developers have participated in the adaptation and porting of OpenHarmony's open source graphics driver. Recently, some users have successfully ported Qualcomm's open source driver to mobile terminals, enabling them to run some 2D and 3D graphics. application. This means that the ecology of open source drivers on OpenHarmony is developing in a good direction.
Judging from the GLmark2 running score, the OpenHarmony open-source driver performs better than the closed-source driver in 2D texture processing, but it performs poorly in key 3D aspects such as coloring, shadows, and terrain. Even so, performance improvements in 2D and 3D open source graphics drivers are sufficient to meet the needs of most products.
Of course, in this process, some partners have referred to the current work and completed the porting of Qualcomm’s freedreno open source driver, and can run and use the open source driver on Xiaomi and other mobile phones, as follows:
In the future, we will also enable open source drivers on X86-based AMD and Intel GPUs to serve OpenHamrony. We also hope that more friends can join the community WeChat group SIG-OpenGfxDrv to build graphics drivers. The corresponding gitee link is: https https://gitee.com/openharmony/third_party_mesa3d
04 ►Summary & Outlook
If you really want to be competitive in graphics, you must understand the working mechanism of the GPU and the principles of graphics drivers. The OpenHarmony community is a good platform for communication and learning; the OpenHarmony open source graphics driver is the future trend and will be the final choice in history. More and more interested developers can participate in the adaptation and transplantation of open source graphics drivers to jointly build the OpenHarmony ecosystem.
Click to follow to learn more about OpenHarmony TSC technical content