Metaverse is a virtual world parallel to the real world, and it is a new generation of Internet. XR with true three-dimensional, interactive, and immersive features is the ultimate digital media form for building the metaverse. How to break the tight coupling between XR terminal devices and XR content, and realize online access on any platform and any terminal, Cloud XR is the only way to open the channel to the Metaverse. Parallel Cloud is an internationally leading Cloud XR concept advocate and technology pioneer, committed to providing industry partners and developers with low-code, out-of-the-box, and efficient deployment of Cloud XR PaaS platform products. It has attracted nearly a thousand enterprise users and tens of thousands of independent developers around the world. The products and solutions are maturely used in education and training, digital twins, medical rehabilitation, virtual live broadcast, digital human, cloud activities, cloud games and other scenarios.
Text/Li Yan
Edit/LiveVideoStack
Today's content mainly revolves around three key words: Parallel Cloud, Metaverse, and Channel. "Parallel Cloud" is the name of our company. "Metaverse" is the topic of our sub-forum today. "Channel" is a positioning of Parallel Cloud in the metaverse industry.
Next, I will combine these three keywords and give a brief introduction from five aspects: parallel cloud and metaverse, audio and video and metaverse, challenge and exploration, about our company and case presentation.
-01-
Parallel Clouds and the Metaverse
First of all, parallel cloud and metaverse. The three words "Parallel Cloud" are derived from the phrase " virtual world on the cloud parallel to the real world ". From the beginning of the company's establishment to the present, this sentence is an original intention and vision that we have always adhered to and never changed.
Speaking of parallel worlds, I actually think this concept is not new, it should be a pursuit of our human beings since ancient times. Whether we talk about the origin of the Metaverse, Avalanche, or The Classic of Mountains and Seas, ancient Greek mythology, or current film and television themes or even online games, all of them are constructing a virtual world parallel to our world. So what's the difference? The difference is that the form of digital media that constitutes the parallel world has been evolving.
Like the ancient Greek mythology and the Classic of Mountains and Seas just mentioned, before the Internet and digitization, hieroglyphs were composed of pictures. With the Internet, gradually from text to pictures to audio and video. Today is actually a world of audio and video. The Internet is composed of text, pictures, audio and video, but no matter whether it is text, pictures or audio and video, they all have a common feature: "The parallel world they build, the virtual world is two-dimensional, and the information is One-way transmission, we have no way to interact with it, which leads to the fact that the virtual is the virtual and the real is the real, and the virtual and the real are separated. It is actually not that easy to enter a virtual world, just like reading a novel when you were young , we may imagine ourselves as a character in a novel, but it requires a possibly strong imagination".
Metaverse is a new generation of Internet. After the term Metaverse came out, people have different opinions, and it is still in a state of being blind. Practitioners in different industries will definitely describe the metaverse from their own perspectives. It is like saying that those who touch the legs of an elephant are called pillars, and those who touch the ears of an elephant are called cattail fans. Applications, such as social networking, office, games, education, payment, etc. are not fresh enough.
So what's new? What is new is the form of digital media on the Internet. From text and pictures to audio and video, the Internet composed of XR, that is, extended reality, is a new form.
What are the new features of this new form? It is three-dimensional and interactive, and different immersive experiences can be achieved with the help of different terminals, which is exactly the same as our physical world. Because our physical world is three-dimensional, we are in it, and we can interact with it. This kind of equal dimension between virtual and real is a major premise, so that we can bring about the so-called co-existence of virtual and real, and the symbiosis of virtual and real.
In the future, the boundary between the virtual world and the physical world may become very blurred, and the virtual and the real are blended together, and even we can't tell which is virtual and which is real. This is the concept of metaverse. Although at the technical level, we think it is an iteration of such digital media, but why is it so popular at the application level? I think this is a major reason.
To sum up the content mentioned above, the new generation of Internet refers to the metaverse we are talking about now. The metaverse is composed of a 3D and decentralized Internet, a virtual world parallel to our physical world. Let's not discuss the matter of decentralization for the time being. Today we will focus on 3D.
-02-
Audio Video and the Metaverse
What is audio and video made of? It consists of pictures and audio. So what is the XR that makes up the parallel world made of?
Let's take a look at this picture. These are a few screenshots I took of the demo released by UE5.0 last year, and two screenshots in the ancient valley. This is also a very well-known screenshot of Black Wukong in the game industry. These things are actually the same, they are composed of a large number of triangular faces plus textures.
Taking the case of the ancient valley as an example, there are more than 33 million triangular faces under a statue, and the entire demo contains hundreds of billions of triangular faces. These elements form XR, and XR forms the entire parallel virtual world. Of course, the XR mentioned here is in a narrow sense and content-oriented, referring to three-dimensional, interactive content that uses different terminals to achieve immersion.
Then this brings at least three pain points:
1. Large amount of data . Judging from the demo of the ancient valley, the storage space of this demo is 205GB, and the amount of data is large, and this is different from the audio and video that everyone is familiar with. Audio and video can support streaming transmission and can be watched while downloading, but For the 205GB content, if several K dynamic libraries are broken during download, the entire program may not be able to run;
2. High computing power required for operation . We have been saying just now that the computing power required to run these hundreds of millions of triangular faces is very high. From the perspective of allowing these triangular faces to be calculated in real time, it requires a very high computing power. The 4090 card just released last year has changed from the traditional two slots to three slots, which cannot be accommodated in ordinary terminals, only very professional computer communication workstations can fit it;
3. The protection of data assets is very difficult . Assets that are more difficult to produce are more valuable. If the complex three-dimensional data assets we saw just now are downloaded by others, there is no good way to protect them, which is very detrimental to the development of the entire ecology.
Let's summarize the above three pain points to see what we want and what the current status is.
1. We hope that any terminal can be used immediately to reduce the cost of use, but at present, the requirements for this kind of hardware configuration are very high, and the cost of use is also high;
2. We don't want to download so much data, we hope to click a URL link and use it immediately like surfing the Internet, but now we need to download the program plug-in data in advance, and even do complicated configuration;
3. We hope that it only takes a few seconds to open, and there is no need to download the content. But now even if it becomes a WebGL-based BS architecture mode, in fact it still needs to download data in the background, and it takes a long time to open;
4. We hope that single-platform development can be used across platforms, but currently there are various versions such as Windows, Linux, and MAC;
5. We hope that it will be a minimal maintenance and update content upgrade, but the status quo is that the cost of this kind of decentralized deployment maintenance update is very high;
6. In terms of data security, we hope that data and users are separated, and intellectual property rights can be well protected. However, it is difficult to control content authorization, and the protection of intellectual property rights cannot be guaranteed.
These goals and pain points must be the problems we have to solve, otherwise the ecology will not be able to move forward benignly.
When we first started doing this, there were still a lot of disputes over various routes between the cloud and the cloud. Fortunately, the entire industry has recognized this saying, which is brought about by the cloud. Endless possibilities.
Cloud XR is the only way for large-scale application of XR. We hope that everyone can enter the metaverse, and we must lower the threshold.
Why Cloud XR can solve these three problems? The above is the flow chart. I have always opposed calling Cloud XR streaming for short, also known as streaming. Streaming refers to point-to-point, which is the process of solving intermediate network transmission, but this process is far more than network transmission.
The terminal sends an interactive command to the cloud. The first step the cloud needs to do is to distribute the computing power. The next step is to calculate and render, and collect the results. A rendering of the video is displayed.
In this process, the terminal only does two things, the first is the upload of instructions, and the second is the decoding and rendering of audio and video. As long as you have a terminal that can watch high-definition videos online, you can access the original requirements. XR applications that are only accessible on very high-performance graphics workstations. At the same time, the data does not fall to the ground. The user can see but cannot touch the terminal through the streaming video stream. Even if the video on the screen is intercepted, no data can be taken away, which can solve the problems listed above. Pain points.
-03-
challenge and exploration
Parallel Cloud only does one thing, which is to open the channel to the Metaverse.
A channel must be something that connects two ends. One end is a three-dimensional, interactive, immersive virtual world parallel to the physical world formed by XR. Partners from all walks of life are building this world. The other end is more and more portable terminal equipment, whether it is the most used mobile phone, iPad, computer, or VR, AR, MR equipment that may become more and more popular in the future. There was no channel before the formation of the channel. It is necessary to download and install the content on the device to become a tightly coupled state, and it does not need a channel.
Just like the audio and video was recorded on video tape before, it is completed by using a DVD player to play CDs and VCDs. Now it can be decoupled through the network. If XR wants to achieve a large-scale application, it must also achieve such decoupling.
In fact, Parallel Cloud focuses more on products at the software layer. It does not do computing power at the bottom, does not do applications at the top, and only does the middle layer. This is the biggest difference from the previous ones.
I list the following points:
1. The challenge of audio and video transmission optimization; 2. The challenge of cloud computing resource allocation
3. The challenge of heterogeneous terminal access; 4. The challenge of out-of-the-box and secondary development
Source: Huawei Wireless X Labs White Paper
Cloud XR has very high requirements for the network, mainly due to the following aspects: a large amount of data, very high real-time requirements, and very strict QoS guarantee requirements.
First of all, let’s talk about the super large bit rate. According to the white paper listed in the year, from the start to the comfortable state and then to the ideal state, the typical bandwidth requirements need to reach 80M, 260M, or even 1G to 1.5G. This is completely different from our current audio and video ITC application bit rate.
Ultra-low latency needs to make the end-to-end latency less than 70 at least in the initial stage, less than 50 in the comfortable stage, and less than 30 in the ideal stage. At the same time, an asynchronous and time-warped calculation is required at the end, which is the so-called ATW. The calculation of ATW will be controlled within 20 milliseconds, and the calculation effect depends on the previous end-to-end experiments, so this is definitely achievable.
Ultra-high reliability, if it is unreliable, it will cause blurred screen and freeze.
The data listed above can actually be said to have little to do with us. We are making products at the software level. These network layers belong to infrastructure, and we can only do what we can do on the basis of the existing network foundation.
In fact, everyone in this industry is doing research in this area. I paid special attention to the situation of LiveVideoStackCon 2021 Beijing Station. Tencent experts shared the "cloud rendering technology framework and new interactive gameplay". In fact, everyone is the same, from coding and transmission These two angles to do some optimization.
Doing this in the industry should be said to be similar with minor differences. I won’t talk much about Datong. Let’s briefly talk about the differences. The following are some of our own explorations.
The current mainstream approach is based on network grouping and decision-making control based on data such as delay and packet loss rate. This kind of decision-making control is currently the most used, but it fully considers the real-time fluctuations of the network, so it is defined as a decision-making control framework from a micro perspective. The advantage of this control framework is that it can have real-time feedback on network changes. In the case of good network conditions, the effect is good, but in weak network conditions, the decision-making control is often lagging behind, and there is no way to bring a better user experience.
On this basis, we propose a role control framework with both micro and macro considerations. Not only consider the micro and macro situation of the network, but also consider the application layer and user feedback layer. In this way, the whole character can be controlled, making it more flexible and more scientific. The ultimate goal is the same, we aim to bring a better user experience under the existing network conditions, especially under the weak network conditions like WiFi.
2. The challenge of cloud computing resource allocation
One is the GPU graphics accelerated computing resource pooling solution. The second is the computing power resource scheduling of the center plus the edge.
Those who are familiar with GPUs may know that the virtualization of GPUs is indeed not mature enough. The current virtualization solutions either use virtualization or pass-through, that is, hang a vGPU or physical GPU on a virtual machine. The advantage of this approach is that it can be strictly isolated. The disadvantage is that its resource usage and scheduling are not particularly flexible. .
A vGPU or GPU can only be used by one user. As mentioned just now, for a certain game, if its computing power is known, it is completely satisfactory. However, in the case of a diverse content platform, different users access different content, and the demand for such computing power has very large elastic changes. This model is not reasonable.
In our products, Athena is a GPU pooling solution. Everyone in the industry has different approaches to pooling. The more unified ones are process isolation and resource limitations, as well as API-level interception and encapsulation. Since it intercepts the API layer, what it can do must be limited.
In the pooling of graphics acceleration computing mentioned above, we only intercept graphics interfaces such as Direct X Open GL. If you are doing HPC, you may use Cuda more. In this case, there is no virtualization layer, which is similar to the difference between Docker and VM. All upper-layer APPs run directly on the entire OS, and multi-user sharing is realized through the isolation between processes, so as to realize a Ultra-fine-grained, dynamic resource scheduling and allocation.
Although we cannot control the network, we can provide users with better conditions and bring users closer to the computing power. After the computing power is built, there is a complex structure about how to allocate and schedule when it is used. For example, groups, regions, distances, concurrency numbers, resource occupancy of each node, etc. need to be considered. Based on such an allocation algorithm, users can finally get their own best node.
Not long ago, we jointly organized a nationwide event with Tmall. Several nodes were built across the country. At the same time, these nodes were also divided into different groups. There were VIP users and ordinary users in the groups. Ordinary users had no nodes to wait for, and VIP users could Access anytime.
3. Heterogeneous terminal access
Although the ideal state is to hope that there will be more and more wearable devices, it is inevitable that flat devices such as mobile phones are still used the most, and they are also streamed from the cloud, but it is different. These mobile phones, iPads, Computers are directed to a video stream from the cloud by a plane and three-dimensional. For devices like VR, two video streams need to be directed from the cloud, and the left and right eyes are still different.
In addition to these two video streams, AR also needs to be superimposed with the physical world. How to enable heterogeneous applications and terminals to be used uniformly on one platform requires opening up the underlying runtime to realize its access.
4. Out-of-the-box use and secondary development challenges
There are many companies similar to us on the market. Parallel Cloud is the earliest and the only recognized company that has commercialized the capabilities of Could XR. We do not do any customization or export of capabilities.
Enter Parallel Cloud official website, you only need to register to download the out-of-the-box products. However, different industries have different needs. How to meet these different needs requires us to provide a large number of flexible secondary development capabilities, so that everyone can combine the needs of their own industries with the needs of their own industries on the basis of standardized products. Know-how, do secondary development and use by yourself.
5. About us
Briefly, Parallel Cloud is the first company in China to commercialize real-time cloud rendering solutions. LarkXR is now the most widely used general-purpose solution in the industry. Some leading companies in various applications in the industry are doing industry development based on our products.
At present, we have attracted tens of thousands of developers and hundreds of enterprise-level industry partners. In the industry, giants such as NVIDIA, AWS, and Alibaba Cloud are our only partners in this field.
At present, Parallel Cloud has started the process of globalization. There are more and more overseas developers, and we didn’t do deliberate expansion before. We found that more and more people rely on natural traffic, so we plan to work hard to do some overseas business from this year.
This picture is the user authorization data captured last month. It can be seen from the picture that among the seven continents in the world, except Antarctica, there are no visitors, and other continents have visitors. This is why we are working hard to promote our products overseas. Firstly, overseas customers have demand, and secondly, our products are competitive overseas.
6. Case display
The Tmall Shopping Festival is to enter the whole set of things through the Taobao app on mobile phones. The traffic control is very strict. At the peak time, it needs to be 20,000 to 30,000, and at the trough, there may be only a few hundred concurrency. It is very flexible and flexible, as long as you search all the keywords through Taobao on your mobile phone, you can enter immediately.
Use this case to briefly share the parallel cloud products.
The main function of Parallel Cloud is the lowest-level cloud rendering capability, and then gradually make some functional components suitable for various industries. Further up, it is a cloud-native development framework. The cloud-native framework is a combination of the device and the cloud. As mentioned earlier, some business processes need to be operated on the mobile phone Taobao, and the pictures, audio and video involved, and even some tactile rendering are all realized on the web. The rendering of these high-precision models is mainly placed on the cloud, and the two are connected based on the cloud-native framework.
What I talked about before is the application of the metaverse of pan-consumer entertainment level, which is an application in the digital twin field of ToB. This partner basically accounts for 70% to 80% of the domestic airport informatization market share.
If all the current digital twin systems want to use it and avoid becoming a product that is only displayed on a large screen, they must get rid of the limitations of the terminal and truly function.
Finally, I hope that with the help of some products and capabilities of Parallel Cloud, we can help you meet yourself in the parallel world as soon as possible!
thank you all!
▲Scan the QR code in the picture or click " Read the original text " ▲
Check out more exciting topics of LiveVideoStackCon 2023 Shanghai Station