Copyright statement: Davidwang's original article is strictly prohibited from being used in any commercial way and may be reproduced only with authorization.
(1) MR concept
MR is the abbreviation of Mixed Reality, which is a technology that integrates the real and virtual worlds. The concept of mixed reality was proposed by Microsoft, which emphasizes the coexistence and real-time interaction of physical entities and digital objects, such as virtual and real occlusion, environmental reflection, etc. In comparison, AR emphasizes the enhancement of the real world, while MR emphasizes the integration of reality and reality, and pays more attention to the interaction between the virtual digital world and the real world, such as environmental occlusion, humanoid occlusion, scene depth, physical simulation, etc. It pays more attention to operating virtual objects in a natural and instinctive way. The effect of MR virtual and real fusion is shown in Figure 1.
In a mixed reality environment, the real-world environment is "augmented" by computer-generated virtual information such as text, images, videos, 3D models, animations, etc., and can even span multiple sensory modes such as vision, hearing, touch, somatosensory and smell. The superimposed virtual information can be constructive (that is, an addition to the real environment) or destructive (that is, a mask of the real environment), and is seamlessly intertwined with the real world, making people feel immersed in it. The sensory experience is difficult to distinguish between true and false, and it is difficult to distinguish between reality and reality. In this way, mixed reality can change the user's continuous perception of the real-world environment, which is completely different from virtual reality that isolates virtual reality and completely replaces the user's real-world environment with a virtual environment.
The primary value of mixed reality is that it brings digital information into an individual’s perception of the real world, rather than a simple display of data, augmenting reality through immersive integration with what is considered a natural part of the environment. With the excellent capabilities of the HoloLens device (such as instinctive gesture operations, voice commands, and eye gaze interaction), the hybrid world around the user becomes interactive and operable. In short, MR is to display virtual information in reality and allow users to interact with virtual information. MR seamlessly connects reality and virtual information through environmental tracking, understanding and other technical means, which will not exist in reality. Things are constructed and displayed, connected and integrated in the same three-dimensional scene that is consistent with the real environment.
The development of mixed reality technology will change the way we observe the world. The world will no longer be a collection of surface phenomena that we see, but can have a more profound and personalized connotation, thus triggering a change in the way humans understand the world. Imagine that the user is walking or driving on the road. Through the augmented reality display (AR glasses or fully transparent windshield display), information-based images will appear in the user's field of vision (such as road signs, navigation, prompts), and these enhanced information will be updated in real time. , and the sound played is synchronized with the scene the user sees; or when we see a mushroom, we can immediately learn its composition and toxicity through AR glasses; or when we need help at any time, digital artificial intelligence The intelligent humanoid assistant immediately appears in front of us and serves us throughout the process with an image that is indistinguishable from a real person.
Not only that, the development of mixed reality technology is in line with the trend of more direct, intuitive, and instinctive human-computer interaction, and will surely create a new human-computer interaction model and connect the virtual and real world in a more natural way.
(2) Hologram
The hologram mentioned in this series refers to the light and shadow image and 3D sound effects complex generated by the HoloLens device. It generally refers to all virtual elements, videos, objects, objects and their 3D space sound effects. These holographic scenes can be virtual objects fixed in the real world space. , or it can be a UX element that follows the user, is holographically superimposed on the real world, and interacts with the real world. Holograms generally refer to the collection of all visible virtual objects superimposed on the real environment. Holograms are also called holographic images.
Since the HoloLens device uses optical waveguide display technology, the hologram generated by the device can only be overlaid on the environment in a superimposed manner. That is, we can see both the hologram and the real environment behind it. The HoloLens device cannot store environmental information. removed, and we cannot generate a pure black color, so the hologram cannot render black (black will be rendered fully transparent). The brighter the holographic image, the clearer it will be displayed in the HoloLens device.
Thanks to the excellent motion tracking capabilities of the HoloLens device, holograms can be placed anywhere in the user's real space. These holographic images placed in the real space will be fixed in the environment like real objects, even if the user moves. , they will also remain in place. Of course, we can also set the hologram to always stay within the field of view or follow the user. In MR applications, keeping the hologram within the field of view is also called display lock. In this mode, Holograms, like UI elements in ordinary applications, always occupy part of the display area. Usually this form of hologram is used to display fixed information such as power and time. However, it should be noted that this display method is different from MR applications. The created 3D mixed reality scene is inconsistent and will cause discomfort. It is recommended not to use this method unless there are special circumstances; keeping the hologram following the user is also called body lock, and the hologram locked by the body will follow users, but they are also in the 3D space of MR applications. A typical example is the performance diagnostic panel (Diagnostics panel) common in MR development. Well-designed delay and elastic easing effects can make this mode very suitable for commonly used menus. , tool display.
Holograms are not just the superposition of light, shadow and sound, they are also part of mixed reality and can interact with the real environment and users. The HoloLens device has the ability to perceive the environment and can reconstruct the surface of the physical scene. The hologram can correctly collide with and block the real environment. , reflection, physical simulation, such as throwing a virtual ball to the ground, the ball will bounce on the real floor, and will also roll under the table and disappear. When working with holograms, aligning them with gravity (Y-axis) is the simplest and most effective way to increase the realism of your holograms.
(3) HoloLens2 device
The HoloLens2 device evolved from the HoloLens1 generation and is a wearable all-in-one holographic computing device. As shown in Figure 2, it has the best optical waveguide display component in the industry and has an independent computing unit that can perform real-time gestures, voice , sensing environment, motion tracking, eye tracking and other solutions. The HoloLens2 device is a complete holographic mixed reality device that can complete all mixed reality calculations and displays without relying on any external software or hardware. The HoloLens2 device is also a mixed reality platform, running Windows 10 Holographic System (Windows 10 Holographic System), and has all UWP (Universal Windows Platform, Universal Windows Platform) feature support.
As a complete mixed reality device, HoloLens2 is equipped with numerous sensors and computing units, providing an excellent immersive experience. Its basic skill parameters are shown in Figure 3.
- Computing performance
The HoloLens2 device is equipped with the Qualcomm Snapdragon 850 computing platform, which is a high-performance and low-power processing platform with a clock frequency of 2.96G, 64-bit, and 10 nanometers. The powerful computing performance ensures the demanding latency requirements of MR applications. It lays the computing foundation for the entire device.
The HoloLens2 device is also equipped with a dedicated HPU (Holographic Processing Unit), which has a total of 13 computing units and is mainly responsible for 6DoF motion tracking, eye tracking, gesture tracking, 3D sound effects, environment perception and spatial mapping, Computational processing such as scene understanding and holographic stability. The emergence of HPU dedicated processors greatly reduces the computing pressure on the CPU and provides better accuracy and power consumption performance.
-
Optical waveguide display
The field of view of the HoloLens2 optical waveguide display module has been increased to 52 degrees, which provides a larger viewing space while maintaining a resolution of 47 pixels per degree, and the image quality is sharper. The HoloLens2 waveguide lens has been reduced from 3 layers to 2 layers, making it more transparent to ambient light and improving the MR experience. -
Depth sensing
HoloLens2 device is equipped with a TOF (Time of Flight, Time of Flight) depth sensor, which provides the ability to directly obtain scene depth information, so it can efficiently perceive the external environment, and cooperates with AI algorithms to improve the integrity of the scene surface geometric grid. -
Environment Understanding
The second-generation HPU has a built-in neural network. After obtaining the scene surface geometric grid information through TOF, it intelligently identifies scene objects, such as floors, ceilings, desktops, etc., through the constructed depth map and motion tracking. The scene understanding ability is very important. Improving the intelligence of MR applications has a crucial impact. -
Gesture recognition
HoloLens2 device has very excellent gesture tracking and recognition capabilities, which is mainly due to the HPU neural network. In order to reduce power consumption, the HoloLens2 device keeps a coarse-grained neural network running at all times. When a gesture is detected, it will start another high-precision neural network to track and recognize the gesture, which not only reduces power consumption but does not lose accuracy. Due to the powerful capabilities of the second-generation HPU, the HoloLens2 device completely gets rid of the dilemma of only two gestures of the first-generation device, allowing users to interact with virtual and real instincts in a more natural way. -
Eye Tracking
The HoloLens2 device has two inward infrared image sensors responsible for iris recognition and eye tracking. These two sensors can not only automatically measure the distance between the interpupillary eyes of the eyes, but also track the gaze direction of the eyes in real time, which provides a better solution for long-distance gaze. It lays the foundation for interaction and also provides a third virtual and real interaction method in addition to gestures and voice. -
Microphone and spatial sound effects
The HoloLens2 device is equipped with 3 forward-facing environmental sound collection microphones and 2 user voice collection microphones. Through this 5-channel microphone array, the HoloLens2 device can correctly recognize the user's voice in an environment of 90 decibels. The HoloLens2 device is also equipped with a spatial sound effect micro-sound system, which can correctly output spatial 3D sound effects.
references
1. What is a hologram? 2.
About HoloLens 2. About HoloLens 2