Table of contents
1 Concepts, features, technologies, devices, platforms, applications and tools of XR
1.1 Concept, Features and Technology of VR
1.2 Concept, Features and Technology of AR
1.2 XR Devices, Platforms, Applications and Tools
4 Paths to becoming an XR builder
4.1 Three ways to develop XR works
I once mentioned the application of ML in ARCore in [Practical Skills] Google I/O 2022 Conference AI/ML Inspiration for Developers , that is, the application of machine learning on the AR platform.
Today, I will introduce the role of virtual reality technology XR in Web3, the concept, characteristics, technology, equipment, platform, application and tools of XR, the method of choosing XR technology, the 4 steps to develop XR works, and the path to become an XR developer. Hope to help novices become good XR creators in the Web3 era.
0 The role of XR in Web3
First briefly explain the related concepts of XR.
VR = virtual reality (computer-generated virtual world)
AR = Augmented Reality (enhancing the physical world)
MR = mixed reality (includes AR+AV, usually not VR)
XR = Extended Reality (X is a wildcard for A, V or M)
I believe everyone has some understanding of the popular concept Web3
Web3 = Read + Write+ Own
XR belongs to the human-computer interaction layer in Web3 technology. By building a virtual world, it can bring users a richer experience and more application scenarios.
1 Concepts, features, technologies, devices, platforms, applications and tools of XR
1.1 Concept, Features and Technology of VR
VR Virtual Reality virtual reality: virtual and real independent, with immersive 3D interface, refers to the support of computer hardware and software and various sensors (such as high-performance computers, graphics and image generation systems, as well as special clothing, special gloves, special glasses, etc.) Generate a realistic, three-dimensional environment with certain perception capabilities such as sight, hearing, touch, and smell, so that users can interact with the "virtual" environment generated by computers in a simple and natural way with the support of these hardware and software devices. interact with objects in the world.
Features are:
- Virtual environments that can be explored in some 3D stereoscopic views
- Immersive Tasks for Conservation Spatial Interaction and Sound
- Have a believable experience that doesn't require photorealistic effects
Concepts include: Autonomy/Agency, Presence and Immersion, Embodiment and Avatar, Cognition and Empathy, Proprioception and Motion Sickness, Tactile and Pseudotactile, Boundary and Redirected Walking, and more.
Technologies include: 360 photos and videos, stereoscopic display, cave CAVE (a virtual reality space), 3 DOF and 6 DOF (Degrees of freedom) tracking, outside-in and inside-out tracking, hand tracking and spatial audio, etc. .
Advanced VR techniques include procedural generation, redirected walking, and custom controllers.
1.2 Concept, Features and Technology of AR
AR Augmented Reality Augmented Reality: Combination of virtual reality and fusion of real and virtual content. With the help of photoelectric display technology, interactive technology, computer graphics technology and visualization technology, virtual objects that do not exist in the real environment are generated, and the virtual objects are accurately "placed" in the real environment through sensing technology. Integrating with the real environment, presenting a new environment with real sensory effects to the user, so that the user is convinced from the sensory effect that the virtual environment is an organic part of the real environment around them.
Features are:
- is a fusion environment where you can view virtual objects blended with the real world
- Information tasks that enable virtual objects to encode real-world information
- With the illusion of holograms, virtual objects appear as if they are part of the real world
Concepts include: augmentation vs. mediation, strong vs. weak AR, marker-based vs. unmarked AR,
SLAM (Simultaneous Localization And Mapping) and VIO (Visual Inertial Odometry) simultaneous positioning and mapping and visual inertial odometry, tracking and registration, field of view, plane detection and object recognition.
Techniques include: landmark tracking, motion tracking, body tracking, spatial mapping, scene understanding, projection mapping, and ray estimation.
Advanced AR includes 3D reconstruction, object tracking and custom displays.
1.2 XR Devices, Platforms, Applications and Tools
equipment
Standalone and built-in: Quest, HoloLens, …
Tethered/Adapters are: Cardboard, Rift, VIVE, ...
platform
Proprietary ones are: Oculus, VIVE, Magic Leap, …
Cross-platform ones: SteamVR, WMR, WebXR, …
application
XR apps: Beat Saber, Snapchat, …
Apps with XR perspective/mode: IKEA, Amazon Shopping, ...
tool
Design tools: Tilt Brush, Quill, Aero, …etc
Development tools include: A-Frame, Unity, Unreal, …etc.
Figure 1 AR platform and equipment
Image credit: Dr. Michael Nebeling
Figure 2 VR platforms and devices
Image credit: Dr. Michael Nebeling
2 How to choose XR
2.1 When to use VR
VR is ideal if the application scenario includes the following characteristics
- lots of visual elements
- 3D space interaction
- physical manipulation
- Interaction with unlimited resolution
VR is not an ideal choice if the application scenario includes the following characteristics
- many non-visual elements
- Need to read or type a lot of text
- need tactile feedback
- Requires a connection to the real world
2.2 When to use AR
VR is ideal if the application scenario includes the following characteristics
- some visual elements
- 3D space interaction
- physical manipulation
- 1:1 or smaller ratio interaction
If the application scenario includes the following characteristics, AR is not an ideal choice
- many non-visual elements
- Lots of text to read or type
- need tactile feedback
- no connection to the real world
3 4 steps to develop XR works
Developing XR works is similar to developing other software works. The characteristic of XR is to develop and test 3D.
- Find needs and brainstorm: use scenarios and use cases, characters and competitors to determine the framework of the problem, and use a story map to determine the goals and tasks that the work wants to support.
- Storyboarding and Prototyping: Using physical (paper) and digital tools to create mock-ups, including XR devices, is a critical step in making good XR work. This process involves designing user experience based on content (environment and 3D characters), context (environment and 3D characters) and story (3D characters and interactions).
- Develop and test 3D: Create applications using WebXR, Unity/Unreal or native SDKs, and conduct user testing.
- Deployment and Analysis: Deploy applications on XR devices, collect data during use, and verify functionality.
4 Paths to becoming an XR builder
For a developer with software development experience, how does one become an XR builder?
4.1 Three ways to develop XR works
Figure 3 The path to becoming an XR builder
Image credit: Dr. Michael Nebeling
As shown in Figure 3, there are currently three main approaches to XR development: web-based development using WebXR, cross-platform development using Unity and Unreal, and native development using platform-specific SDKs such as Oculus, VIVE, ARKit, or ARCore .
For developers with different backgrounds, the learning curve is somewhat different. It seems that developers with game development background and mobile development background have more advantages. :)
Web-based: Create AR/VR applications based on web technology based on the WebXR standard. Developers can create applications for a variety of AR and VR devices using common web languages, HTML, CSS, and JavaScript, as well as new libraries created specifically for developing AR/VR display and tracking technologies. This is similar to using web technologies to build mobile applications that run on Android and iOS devices.
Cross-platform: Use one development kit to develop for multiple AR/VR devices with a common set of methods and tools. Unity and Unreal have grown into sophisticated AR/VR development toolkits that are widely used. Among them, Unity has a lot of built-in support for many AR/VR display and tracking technologies. To develop interactivity, Unity requires C# or JavaScript, while Unreal requires C++ or its visual scripting language Blueprint.
Native development: That is, developers use a specific software development kit (SDK) to develop specific AR or VR devices. Developers who are familiar with related programming languages and platforms, such as C# (WMR) or C++ (Oculus/VIVE) for Windows, Java (ARCore) for Android, or Swift (ARKit) for iOS, can easily develop works. Usually, if we need to develop the functions required by a specific AR/VR application and are only available on a single platform, we can prefer this method of native development.
4.2 Conversion from 2D to 3D
To complete the transition from 2D to 3D, developers need to understand the following.
1 2D layout and 3D geometry
- Transformation = position, rotation, scale
- Left-handed and right-handed coordinate systems (Unity and A-Frame)
2 2D design vs. 3D materials
- Material is an enhanced version of texture map
- There is still a lot to learn about materials, for example, physically based rendering (PBR).
3 3D models and 360 photos
- Developers don't need to be a 3D artist, there are many ways to obtain 3D models. For example, you can find 3D models on the following websites Sketchfab - The best 3D viewer on the web , 3D Warehouse , Clara.io: Online 3D Modeling, 3D Rendering, Free 3D Models , Mixamo , you can use Blender , Tinkercad , Maya or Maya LT , 3ds Max , Cinema4D builds 3D models.
- 360 photos are great placeholders for 3D models of environments, built using tools like Google's Tour Creator , Microsoft's Sketch 360, and more.
Regarding the specific details of other XR development, such as the content and interaction of developing VR applications and the difference between developing marker-based AR and five-marker AR, I will have the opportunity to communicate with you in the future.
Of course, to do a good job in XR works, mastering skilled technology is one aspect, and on the other hand, it is necessary to think and design user experience and interaction design from the perspective of people, tasks and technology.
thank you
Thanks to Dr.Michael Nebeling for the English pictures
other references
- The Ministry of Industry and Information Technology, the Ministry of Education, the Ministry of Culture and Tourism, the State Administration of Radio and Television, and the State General Administration of Sports issued the "Action Plan for the Integration and Development of Virtual Reality and Industry Applications (2022-2026)" (link: http://www.gov .cn/zhengce/zhengceku/2022-11/01/5723273/files/23f1b69dcf8b4923a20bd6743022a56f.pdf
I hope that the above content can help developers find a path to become an XR creator, and I look forward to your excellent works as soon as possible.