1 Introduction
The Yuanverse quickly fell into silence after the fire. The main reason was that some hardware failed to meet the requirements and there was a lack of effective carriers that combined reality and reality. The head-mounted display track cooled down, and the enthusiasm of the giants subsided. The MR head-mounted display product released by Apple in June took Metaverse technology one step further, marking Apple's official entry into the MR market.
Apple believes that “just as Mac brought us into the era of personal computing and iPhone brought us into the era of mobile computing, Apple Vision Pro will bring us into the spatial computing era a>”. The so-called spatial computing initially refers to geospatial positioning and measurement technology. The "spatial computing layer" in the Metaverse has expanded concepts such as 3D engines, VR/AR/MR equipment, and geographic information mapping. Why do you need spatial computing? The metaverse has the characteristics of fusion of virtuality and reality, and there is a connecting point between virtuality and reality, which needs to be completed through space calculation.
At this stage, Apple Vision Pro is launched as the latest powerful head display product. Then on June 22, the software development kit (SDK) of the VisionOS system was released, allowing developers to work together. Build the "spatial computing era". Apple's series of actions deserve everyone's attention. This article will explain Apple Vision Pro and its VisionOS SDK, and introduce its software and hardware architecture,< /span>, and a simple outlook on the integration scenarios with GIS. Application scenarios and Development conditions, Main features
2.Vision Pro
Vision Pro is essentially a VR device. The user cannot see the outside world with the naked eye. It transmits the outside world to the user through more than ten lenses throughout the body , this video transparent transmission calculation is the basis of user experience. Vision Pro's product positioning is an AR device similar to Hololens, except that Hololens uses a lens solution, while Vision Pro uses Transparent transmission scheme. VR headsets generally cannot see the external world, but MR headsets must be able to see the real world. Apple Vision Pro uses a transparent transmission solution to combine virtual reality and augmented reality.
3.VisionOS SDK
3.1 VisionOS Overview
Apple Vision Pro is equipped with the latest VisionOS operating system. Just like iOS for iPhone, iPadOS for iPad and MacOS for Macbook, VisionOS is suitable for Vision Pro and is called " The world's first space operating system". Unlike traditional desktop computing and mobile computing, spatial computing suspends the work environment in front of the eyes, helping users understand proportions and distances through changes in natural light and shadow, making digital content look like it exists in the real world. Users can interact with the application while staying connected to their surroundings, or completely immerse themselves in the created world.
VisionOS is built on macOS, iOS, and iPadOS and includes iOS and spatial frameworks, a multi-app 3D engine, an audio engine, a dedicated renderer subsystem, and a real-time subsystem. At the architectural level, VisionOS shares core modules with MacOS and iOS, including the newly added "real-time subsystem" to handle interactive visual effects on Apple Vision Pro. VisionOS will have a new App Store, and Vision Pro will also be able to run “hundreds of thousands of familiar iPhone and iPad apps.
3.2 Content display format
In the VisionOS SDK, Apple divides its application UI into three basic elements, namely: 2D window (Windows), 3D volume (Volume) and physical space (Space).
(1) Windows:2D windows for creating 2D interfaces based on SwiftUI. These windows contain traditional views and controls and can add 3D content to increase the depth of the experience. . The application interface includes traditional views and control mechanisms, and can also display 3D models. For example, combined with the Model3D view, dynamic 3D content is embedded in the 2D interface. Model3D can be loaded directly in the application and displays 3D models rendered with RealityKit.
(2) Volumes:Volume view is a 3D extension of 2D Windows, which can display 3D models in all directions and allow users to view them from multiple angles. The volume view is designed for the Shared Space interface, so the 3D model displayed has a bounding box. Volume can use RealityKit or Unity to display SwiftUI scenes of 3D content.
(3) Spaces:Entity space, the application starts to Shared Space by default, the application can display content based on window view and block view, and the user can restart it anywhere Position these elements. Shared Space allows users to see their surroundings while using it. For a more immersive experience, apps can also open a dedicated Full Space that displays only the app's content.
3.3 Main features
(1) visionOS is compatible with applications on macOS, iOS and iPadOS
visionOS is fundamentally an extension of iOS and iPad OS, making it easy and painless to integrate with the computers and mobile phones that users are accustomed to. Users can directly use the familiar Apple Vision Pro large-screen display to view mobile phone or computer screens, or use voice or physical keyboards to work, or use mobile phone touch as a mouse to control the screen, without arbitrarily escaping from familiar smart devices. usage habit.
(2) All applications need to exist in 3D space
All visionOS apps "float" in 3D space, even basic 2D apps ported from iOS or iPad OS will float in space. Traditional UI elements get new Z-offset options, which allow developers to push panels and controls into 3D space, making specific interface elements float in front of or behind other elements.
(3) No handle interaction
The actual use of Vision Pro is completely independent of the handle, completely eliminating the need for additional controllers. Vision Pro is equipped with a high-precision retina sensor inside the glasses. The eyes themselves are like extended light beams. Wherever you look, the UI will automatically float up.
(4) Video transparent transmission
Thanks to more than a dozen cameras and the precise tracking of content eye-tracking series sensors, the integration of VR and AR is achieved. By restoring and modeling the internal eye focus, a penetration effect can be achieved when viewing screen content and external content, which integrates the software UI with the real world. By rotating the digital knob, users can adjust the degree of background perspective, allowing the UI to transition naturally between full immersion and full perspective.
(5) Application development relies on existing tools
Developers will use SwiftUI and UIKit to build user interfaces, RealityKit to display 3D content, animations and visual effects, and ARKit to achieve real-world spatial understanding. At the same time, Apple's cooperation with Unity enables all Unity-based content to be migrated to VisionOS-based applications without much conversion work. These allow developers with rich experience in developing old tools to quickly create products, reduce developers' knowledge migration costs, and facilitate the rapid promotion of products.
3.4 Development methods
Apple’s spatial computing platform is built on the frameworks of SwiftUI, RealityKit, and ARKit that developers are familiar with. Apple has expanded these frameworks for VisionOS, adding support for new hardware and full space: 2D windows—SwiftUI, 3D scenes— RealityKit/Unity3D, physical space - ARKit.
SwiftUI is a rapid development kit based on iPadOS and iOS that helps users quickly complete the development of window interfaces and can seamlessly integrate iPhone and iPad applications into SwiftUI. This kit allows an application to display multiple windows and 3D scenes in space, and realize linkage in 2D and 3D; RealityKit is a development engine similar to U3D, using Apple's 3D rendering engine RealityKit to present 3D content and animation in the application and light and shadow visual effects; ARKit provides a way to interact with the surrounding space, allowing Vision Pro to understand the environment around the user, with capabilities such as plane estimation, scene reconstruction, image anchoring, spatial positioning, and head-mounted display motion tracking.
SwiftUI + Xcode + Reality Composer can complete new product development. At the same time, all iOS, iPad App, etc. will be automatically supported on the first day of launch, which alleviates the problem of insufficient early content.
4.Interaction method
VisionOS mainly revolves around direct interaction with eyes, gestures and voice.
Visual interaction: By using eye tracking technology, visual focus navigation is achieved. Whichever button the user looks at, the button enters the alternative state; whichever window the user looks at, the window enters the active state. From the moment the user sees this element, he can already interact with it using gestures, eliminating the concept of a cursor at the GUI level.
Voice interaction: When both visual and tactile interaction channels are occupied, voice interaction can play the role of communicating with and controlling the system. Voice is good at flattening menus. Structure that provides the direct access users need.
Gesture interaction: When the user selects the button with his eyes, visionOS will further determine which event should be triggered based on the user's gestures. For example, when the user pinches the thumb and index finger with one hand, the button will trigger a "click" event, and when the user keeps pinching and moves the hand, the button will trigger a "drag" event. visionOS also supports custom gestures.
When the user selects the button with his eyes, visionOS will further determine which event should be triggered based on the user's gestures. For example, when the user pinches the thumb and index finger with one hand, the button will trigger a "click" event, and when the user keeps pinching and moves the hand, the button will trigger a "drag" event.
VisionOS also supports users to directly interact with virtual objects, such as grabbing a model, moving its position, and changing its size. At this time, the user no longer operates the model in the air, but actually puts his hand near the model. This is of great practical significance for digital twin applications.
5.Usage scenarios
The head-mounted display is relatively large and in principle is not suitable for wearing outdoors and is mainly used for indoor applications. The officially promoted application scenarios mainly include four major scenarios: communication, collaboration, work and entertainment, and each scenario will be reflected in the form of 3D space. Here we mainly give a brief introduction to communication scenarios and collaboration scenarios.
5.1 Communication scenario
Integrating reality and virtuality, communication does not just stay on the two-dimensional screen, but enters the three-dimensional space, fully utilizing the space around you, allowing you to share content and communicate with others in this space; everyone in this space They all interact in the form of windows, allowing multiple people to share videos/movies/chat in one space (regardless of venue); when someone interacts with you in the real world, you can clearly see the surrounding environment, regardless of the location. Will be disconnected from the scene and can maintain communication with them, maintaining interactivity with everything around it.
5.2 Work scenario
VisionOS allows users to share any window just like in macOS. Workers can use Mac smoothly in an ideal workspace, switch between different visual-spatial environments, and use other apps in Vision Pro side by side with Mac; when traveling, they can be in the same space with team colleagues For remote work, each other can process documents, discuss work, and hold meetings in the same space, including quickly viewing content in AR. When people share 3D models, visionOS synchronizes the user's orientation, as well as AR scale, animation, and contextual information, allowing everyone to get the same experience and enable remote visual collaboration.
6. Integration point with GIS
As Apple's first 3D camera, Vision Pro can capture spatial photos and spatial videos, allowing users to record and recreate geographical location scenes and features in a whole new way. It can also analyze three-dimensional spatial data in real time, providing GIS professionals with higher efficiency and accuracy. For example, Vision Pro can quickly measure parameters such as height, area, and volume of buildings on site, or identify terrain features such as slope, direction, and curvature.
Vision Pro supports multiple input methods, including eyes, gestures, voice and accessories, allowing users to flexibly interact and operate with GIS data. Users can communicate and communicate on the same map through voice, gestures or touch. . Vision Pro can be seamlessly integrated with Apple's other devices and services, and can project GIS data into a shared space in a three-dimensional form, making it convenient for users in different spatial locations to observe and operate maps from different angles and distances, to achieve tasks such as urban planning, emergency rescue and disaster relief. Multi-departmental collaboration on command tasks such as anti-terrorism and riot prevention.
7.Conclusion
Apple Vision Pro and its visionOS system have many highlights: First, the interaction method is richer and more convenient, and the support of large FOV + clear VST + super computing power gives a better scene experience. At the same time, compared to Microsoft's Hololens, visionOS integrates AR and VR based on video transparent transmission technology,with stronger immersion and interactivity . Second, Apple Vision Pro can seamlessly connect macOS, iOS, and iPadOS. Relying on the App Store application platform, the development of the software ecological environment is basically guaranteed . Third, the development of VisionOS products relies on existing mature products, which reduces the learning cost of new product development and facilitates rapid development by developers. Fourth, shared space processing occurs on-device, no cloud processing is used for 3D mapping, and room visualization and mapping remain private.
As for the combination of VisionOS and GIS applications, currently has great application potential in virtual geographical environment simulation, spatial environment enhancement and digital twins. Spatial computing involves the combination of physical environment and digital environment. It is the cornerstone of the development of the metaverse and the future market value high ground. In this sense, VisionOS, as the first space operating system, deserves long-term follow-up and attention.
reference:
https://developer.apple.com/visionos/
https://juejin.cn/post/7243725358129561637
https://www.macrumors.com/roundup/visionos/
https://medium.com/@anishalok/beyond-screens-redefining-user-interfaces-and-experience
https://quan.ithome.com/content/sharedetail?id=274690
Comment | Vision Pro in the eyes of AR/VR industry executives - Zhihu
Introducing Apple Vision Pro, Apple’s first spatial computing device - Apple (China)
https://benic360.com/apple-vision-pro-introduction/
(Some materials come from the Internet, and the copyright belongs to the original author. This article is only for sharing and interpretation, not for commercial purposes. If there is any infringement, please inform us and we will delete the relevant content in time. Thank you!)
「Future GIS Laboratory」, as the upstream scientific research institution of SuperMap Research Institute, is committed to gaining insight into the future development direction of the GIS industry and verifying cutting-edge technologies Feasibility of implementation and rapid transformation of latest research results into key products. The department focuses on scientific research and innovation. The team atmosphere is free and harmonious, and the scientific research atmosphere is relatively strong. Everyone has the opportunity to delve into the cutting-edge directions that interest them.