In the last lesson, we took you to decrypt AR augmented reality . After understanding the past and present of this technology and its application scenarios, you can see that AR is beginning to ride the wind and waves in the market now, and it seems that it has the next trend. The emergence of AR products has set off a wave of innovation in the science and technology circle, allowing the virtual world and the real world to merge with each other, completing the "leapfrog" of the screen, and the design of human-computer interaction has begun to move from a two-dimensional plane to a three-dimensional world. So augmented reality sounds so high-end, like the technology of the future, how to understand and produce it? This article gives you the answer.
Extended reading: [ ARCore Development Introduction Course Series] Decrypt AR Augmented Reality
Explain the profound things in a simple way,
Understand the current situation and core ideas of AR technology
AR content creators need to keep in mind the following six core concepts-the realization of these concepts is the key to the “combination of virtual and real” in AR.
放置 (Placing and Positioning assets)
First, we need to introduce the concept of "placement": a stationary AR object needs to be fixed to a certain point in the real environment, such as a wall, floor, ceiling, or a certain position in mid-air, so that the experiencer can feel the "combination of reality and reality." In either case, placement means that the virtual object must be stably "fixed" in the corresponding position in the real environment. Even if the experiencer himself moves the position, the virtual object can still stay in the original position. Let's look at the following specific examples:
The coffee cup in the picture is an AR virtual object. When the user switches the perspective, the virtual coffee cup on the table is placed as before. This AR experience prevents the user from jumping in and maintains the realism of the AR experience.
Scale & Size of assets
In addition to staying in the real environment where the AR object is, it also needs to be able to reflect changes in size as the observation distance changes. You must be familiar with these scenes: a car is coming towards you from a distance. It is very small at first. As it keeps getting closer, the car gets bigger and bigger; when you are admiring a painting, stand angle, distance High and low, the view is also different. In a carefully crafted AR experience, the virtual object also needs to change its size and image according to the actual distance, viewing angle, and direction from the observer, giving people the desired visual impact, which is zooming. It can make the AR virtual object break away from the virtual and become a "god operation" that can truly replace the real thing.
Occlusion
Occlusion, as the name suggests, refers to the situation where one object is blocked by another object. In the AR field, if virtual objects want to look alive, they must be able to hide behind other virtual objects and real objects according to their relative positions, reflecting the occlusion relationship in the real world. This means that AR hardware must not only understand the location of the object in the room, but also determine the relative distance between the virtual object and other entities, and the relative distance between the virtual object and the user, which is the depth map of the entire scene.
The feeling that the AR object actually exists in the real environment in the application is called "immersion." To obtain this immersive interactive experience, you need to constantly recalibrate to achieve seamless occlusion. Because users are always on the move, this is one of the biggest challenges in creating successful AR content.
Lighting
Just like real objects, AR virtual objects need to map the light and dark changes of the object to different lighting conditions to make us feel realistic. The color, light and shade, and shadow of these objects need to be adjusted accordingly when the initial lighting of the scene and the lighting changes. For example, in the AR experience, if you dim the lights in the house, the color and brightness of the virtual object should also change; if the object moves, then its light receiving direction and shadow should also move.
Solid
AR objects need to be physical. Although this sounds self-evident, it still needs to be deliberately implemented by developers. When creating AR content, virtual objects must not overlap with real objects or occupy the same space, nor should they be suspended in mid-air, otherwise it will affect the user's immersive interactive experience.
Context Awareness
The last element to create a sense of reality is environmental perception. This is the most difficult feature to achieve. It requires AR hardware devices to understand everything in the environment and perceive its height, size, etc. When the subject objects move in space, how their situation will change, such as the placement of tables and chairs in the real environment; what items are on the table. There are many factors that need to be tracked when the device perceives the surrounding environment. AR devices need to quickly generate perceptual information while ensuring that the realism, fluency, and functionality of virtual objects will not decrease. This is one of the biggest challenges facing AR content creators today.
Important features that ARCore must know
In order to realize the above six AR concepts and help developers create better AR works, facing various challenges, companies are investing in the development of software tools. ARCore, as a platform for building augmented reality applications launched by Google, can take advantage of advances in cloud software and hardware devices to place virtual objects in the real world, helping developers build augmented reality experiences with simpler methods and more powerful tools. Next, let us understand how ARCore helps developers achieve this vision.
Motion Tracking-Motion Tracking
AR relies on computer vision to perceive the environment and recognize the objects in it. The first step in the computer vision process is to transmit visual information, that is, the environment around the hardware to the "brain" inside the device. The process of scanning, identifying, subdividing and analyzing environmental information in immersive technology is called tracking. In the AR field, there are two ways to track: Outside-in Tracking and Inside-out Tracking.
Outside-in Tracking
When using outside-in tracking, the camera or sensor is not located inside the AR device, but installed at other locations in the space. For example, on the wall or bracket to ensure that the AR device's view is not blocked. At the same time, these "plug-in devices" will transmit information to the AR device directly or through a computer. In theory, the size of the external camera or sensor can be selected according to the size of the placement space, but it will encounter power consumption problems. As functionality is enhanced, portability will decrease. If the link between the headset and the external sensor is interrupted, even for a short while, it may lose the tracking signal and destroy the immersion of the visual effect.
(Tracking from the outside in)
(Tracking from the inside out)
Inside-out Tracking
When using inside-out tracking, the camera and sensor are built into the device body. In our daily lives, many wireless devices have built-in cameras and information processors to observe the environment. The most typical example is a smartphone. AR headsets such as Microsoft HoloLens also use AR tracking from the inside out. There are 5 cameras for analyzing surroundings, 1 depth measurement camera, 1 HD camera, 1 light sensor, and 4 microphones in the HoloLens frame.
However, all these hardware require space, power consumption, and heat generation. Ideally, a standalone AR device can be portable and used at any time like a smartphone. But before that, smartphone-based AR has become the main way for most people in the world to interact with AR content.
Whether providing an AR experience on a smartphone or a stand-alone headset, every AR application is designed to display realistic virtual objects. One of the most important functions of the ARCore system is motion tracking . The AR platform needs to know when a user moves. The general technology behind this is called "Real-time Synchronous Positioning and Mapping (SLAM)". This is the process by which robots and smartphones perform technical analysis, understand the real world, and determine their orientation. The SLAM process requires a series of hardware to collect data, such as cameras, depth sensors, light sensors, gyroscopes, and accelerometers. ARCore also uses these hardware to understand the environment, and uses this information to set appropriate anchor points and render the AR experience by detecting planes and feature points. Specifically, the process of motion tracking through ARCore is the process of "parallel ranging and mapping" (COM for short).
Simply put, COM can tell the smartphone its position in space relative to the surrounding environment, and achieve this goal by capturing the visually different features in the environment. We call this visual difference feature "feature point". The "feature point" can be the edge of the chair, the lighting switch on the wall, the corner of the carpet, or any point that may appear in the environment and the position is always the same. Therefore, any high-contrast visual effect can become a "feature point". ARCore combines the perceived feature points with the motion information (also called inertial data) provided by the smartphone.
Many smartphones nowadays are equipped with a gyroscope for measuring the angle of the phone, an accelerometer for measuring the speed of the phone, feature points, and inertial data combined to help ARCore determine the orientation and position of the phone in the environment. When ARCore has an understanding of the environment, it can better place virtual objects in the real environment and bring users a better interactive immersive experience.
Environmental understanding: Finding characteristic points and planes
—— Environmental Understanding: Feature points and Plane-finding
Environment understanding refers to the process by which ARCore sees the environment around the AR device and processes and uses related information. This process starts with "feature points". ARCore will use the phone's camera to capture clusters of feature points on the surface to create a plane. The concept of "plane search" refers to ARCore's ability to detect and generate flat surfaces. ARCore can perceive these planes, and can correctly place and adjust 3D objects in the physical space. For example, place the virtual object on the real ground and table, otherwise the virtual object will "fly through the clouds and fog" in reality. Therefore, even virtual objects must follow the same physical rules as real objects.
Light Estimation-Light Estimation
I believe you must have noticed that when you pick up the phone to answer a call, the phone screen will automatically go off. When the surrounding environment changes, the phone screen will adjust the light and dark changes accordingly. The realization of the above functions benefits from the light sensor equipped with the smart phone. The current AR technology supports global estimation of light, such as perceiving the brightness, color, and color temperature of the entire environment. ARCore scans the pixels of the camera image to determine the average value of incident light to provide the best illumination for the AR object.
Light and shadow are the key factors for the naked eye to judge whether things are real. The reason why virtual objects look real is largely because they truly reflect the effects of lighting. In reality, conditions such as the direction of the light source, color temperature, and highlights will change the appearance of objects. Only when there is light and dark interlacing, and shadow, an object can look real. ARCore can use light estimation to help users create realistic AR applications and gaming experiences.
Anchor-Anchor
After ARCore completes the analysis of the surrounding environment and places the feature points, you can start to set anchor points for the AR object . The anchor point is also called the anchor point, which is the position where the virtual object is always located, and is mainly suitable for static virtual objects. For example, if you want to place a virtual lamp on the table, you need to set the anchor point on the table. As shown in the figure below, this is a horizontal plane discovered and recognized by ARCore. Now when the light is placed, it will be fixed in a position. When the device moves, it will react accordingly, but if you turn around, this virtual light will stay on the desktop.
For things like virtual airplanes that need to be suspended in the real environment, the above anchoring method obviously does not work. Because the setting of anchor points requires plane search, motion tracking, and coordinated work with a vision system. Therefore, for the AR platform, it will be difficult to accurately determine the anchor point. Only by continuous exploration can the plane image really come alive.
I believe that through this course, you have a certain understanding of the core technology of AR and the powerful functions of the ARCore platform. In the next lesson, we will sort out the technical challenges that AR currently faces in detail for you, and explore the possibility of further innovation in AR technology.