ARKit functions for beginners

1. Introduction to ARKit

ARKit is a framework released by Apple in WWDC2017 for developing AR functions on the iOS platform. AR, the full name of Augmented Reality, is a technology that adds a virtual world created by a computer program to the real world captured by a camera.

The AR system consists of the following basic parts:

• Capture the real world: ARKit uses cameras to capture images of real-world scenes.
• Virtual world: Use SceneKit to build a virtual world.
• Combining the virtual world with the real world: ARKit is responsible for integrating information from the real world and the virtual world and rendering an AR world.
• Maintain world tracking: When the real world changes (moving the camera), the position and angle change information of the current camera relative to the initial position must be tracked, so that the position and angle of the virtual world relative to the real world can be rendered in real time.
• Scene analysis: refers to analyzing whether there are key information such as feature points and planes in the real world.
  Processing interaction with the virtual world: This refers to processing whether a virtual object is clicked or whether to add/delete an object when the user clicks or drags the screen.

2. Several main classes in the ARKit API

1. ARSCNView

This is a subclass of SceneKit's main view SCNView, which helps us enhance the real-time camera view with 3D content rendered by SceneKit, renders the real-time video stream of the device camera as the scene background, and automatically matches the SceneKit space and the real world. This class does the following things:

• Render a live video stream from the device camera in a view and set it as the background of the 3D scene
• ARKit's 3D coordinate system will match SceneKit's 3D coordinate system, so objects rendered by this view will automatically match the enhanced ARKit world view
• Automatically moves the virtual SceneKit 3D camera to match the 3D position tracked by ARKit, so there is no need to write code to connect ARKit movement events with SceneKit 3D rendering.

ARSCNView itself will not do AR processing, but it requires an AR session object to manage the device camera and motion processing. Responsible for synthesizing the information of the virtual world (SceneKit) and the real world information (collected by the ARSession class), and then comprehensively rendering them to present an AR world.

2. ARSession

Each augmented reality session requires an ARSession instance. It is responsible for controlling the camera, aggregating all sensor data from the device, and more to build a seamless experience (responsible for collecting real-world information). The ARSCNView instance already has an ARSession instance, you just need to configure it at the beginning.

3. ARFrame

ARFrame contains two parts of information: ARAnchor and ARCamera.
Among them, ARCamera refers to the current camera position and rotation information. This part of ARKit has already been configured for us, no special configuration is required.

4.ARanchor

It refers to the anchor point in the real world. The specific explanation is as follows: ARAnchor (anchor point) can be understood as a certain point or plane in the real world. The anchor contains position information and rotation information. After getting the anchor, you can place some virtual objects at the anchor. It can also be bound to SCNNode. It has a subclass: ARPlaneAnchor, which specifically refers to an anchor point representing a horizontal plane.

5. ARWorldTrackingSessionConfiguration

This class will tell ARSession that six degrees of freedom need to be used when tracking the user in the real world, roll, pitch, yaw and transformation on the X-axis, Y-axis, and Z-axis. Without this class, you can only create an AR experience that rotates at the same point to view enhanced content. With this class, you can move around objects in 3D space. If you do not require transformations on the X, Y, and Z axes, and the user will remain in a fixed position while projecting enhanced content, you can use the ARSessionConfiguration class instead of this class to initialize the ARSession instance.

ARSession is the core of the entire ARKit system. ARSession implements important functions such as world tracking and scene analysis. ARFrame contains all the information output by ARSession and is the key data source for rendering.
ARKit itself does not provide an engine for creating virtual worlds, but uses other 3D/2D engines to create virtual worlds. The main engines available on iOS systems are:

• Apple 3D Framework - SceneKit.
• Apple 2D Framework - SpriteKit.
• Apple GPU-accelerated 3D graphics Engine - Metal.
• OpenGl
• Unity3D
• Unreal Engine

ARKit needs to look at content that can detect many useful feature points. The situations in which feature points may not be detected are as follows:

1. Poor light: not enough light or too strong specular reflection. Try to avoid these poor lighting situations.
2. Lack of textures: If the camera is pointed at a white wall, features cannot be obtained and ARKit cannot find and track the user. Try to avoid looking at solid colors, reflective surfaces, etc.
3. Fast movement: Usually, detection and estimation of 3D poses only rely on pictures. If the camera moves too fast, the picture will be blurred, resulting in tracking failure. However, ARKit will use visual inertial odometry, comprehensive picture information and device motion sensors to estimate the position of the user's steering. So ARKit is very powerful in terms of tracking.

6. ARSCNViewDelegate

First, let’s introduce the agent of ARSCNView: ARSCNViewDelegate, which has the following callback methods.

func renderer(SCNSceneRenderer, nodeFor: ARAnchor)

When ARSession detects an anchor point, you can decide in this callback method whether to return a SCNNode to it. The default is to return an empty SCNNode(). We can change it according to our own needs to only return an anchor point when a plane anchor point (ARPlaneAnchor) is detected, and so on.

func renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor)
func renderer(SCNSceneRenderer, willUpdate: SCNNode, for: ARAnchor)
func renderer(SCNSceneRenderer, didUpdate: SCNNode, for: ARAnchor)
func renderer(SCNSceneRenderer, didRemove: SCNNode, for: ARAnchor)

The above method will call back when an anchor point has been added, will be updated, has been updated, or a virtual anchor point has been removed.

7. ARSessionDelegate

The ARSession class also has its own delegate: ARSessionDelegate

func session(ARSession, didUpdate: ARFrame)

In ARKit, when the user moves the phone, many ARFrames are updated in real time. This method will call back when the ARFrame is updated. It can be used in scenarios where you always want to keep a virtual object in the middle of the screen. You only need to update the position of the node to the center point of the latest ARFrame in this method.

func session(ARSession, didAdd: [ARAnchor])
func session(ARSession, didUpdate: [ARAnchor])
func session(ARSession, didRemove: [ARAnchor])

If ARSCNViewDelegate or ARSKViewDelegate is used, the above three methods do not need to be implemented. Because in addition to the anchor point, the other Delegate method also contains node information, which allows us to have more information to process.

3. ARKit example

1. Import the framework

import ARKit

2. Set up SceneKit View

private lazy var sceneView: ARSCNView = {
        let tmpView = ARSCNView()
        tmpView.frame = CGRect(x: 0, y: kNaviBarMaxY, width: kScreenWidth, height: kScreenHeight - kNaviBarMaxY)
        // 添加光源
        tmpView.autoenablesDefaultLighting = true
        tmpView.automaticallyUpdatesLighting = true
        return tmpView
    }()

3. Configure ARSCNView Session

Our AR application observes the world and surrounding environment through a camera. So next we need to set up the Camera:

Configure ARKit SceneKit View. Insert the following code in the ViewController class:

override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        let configuration = ARWorldTrackingConfiguration()
        sceneView.session.run(configuration)
    }

The World Tracking configuration tracks the device's orientation and location, and it can also detect real-world surfaces through the device's camera.
Set up sceneView's AR session to run the configuration we just initialized. AR sessions manage motion tracking and camera image processing of view content.

override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)
        sceneView.session.pause()
    }

Stop tracking and view the content of the image.

4. Camera authorization

5. Add 3D objects

private func addBox() {
        let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0)
        
        let boxNode = SCNNode()
        boxNode.geometry = box
        boxNode.position = SCNVector3(0, 0, -0.2)
        
        let scene = SCNScene()
        scene.rootNode.addChildNode(boxNode)
        sceneView.scene = scene
    }

• Create a Box with 1 Float = 1 meter.
• Create a node. node represents the position and coordinates of the object in three-dimensional space. The node itself has no visible content.
• Set a shape (Box) for node.
• Set the position of the box. This position is relative to the camera. The right side is X positive and the left side is X negative. Up means Y is positive, downward means Y is negative. Backward means Z positive, forward means Z negative.
• Create a scene (SceneKit scene) and add boxes to the scene.
• Set the sceneView's scene to display the scene just created.

6. Add gestures

1. (Click to delete 3D object)

private func addTapGestureToSceneView() {
        let tap = UITapGestureRecognizer(target: self, action: #selector(didTap(withGestureRecognizer:)))
        sceneView.addGestureRecognizer(tap)
    }
    
    @objc func didTap(withGestureRecognizer recognizer: UIGestureRecognizer) {
        let tapLocation = recognizer.location(in: sceneView)
        let hitTestResults = sceneView.hitTest(tapLocation)
        guard let node = hitTestResults.first?.node else { return }
        node.removeFromParentNode()
    }

2. To add multiple 3D objects
   create an extension at the end of the ViewController class:

extension float4x4 {
    var translation: float3 {
        let translation = self.columns.3
        return float3(translation.x, translation.y, translation.z)
    }
}

extension converts the matrix to float3. Modify the addBox() method:

func addBox(x: Float = 0, y: Float = 0, z: Float = -0.2) {
    let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0)
    
    let boxNode = SCNNode()
    boxNode.geometry = box
    boxNode.position = SCNVector3(x, y, z)
    
    sceneView.scene.rootNode.addChildNode(boxNode)
}

Modify the didTap (using gesturerecognizer:) method, inside the guard let statement and before the return statement. Add the following code:

let hitTestResultsWithFeaturePoints = sceneView.hitTest(tapLocation, types: .featurePoint)
 
if let hitTestResultWithFeaturePoints = hitTestResultsWithFeaturePoints.first {
    let translation = hitTestResultWithFeaturePoints.worldTransform.translation
    addBox(x: translation.x, y: translation.y, z: translation.z)
}

Next implement using x, y and z to add a new box when a click is detected. The code of the didTap(withGestureRecognizer:) method is as follows:

@objc func didTap(withGestureRecognizer recognizer: UIGestureRecognizer) {
    let tapLocation = recognizer.location(in: sceneView)
    let hitTestResults = sceneView.hitTest(tapLocation)
    guard let node = hitTestResults.first?.node else {
        let hitTestResultsWithFeaturePoints = sceneView.hitTest(tapLocation, types: .featurePoint)
        if let hitTestResultWithFeaturePoints = hitTestResultsWithFeaturePoints.first {
            let translation = hitTestResultWithFeaturePoints.worldTransform.translation
            addBox(x: translation.x, y: translation.y, z: translation.z)
        }
        return
    }
    node.removeFromParentNode()
}