Guide
It takes 20 minutes to read about 6000 words in this article. The content includes:
- Developers' basic knowledge of AR
- Selection outline for mainstream AR engines
- Evaluation and implementation plan of Vuforia engine
- Evaluation and implementation plan of EasyAR engine
- Evaluation and implementation plan of ARFoundation engine
- Evaluation and implementation plan of self-developed AR engine
- Multi-engine integration solution
- AR engine selection and implementation summary
EEA is the Explore Entertainment Academy (Explore Entertainment Academy). We will regularly share our articles on game research, hoping to help everyone and our own theoretical system of game design by solving one proposition after another. And ultimately increase productivity.
1. Research environment
研究目的：实现AR引擎，提供运动追踪SLAM，平面识别、图像识别、人物遮挡、光照估计、阴影等功能支持。
开发工具：Unity 2020.1.4f1c1。
开发言语：C#
2. Developers' basic knowledge of AR
AR全称Augmented Reality，即为增强现实技术。由现实场景和虚拟场景两部分叠加显示，为用户提供更丰富的感官体验和更立体的信息传达。
开发场景中必定有两个“摄像机”分别拍摄现实场景和虚拟场景，渲染叠加显示即为AR场景。
现实场景摄像机-手机摄像头硬件：程序获取手机摄像头拍摄的现实画面，并渲染到画布上（需要获取手机摄像头权限）。
虚拟场景摄像机-即Unity的Camera：拍摄3D场景渲染。
AR的最大应用价值是基于AI扩展功能，通过图像识别、动作识别、人脸识别等方式，智能分析认知现实场景，在虚拟场景做对应的显示，达到增强现实效果。
Further reading materials:
From AR to ZR: 26 concepts about reality!
3. Selection outline of mainstream AR engine
本人能力时间有限，无法对每个引擎都研究透彻，若有错误的地方欢迎大家指出。
大多数AR引擎都是基于ARCore和ARKit，分别用于支持Android设备和IOS设备，并在基础上再次封装扩展开发。少部分AR引擎有自行研发核心功能，用于适配更多手机型号或增强功能。
1. ARCore:
Description: The AR development platform launched by Google, which can be used to develop AR applications on the Android platform.
设备支持：支持Android，不支持IOS设备。有相当一部分Android设备不支持，与是否有安装Google ARCore的有关，与手机价格和新旧程度无关。
功能：运动追踪、平面追踪、点云图、云锚点、光照估计、环境探针、人脸追踪、2D图片追踪、人物遮挡、射线测试
2. ARKit:
Description: Apple's AR development platform, which can be used to develop AR applications on the iPhone and iPad platforms.
设备支持：支持IOS设备，不支持Android
功能：运动追踪、平面追踪、点云图、云锚点、光照估计、环境探针、人脸追踪、动作捕捉，2D图片追踪、3D物体追踪、人物遮挡、射线测试
3. Vurforia:
Description: Also known as Qualcomm AR, it cooperates with Qualcomm to develop products. Support Unity
依赖引擎：ARCore、ARKit
设备支持：支持Android和IOS设备
识别数据：平台云端存储
4. EasyAR:
Description: The AR engine developed by Visionstar Information Technology has developed a set of SLAM, which can be adapted to more models. Support Unity
依赖引擎：ARCore、ARKit
设备支持：支持Android和IOS设备
识别数据：平台云端存储
功能：最新一些功能，如人物遮挡、光照估计是否有接口实现，待确定暂时未找到。
5. ARFoundation:
Description: Unity encapsulates ARKit and ARFoundation, no new functions are implemented, which is convenient for unified interface and multi-platform use. Support Unity
依赖引擎：ARCore、ARKit
设备支持：支持Android和IOS设备
识别数据：自行存储
6.SenseAR:
Description: AR engine developed by SenseTime . No further research and evaluation. Support Unity
设备支持：设备需要预装或手动安装SenseAR，版本间不兼容。
功能：运动最终、光照估计、手饰识别、图像识别和跟踪、人脸跟踪与重建、云锚点、三维物体识别与跟踪、实时三维重建
7. Baidu AR:
Description: An AR engine developed by Baidu. Mainly in AI image recognition, face recognition function. Support Unity
8. Ali AR:
Description: AR engine developed by Ali. Mainly in 2D image recognition, 3D object tracking function. SDK development can only be obtained after commercial contact.
9. Tencent AR:
Description: AR engine developed by Tencent. The official website information is very simple, and no further research and evaluation have been done. Support Unity
10. Huawei AR:
Description: The AR engine developed by Huawei. Added additional support for Huawei equipment, which can be adapted to more Huawei mobile phones. Support Unity
依赖引擎：ARFoundation
11. NetEase AR:
Description: AR engine developed by NetEase. Through business cooperation, NetEase completes the delivery of requirements.
12. LandMarkAR:
Description: AR engine developed by Douyin . The official website and available SDK were not found.
13. VoidAR:
Description: The AR engine developed by Chengdu Miyou . The customer service said that it is fully independent research and development, does not rely on ARCore, ARKit, and the equipment is highly adaptable. The SLAM function test results are very volatile and cannot be used in practical applications. Support Unity
Fourth, the evaluation and implementation plan of the Vuforia engine
1. Research environment:
Vurforia version: add-vuforia-package-9-3-3
Unity版本：Unity 2020.1.2f1c1
2. Configure SDK and create AR project:
2.1 Log in to vuforia official website, click Register to register a developer account:
2.2 在开发者平台中，点击Develop，在License Manager中，创建AR App的license Key；
2.3 登陆官网下载SDK，将下载好的unitypackage文件导入到Unity项目；
2.4 Unity项目删除原来的Main Camera，点击GameObject-Vuforia Engine-AR Camera添加AR Camera；
2.5 选中AR Camera点击Open Vuforia Configuration，将申请的license Key复制到App License Key输入框中，AR场景必要设置已完成了；
If we want to further create an AR project for image recognition, we continue to do:
2.6 在开发者平台，点击Develop，在TargetManager中点击Add Database，添加对象数据库；
2.6.1 数据库类型有三种：Device（设备识别），Cloud（云端识别），VuMark（条形码）
2.7 点击进入数据库后，点击Add Target将需要识别的图片传入，并填入相应的信息点击Add；
2.7.1 Type：根据需求选择，这里选择Single Image（单个图片）
2.7.2 File：在本地选择识别图上传
2.7.3 Width：识别图片的宽度。用于建立Unity场景中的单位长度，场景中所有其他物体的大小是以这个值为参照建立的。Vuforia中的单位长度是以米来计算。这个值可以是任意的，但是最好比Camera的Near Clip值要大，不然在镜头靠近时你可能会看不到相关内容。这里可输入为1；
2.7.4 Name：识别图片名字。名字唯一，用于识别各个识别图。
2.7.5 Status：Active表示可用。（创建完成后可查看）
2.7.6 Augmentable：可识别度评级，最少需要3颗星才能保证高质量的识别，最高5颗星。（创建完成后可查看）
2.8 将Vuforia - Prefabs中的ImageTarget预制体拖入场景中，并将识别后要显示的模型作为子对象挂在ImageTarget下面；
2.9 在ImageTarget的ImageTargetBehaviour中，Type选择From Database，在Database和ImageTarget选项分别选择之前设置的数据库和图片对象；
2.10 至此AR图像识别已完成，可运行，用摄像头拍摄识别图，对应的模型就会显示出来。
Comparison of Database Device and Cloud Type
rWfTQQuYzfVD1tp4NPNa.png
3. Analysis of key classes:
VuforiaBehaviour: Start the start class of the Vurforia engine
VuforiaManager: Manage the update of trackable objects, their status and position in the camera
VuforiaARController: The VuforiaBehaviour class handles tracking and triggers local video. Background rendering. This class will update all Trackables in the scene.
WebCamARController: MonoBehaviour manages the use of the "Play Mode" webcam in Windows or Mac.
4. Evaluation:
4.1 One of the most popular AR engines in the world, and the accuracy and flexibility of image recognition are well optimized;
4.2 测试效果总体不错，但Android设备支持限于ARCore，适配性一般；
4.3 开发需要在国外找资料，国内资料零散不全
Further reading materials:
Vuforia Chinese Community
5. Evaluation and implementation plan of EasyAR engine
1. Research environment:
EasyAR version: EasyARSenseUnityPlugin_4.1.0.811
Unity版本：2020.1.4f1c1（官方建议长期支持版本 Unity 2019.4、Unity 2018.4、Unity 2017.4。测试反馈：2020版本是暂没发现影响因素）
EasyAR Sense Unity建立在EasyAR Sense C# API之上的非常薄的封装，用于在Unity中暴露EasyAR Sense的功能。建议阅读 EasyAR Sense 文档 来理解EasyAR Sense是如何工作的。
2. Configure the SDK and create an AR project (the official website document has a detailed tutorial):
2.1 Log in to the EasyAR official website and register an account;
2.2 进入开发者中心，申请Sense许可证密钥License Key；
2.3 登陆官网下载SDK，将下载好的unitypackage文件导入到Unity项目；
2.4 从Unity菜单中选择<EasyAR -> Change License Key>并在Inspector面板中输入准备工作中创建的License Key；
If you can’t wait to see the effect of the AR scene
2.5 可登陆官网下载实例文件夹，将下载好的unitypackage文件导入到Unity项目；
2.6 项目中Project - Samples - Scenes文件夹中，打开任意场景运行即可体验对应的AR场景（场景说明看英文名或官网教程）；
If you want to freely learn how to use the engine
2.7 打开项目摄像机Main Camera，修改Camera属性设置：
2.7.1 Tag: 设置Camera Tag为 MainCamera 。如果不做这个设置，则需要在场景中 EasyAR 节点的配置中选择 Assemble Mode 为 Manual 并手动配置Camera。
2.7.2 Clear Flags: 需要选择为 Solid Color 以确保Camera图像可以正常渲染。如果选择为 Skybox ，Camera图像将无法显示。
2.7.3 Background: 这个非必需配置，考虑到使用体验，建议将背景颜色设为黑色以便在Camera设备打开前和切换时以黑色显示。
2.7.4 Clipping Planes: 根据识别物体实际的物理距离设置。这里设置Near为0.1（米）以避免相机离物体较近时无法显示。
2.8 在Project - EasyAR - Prefabs文件夹中：
Composites：该文件夹下的预制体是引擎预先配好的AR组件，拖入场景中即可使用对应的AR功能（适合新手）；
Primitives：该文件夹下的预制体是单个功能的引擎，可自由组合使用自己想要的AR功能
3. Analysis of key categories:
ARSession: MonoBehaviour that controls AR sessions in the scene. A session contains a set of components assembled into ARAssembly and controls the data flow throughout the life cycle. This class is the entrance of AR. If you want to implement a completely different AR workflow, you can create a new session class and replace this class in the scene.
EasyARController: Run first. Control the initialization of EasyAR Sense and some global settings of MonoBehaviour in the scene.
RenderCameraController: Controls the MonoBehaviour of the Camera in the scene, and the Camera projection matrix will reflect the CameraDevice or other optical devices in the real world.
CameraImageRenderer: MonoBehaviour that controls camera image rendering in the scene. This class currently does not support Unity universal render pipeline (URP), but you can extend the implementation of this class to support URP.
VIOCameraDeviceUnion: Motion tracking related classes. MonoBehaviour, which controls VIO camera devices (MotionTrackerCameraDevice, ARKitCameraDevice, ARCoreCameraDevice) in the scene, provides function extensions in the Unity environment. If necessary, you can use Device directly. MotionTracker is a set of motion tracking implemented by EasyAR itself.
ARAssembly: It implements a typical assembly strategy for all EasyAR Sense components, while controlling the data flow in ARSession. Available for ARCamera.
To check whether the mobile phone supports EasyAR, the following functions can be used:
t36JB2PTY7NxIalcJsEp.png
4. Matters needing attention/project configuration:
4.1 Android project, Android API Level must be greater than or equal to 17;
4.2 不支持Universal Render Pipeline，即URP项目不支持的。
手机摄像头渲染与URP渲染相互有影响，强制使用URP项目手机摄像头运行不了；
可以通过扩展EasyARSenseUnityPlugin的脚本来支持URP和HDRP；
所有的渲染API调用都组织在CameraImageRenderer脚本组件中。可以扩展这个脚本并修改一些shader来支持URP和HDRP。
4.3 Unity使用mono脚本后台不支持Android arm64-v8a。如果需要使用应开启IL2CPP；
4.4 对于运动追踪功能，各大AR引擎主要通过图像识别、陀螺仪、加速传感器等传感器结合实现。经测试，在手机摄像头朝向上方的时候，都有一定几率会追踪失败，ARCore会重定位恢复显示，EasyAR的MotionTracker会直接卡死。
5. Evaluation:
5.1 One of the best AR engines in China, with complete data and friendly interface;
5.2 MotionTracker可以适配更多的手机型号，但对比ARCore，还是有差距，一些BUG还待需优化；
5.3 稠密矩阵对现实物体的贴合度体验有差距，还待续优化；
5.4 总之来说EasyAR自研发的功能个人感觉有些鸡肋，但总体效果还是不错。
Further reading materials:
EasyAR community tutorial
6. Evaluation and implementation plan of ARFoundation engine
1. Research environment:
ARFoundation version: Version 4.1.0-preview.12-November 03, 2020
Unity版本：2020.1.4f1c1
2. Configure SDK and create AR project:
If you can’t wait to see the effect of AR scene operation
2.1 在Github上下载ARFoundation的示例代码，里面所有功能都有对应的示例场景；
2.2 在示例项目中，Project - Scenes文件夹下，打开任意场景运行即可体验对应的AR场景；
If you want to freely learn how to use the engine
2.3 在Unity项目中，打开Window - Package Manager安装引擎包，AR Foundation、AR Subsystems、ARCore XR Plugin、ARKit Face Tracking、ARKit XR Plugin，有些包会跟着其他包一起自动安装，注意版本一定要一致：
2.4 在Hierarchy中创建一个空对象A，添加组件ARSession和ARInputManager；
2.5 在Hierarchy中创建一个空对象B，添加组件ARSessionOrigin；
2.6 打开项目摄像机Main Camera，修改Camera属性设置Clear Flags需要选择为 Solid Color，添加组件ARCameraManager，ARCameraBackground，ARPoseDirver；
2.7 再将Main Camera移动对象B下面，作为子对象。AR场景已完成完毕可以运行查看效果了。
3. Key category analysis:
ARSession: AR session controls the life cycle of AR experience by enabling or disabling AR on the target platform
ARSessionOrigin: Convert trackable features (such as planes and feature points) to their final location in the Unity scene
ARPoseDriver: Drive the local position and direction of the parent GameObject according to the tracking information of the device
ARCameraManager: The enabling function of the AR camera, including the management of the device camera texture and the properties of setting the light estimation mode.
ARCameraBackground: displays the video feed from the device camera as the rendered background of the scene at runtime
ARTrackedImageManager: A class used for image recognition and tracking, it needs to be hung under object B to take effect;
ARPlaneManager: A class used for plane detection, which can take effect after mounting object B;
AROcclusionManager: A class used for character occlusion, which needs to be hung under the Main Camera to take effect;
4. Matters needing attention/project configuration:
4.1 Player Setting setting, Android project, Android API Level must be greater than or equal to 24;
4.2 Player Setting设置，XR Plug-in Management勾选ARCore和ARKit；
4.3 Player Setting设置，删除 vulkan，Android不支持Vulkan；
4.4 还有一些Player Setting设置因版本不一样，根据实际情况自己选择设置，非必须；
4.4.1 不勾选 Multithreaded Rendering；
4.4.2 不勾选 Auto Graphics API；
4.4.3 不勾选 ARCore Supported；
4.5 ARSession.state可获取引擎的运行状态，可用于检测ARFoundation是否支持设备，各功能可通过subsystem???.descriptor???.supports???的方式查看是否支持。注意引擎和子系统必须要先运行才能知道是否支持；
5. Evaluation:
5.1 ARFoundation has a good interface encapsulation and is easy to use. The latest functions released by ARCore and ARKit are provided in a timely manner, and the version is continuously optimized;
5.2 ARFoundation版本和Unity版本有匹配要求，在Unity2019打包最新版本会失败；
5.3 ARFoundation配置出错后，没有社群可以问答，只能对比示例场景一点一点找问题；
5.4 如果环境太暗，则设备运动追踪可能会“丢失跟踪”，因为基于图像识别，所有SLAM都会出现此情况；
Further reading materials:
Unity AR + GPS Location supports ARFoundation and Vuforia
7. Evaluation and implementation plan of self-developed AR engine
1. R&D purpose:
Goal 1: Realize AR scene, overlay display of real scene and virtual scene
目标二：虚拟场景摄像机Camera能同手机摄像头一起旋转、移动
目标三：最大程度上适配更多的手机、解决ARCore和ARKit对大量手机不适配情况
2. Development idea:
Based on the basic cognition of AR scenarios, and based on the hardware devices available in most mobile phones, complete the most basic AR functions
2.1 获取手机摄像头，并获取摄像头拍摄的画面渲染出来；
2.2 通过手机传感器（陀螺仪、加速传感器、指南针等）识别手机的方向和移动，实现控制摄像机Camera旋转移动；
3. Implementation scheme:
3.1 WebCamTexture can obtain the picture taken by the mobile phone camera and convert it into display material. The core code is as follows (variable type and variable name are the same):
3.1.1 判断是否支持手机摄像头，并设置摄像头渲染显示
tXQvab6cKUjbHP3LiVIb.png
3.1.2 Update the Update call every frame, dynamically detect the camera display width and height and the mobile phone rotation direction modification
otzDLownCzk2BTB2QptV.png
3.2 Input.gyro can get a gyroscope, which is used to get the phone's spatial orientation, and then control the synchronous rotation of the spatial camera
3.2.1 打开项目摄像机Main Camera，修改Camera属性设置Clear Flags需要选择为 Solid Color。
3.2.2 在Hierarchy中创建一个空对象A，将Main Camera拖入对象A下，作为子对象；
3.2.3 将对象APosition设置为（0,0,0），Rotaion设置为 Quaternion.Euler(90f, 180f, 0f)；
3.2.4 在帧循环Update中，写camera.localRotation = gyro.attitude * new Quaternion(0f, 0f, 1f, 0f); 摄像机即可随着手机旋转而旋转；
3.3 gyro.userAcceleration可获取空间移动加速度，Input.acceleration可获取加速传感器的线性加速度，预想根据加速度计算空间位置从而控制摄像机移动，但实际效果误差太大，移动效果尝试失败ε=(´ο｀*)))唉。
3.4 Input.compass.trueHeading可获取设备指南针方向，gyro.attitude.eulerAngles.z陀螺仪的方向角也可以获取，但两者获取值有一点差异，但问题不大，可通过算法优化。通过方向值调整空间锚点方向，从而达到虚拟世界的朝向同现实世界一致；
4. Evaluation: The
self-developed AR engine can realize the real AR scene and control the camera rotation. It can be used as an in-situ AR scene. The motion tracking SLAM function is only based on mobile phone hardware and cannot be implemented without the use of AI algorithms. Therefore, the mainstream AR engine SLAM function is implemented based on a combination of image recognition and fixed space anchor points. .
自研发AR引擎对手机的设备适配性达到MAX，只要有陀螺仪的手机都能使用，一些简单AR应用完全能满足要求。
Further reading materials:
Mobile phone model query that supports ARCore
8. Multi-engine integrated use plan
1. After introducing several mainstream AR engines, let’s make a comparison first:
基础功能支持：Vurfora ~ EasyAR > ARFoundation >> 自研发AR
Vurfora和EasyAR对ARCore和ARKit基础功能想图像识别定位做了一定的AI优化处理，效果理论下更好
高级功能支持：ARFoundation > Vurfora ~ EasyAR >> 自研发AR
ARFoundation只对ARCore和ARKit进行封装，对最新的高级功能支持最好
Vurfora和EasyAR底层封装，ARCore和ARKit的最新高级功能反而不能及时支持
URP项目支持：
支持：ARFoundation、Vurfora、自研发AR
不支持：EasyAR
设备适配性：自研发AR >> EasyAR > Vurfora ~ ARFoundation
EasyAR自研发了一些系统，能支持更多的设备
Vurfora和ARFoundation限于ARCore和ARKit支持限制
（PS：华为AR对华为手机有更多的适配支持）
Unity友好性：自研发AR > ARFoundation > Vurfora > EasyAR
ARFoundation本来就是Unity封装的，包可以直接下载使用，随时更新最新的
Vurfora也是Unity嵌入支持的
开发友好性：自研发AR >> EasyAR > ARFoundation > Vurfora
EasyAR官网提供完整的使用说明文和示例项目，并且可在社区随时询问开发问题，很友好
ARFoundation在Github上提供完整的示例项目，加上接口封装的不错，使用上还是比较友好
2. Each engine has its own advantages and disadvantages, try to combine the use of the plan as follows
2.1 However, because the control of each engine to the camera is encapsulated to the bottom layer and cannot be modified again, the simultaneous use of multiple engines is excluded.
2.2 根据手机情况，智能选择最合适的引擎使用，但由于主流AR引擎均基于ARCore和ARKit，并且支持的版本不一样，同时在项目中，会因为ARCore和ARKit版本不统一，但只能存在一个版本的，导致使用失败，方案排除；
2.3 主流AR引擎+自研发AR引擎方案，优先使用主流AR引擎，如判断主流AR引擎不支持，就切换使用自研发AR引擎，可行；
3. Extended function realization scheme:
AR shadow realization idea:
因为AR场景中，背景是现实场景，虚拟场景没有地面存在，3D对象需要显示阴影，就需要在3D对象下方一个透明的平面，然后将阴影渲染到平面上显示，要注意平面大小，如果过小，阴影会显示不全。提供一个阴影shader。
k9HyYTn0kGFFXLWiYD7o.png
tzophU1RHeWvxxrBDV1R.png
GPS positioning realization ideas:
方案一：Input.location是Unity提供的获取GPS定位的接口，自行编写换算算法后，便可使用；
SystemInfo.supportsLocationService 可检测手机是否支持GPS
Input.location.isEnabledByUser 可检测用户是否许可开启GPS
方案二：前文提到的Unity AR + GPS Location是有封装好的定位类，算法完整，可直接使用；
方案三：可从地图SDK（如百度地图、高德地图）获取定位信息，传到unity使用，相比Unity的GPS定位，添加了wifi定位和基站定位，更稳定精准；
Nine, AR engine selection and implementation summary
这次AR引擎选型研究遇到最大问题就是设备适配性，特别是Android设备，主流AR引擎依赖ARCore，ARCore依赖手机要有Google ARCore，现在相当一部分手机不支持Google ARCore，最新的高级手机都有不支持的，最终导致基于主流AR引擎开发的AR应用无法使用。
这点抖音做的很好，LandMarkAR实际使用情况大部分手机都支持，适配性不错，可惜未开放SDK，无法基于LandMarkAR开发。
国内第三方AR开发商很多，但很多都是商业合作方式代开发AR应用，面向开发者的第三方SDK提供商不多。
最终AR引擎选型为ARFoundation+自研发AR方案，AR高级功能由ARFoundation实现，适配性由自研发AR保底。
This article is reprinted at: https://gameinstitute.qq.com/community/detail/133606