AR (Augmented Reality) technology can seamlessly connect virtual digital information with real world information, and bring virtual objects into reality to interact with users. With its rich and real interactive experience, AR technology has become a marketing tool for major brands.
Today, let's briefly sort out what kind of technical principle "AR" is, and provide some basic functions and code implementations for front-end students who want to understand AR technology. Let's take a look~
This article is suitable for readers who do not know much about AR technology
Composition of AR effects
As can be seen from the above video, the main effect of AR is "integration of reality and reality", so three technical parts are essential:
- Camera picture - presents the real reality picture, which exists as a "real" part. Generally speaking, it is enough to turn on the camera of the user equipment (in special cases, it can also be other media such as video stream).
- Virtual Information - Presents the virtual content of the simulation, existing as a "virtual" part. Generally speaking, there is a logical connection with real content, and we will try our best to make it real, interesting or highly enjoyable. This is also the most attractive part of AR technology. Commonly used 3D models, transparent special effect videos, ordinary videos, volumetric videos, pictures, etc. as media.
- AR Image Algorithm - The core part of AR technology, used to associate, track or fuse real content with virtual content, the purpose is to bind the two, rather than appearing or running independently. Only with the ability of "virtual-real fusion algorithm" can it show the texture of "augmented reality".
Camera Technology
In principle, displaying the camera screen is an independent and platform-related technology, and Android/iOS/Web/applets are different.
Android needs to use CameraManager to obtain the Camera data stream CameraCaptureSession, and use TextureView to present the Camera screen.
iOS needs to use AVCaptureDevice to obtain the Camera data stream AVCaptureSession, and use AVCaptureVideoPreviewLayer to present the Camera screen.
On the Web, you need to use mediaDevices to obtain the Camera data stream MediaStream, and use the video tag to present the Camera screen.
The applet can use the camera component to directly present the camera screen. To obtain the Camera data stream, you need to use CameraContext.onCameraFrame .
Let's briefly summarize it in tabular form:
| platform |
control API |
UI |
data flow |
|---|---|---|---|
| Android | CameraManager | TextureView | CameraCaptureSession |
| iOS | AVCaptureDevice | AVCaptureVideoPreviewLayer | AVCaptureSession |
| Web | mediaDevices | video | MediaStream |
| WeChat applet | CameraContext | camera | CameraContext.onCameraFrame |
As can be seen from the table, there are three technical points in the technology of the Camera part:
- API for Camera hardware control - open, close, flash, focus, etc.
- The UI that shows the Camera screen - presented to the user.
- Camera data flow - this is the most important part. We need to obtain the bitmap of the current frame from the data flow (which can be simply understood as a screenshot), and give it to the AR image algorithm for calculation to obtain a 4x4 tracking matrix . And this tracking matrix is used to achieve the effect of virtual and real fusion.
virtual information technology
In order to create a more realistic virtual-reality fusion effect, the most important feature of the virtual part is to have a sense of 3D, otherwise a flat 2d UI will not be able to create a sense of integration with the real content. Therefore, the commonly used UI rendering technology will basically not be used in AR content.
We usually use 3D rendering technology, which is widely used in games, simulation, film production and other fields, and the common solution to realize this technology is a rendering engine, or a game engine.
Unlike Camera technology, usually rendering engines can support cross-platform. For example, some common game engines: Unity3D, Unreal, Frostbite, CE, etc.
Especially Unity3D, in the AR field of developing App (Android/iOS), Unity3D is the first choice for most people. So when you search for AR technology on the Internet, you can see many articles related to the Unity3D engine.
Unfortunately, although Unity3D supports the Web platform, few AR algorithms are adapted to the Web platform, which leads to the inability of the AR algorithm to be used on the Web platform. Then if you only use the rendering capability of Unity3D, it will be particularly cumbersome (large resources and slow loading), and there are few related plug-ins. And for front-end development, it is necessary to learn a new language (C#) and platform (Unity3D), and the cost becomes very high.
So on the applet/web platform, is there a rendering engine similar to Unity3D?
Although there are many on the market, in all aspects, they are not as good as Unity3D. Mainly, Three.js may be leading in popularity and market share, but Babylon.js is not far behind (and better in some ways).
For front-end development, 3D rendering is a completely different technical field. At present, there are many people who know about it, but very few are good at and proficient in it.
| platform |
Common Rendering Engines |
|---|---|
| Android | Unity3D、Filament |
| iOS | Unity3D、SceneKit |
| Web | Three.js、Babylon.js |
| WeChat applet | threejs-miniprogram、three-platformize |
Next, we will respectively demonstrate some of the most basic functions and code implementations when developing AR effects on the Web and applet platforms.
WebAR Development
turn on the camera
// 打开后置摄像头
const stream = await navigator.mediaDevices.getUserMedia({
video: { ideal: "environment" }
});
// 呈现Camera画面
const video = document.createElement("video");
video.srcObject = stream;
document.body.appendChild(video);Important restriction: The webpage needs to be opened with the https protocol or the localhost domain name, so that the navigator.mediaDevices object exists, otherwise it is undefined.
Render the 3D scene
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer({ alpha: true });
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);
renderer.domElement.style = "position:absolute;left:0;top:0;width:100%;height:100%";
const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);
cube.positiin.z = -5;
function animate() {
renderer.render(scene, camera);
requestAnimationFrame(animate);
};
animate();AR image algorithm achieves virtual and real fusion effect
const canvas = document.createElement("canvas");
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext("2d");
cube.matrixAutoUpdate = false;
cube.frustumCulled = false;
cube.onBeforeRender = () => {
// 获取Camera当前一帧画面数据
ctx.drawImage(video, 0, 0);
const cameraFrame = ctx.getImageData(0, 0, canvas.width, canvas.height);
// 利用AR图像算法，输入Camera画面数据，计算出追踪矩阵，再赋值给3D内容。
// 这样，真实内容和虚拟内容，就融合在了一起。
const trackMatrix = SomeArAlgorithm.tracking(cameraFrame);
cube.matrix.copy(trackMatrix);
};The above sample code is for reference only, and can provide the basic implementation idea of the application layer of WebAR technology. Actual available code reference: webAR development example
Small program AR development
turn on the camera
<camera resolution="high" frame-size="medium"></camera>
<canvas type="webgl" id="webgl"></canvas>Render the 3D scene
import { createScopedThreejs } from "threejs-miniprogram";
Page({
onReady() {
wx.createSelectorQuery()
.select('#webgl')
.node()
.exec((res) => {
const canvas = res[0].node;
const THREE = createScopedThreejs(canvas);
this.init3d(THREE);
});
},
init3d(THREE) {
const scene = new THREE.Scene();
const { windowWidth, windowHeight } = wx.getSystemInfoSync();
const camera = new THREE.PerspectiveCamera(75, windowWidth / windowHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer({ alpha: true });
renderer.setSize(windowWidth, windowHeight);
const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);
cube.positiin.z = -5;
function animate() {
renderer.render(scene, camera);
requestAnimationFrame(animate);
};
animate();
}
});AR image algorithm achieves virtual and real fusion effect
const cameraCtx = wx.createCameraContext();
const listener = cameraCtx.onCameraFrame((cameraFrame) => {
// 利用AR图像算法，输入Camera画面数据，计算出追踪矩阵，再赋值给3D内容。
// 这样，真实内容和虚拟内容，就融合在了一起。
const trackMatrix = SomeArAlgorithm.tracking(cameraFrame);
cube.matrix.copy(trackMatrix);
cube.matrixAutoUpdate = false;
});
listener.start();Similarly, the above sample code is for reference only, and provides you with the basic implementation idea of the AR technology application layer of the applet. Actual available code reference: https://developers.weixin.qq.com/s/aAUxADmW7Nzs
Kivicube platform
In the above example, the AR content is just a very simple Cube, meaningless. In real commercial projects, we also need to adapt to a lot of details and functions.
for example:
- Multiple types of material support, such as 3D models, ordinary videos, transparent special effects videos, sprites, pictures, skyboxes, music, etc.;
- Simple and convenient scene construction (just drag and drop);
- Multi-browser compatibility of camera function;
- Multi-platform compatibility of rendering effects;
- Real-time preview adjustment of model materials;
- Optimization of rendering effect;
- etc.
For most front-end developers, the handling of these problems has entered a new field that has never been touched, and the learning time and energy required are very high.
So now that low-code platforms are popular, is there such a platform that can improve development efficiency and quickly produce results?
The free AR online editor - Kivicube - https://www.kivicube.com , can roughly meet everyone's needs.
Kivicube not only has a visual low-code construction platform, but also provides a simple and flexible API for developers to quickly realize demand interaction.
WeChat applet
On the WeChat applet side, the platform provides a Kivicube applet AR plug-in , which is directly used by developers in the form of components.
The component kivicube-scene can directly open the AR scene created on the Kivicube platform. It eliminates the need for developers to deal with Camera, 3D rendering, AR algorithms, multi-type material support, and other related work.
Since it is in the form of components, it is natural that if there are special UI and interaction requirements (which are not supported by the Kivicube platform), developers can also implement them by themselves.
Plug-ins and component access methods
Before using the plug-in, you must first add the plug-in in "Settings-Third Party Settings-Plug-in Management" in the applet management background. Developers can log in to the applet management background, find and add plugins by appid [wx3bbab3920eabccb2].
After that, you can add the declaration configuration of the plug-in in the app.json configuration file of the applet, as follows:
{
...
"plugins": {
"kivicube": {
"version": "1.6.6",
"provider": "wx3bbab3920eabccb2"
}
}
...
}
再在需要接入AR场景的页面中，配置好kivicube-scene组件，如下所示：
{
"usingComponents": {
"kivicube-scene": "plugin://kivicube/kivicube-scene"
}
}Then, you can use the component directly in the wxml of the page.
<kivicube-scene
class="kivicube"
scene-id="9vR08tpLesfKVWs2XbbnNKSsX3JqpaAp"
/>
.kivicube {
display: block;
width: 100vw;
height: 100vh;
}Through a few short lines of code above, we have realized an AR function. It can be seen that the advantages brought by the Kivicube platform are obvious.
The component needs to specify the opened scene through the sceneId attribute, and the scene Id needs to be obtained on the Kivicube platform.
After making and saving the scene on the Kivicube platform , click the "Share" button in the upper right corner, and then click "Copy Link". The last forward slash in the link, followed by a string of garbled characters, is the scene id. For example, the copied link is: https://www.kivicube.com/scenes/9vR08tpLesfKVWs2XbbnNKSsX3JqpaAp, then 9vR08tpLesfKVWs2XbbnNKSsX3JqpaAp is the scene id.
For a more detailed tutorial, please refer to: Kivicube Yuque
Web
We are preparing the AR plug-in capability on the web side to make the development of AR on the web platform as simple and fast, and we will meet you in the near future!
This is the end of today's explanation on the principle of "AR" technology. Welcome all front-end students to try it out and give us valuable suggestions~
At the same time, everyone is welcome to register and use the Kivicube platform for free, and use the low-code editor to quickly create your own AR scenes.