Implementation principle of AR red envelope on Android
Not long ago, Alipay launched the function of AR red envelopes, which is all the rage. Recently, such a demand has just been fulfilled. Hereby, we organize and share them in order to learn from each other and make progress together. In view of the company's possible commercial use in the future, I will not post the source code here, just briefly talk about the implementation ideas and principles.
Without further ado, let’s start with the flow chart:
saffron
1. Initialize the camera
The camera class android.hardware.Camera, but unfortunately it is outdated. Google launched a more powerful android.hardware.camera2package after Android 5.0, but in order to be compatible with lower versions, the android.hardware.Cameraclass is still used here (interested students can learn this article to understand the implementation of camera2 through Android Camera2 Photography Introduction Way).
Camera.open()turn on the camera;Camera.getParameters()Get the parameter class and set it. Without this step, the preview image of the camera will become horizontal and blurry.
Contents that need to be set:parameters.setPictureSizepicture resolution,parameters.setPreviewSizepreview image resolution, these two need to be calculated according to the list supported by the device. Theparameters.setPreviewFormatpreview image transcoding format andparameters.setJpegQualityphoto quality will not be described here. It is not necessary to take pictures here and can be ignored.parameters.setFocusModeFocus mode, continuous Focusing needs to add thecamera.cancelAutoFocus()camera.setDisplayOrientationorientation of the preview screen, which is rotated by 90 here.
Remember permissions (dynamic application is required after Android 6.0.):
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.FLASHLIGHT" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />
<uses-feature android:name="android.hardware.camera.flash" />2. Binding surfaceView
- Get the surfaceHolder reference after
getHolder()finding the surfaceView; holder.setTypeset type;holder.addCallbackadd callback;Camera.setPreviewDisplay(holder)Pass surfaceHolder to Camera;Camera.startPreview()Open the preview image.
3. Get a frame preview image
Camera.setOneShotPreviewCallback(PreviewCallback cb)方法可以获取一帧预览图像，数据会返回到PreviewCallback 回调中的 onPreviewFrame(byte[] data,Camera camera)方法中，data即为帧数据。
四、预处理图像
首先明确一点，处理图像和接下来的其他耗时计算都要放到子线程中，不要问我为什么，编译器会告诉你答案。onPreviewFrame方法中的data并不能直接直接转为bitmap，需要先进行转码：
Camera.Size size = camera.getParameters().getPreviewSize();
YuvImage image = new YuvImage(data, ImageFormat.NV21, size.width, size.height, null);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
image.compressToJpeg(new Rect(0, 0, size.width, size.height), 80, stream);
Bitmap bmp = BitmapFactory.decodeByteArray(stream.toByteArray(), 0, stream.size());五、奇数帧？偶数帧？
为什么要分奇数偶数呢？当做好相机的准备工作后，要藏红包，实际上需要解决的就是排除不需要的图像，首先要做到的就是舍弃用户移动时获取的图像，最先的想法是利用加速度传感器，但是随后发现手机匀速移动的时候是没有加速度的，而这时获取的图像仍需舍弃。后来找到的解决方案是：不停获取帧图像，比较相邻两帧的相似度，当相似度达到某个阈值时视为相似，连续数次比较皆为相似可知用户所对位置即用户想要藏红包的地方，将此时获取的帧图保存即可。
- 设置一个计数器记录获取帧图的次数，计数器%2即可区分奇偶；
- 奇数帧存到图1，偶数帧存图2，比较图1图2；
- 不足两图、比较不相似就获取新的一帧，即可实现不停的比较相邻两帧图像。
六、相似？
比较相似的方法网上有挺多，有直接拿像素比较的，有处理成灰度图像比较的，有转成哈希值比较的，比较方法不同得到的数值也不同，所要设定的阈值也随之改变，具体看需求能达到目的就好。细心的同学会发现支付宝在藏红包的时候并不是所有地方都可以藏，有些没有意义、过度单调的地方是没办法藏的，这里实现的方式是计算图片像素“丰富度”（缺少图像学知识暂且这么叫吧）。得到图片所有像素点的值，去除重复，所得到的所有不同像素值的个数越小说明图片颜色越“单调”，然后根据这个个数值设置阈值进行判断。
七、确定红包位置
得到满足要求的帧图后，停止获取帧图，停止相机预览Camera.stopPreview()，将图片直接上传服务器或者转成哈希值传给服务器，也可以同时传些位置坐标等参数，这个看具体需求，这些数据即为所藏红包的“指纹”。
找红包
找红包相对简单，相同的逻辑不在赘述，只需要：
- 获取已藏红包的位置图片，
- 与获取的预览帧图比较，
- 不相似继续取帧图，相似停止取帧，即找到红包。
可以根据需求增加位置坐标范围判断和用户身份判断。
一点思考
首先我们对比下扫描二维码和扫描AR红包的实现原理：
不难发现，整个扫描流程十分相似。其实在我看来，AR红包可以理解为是一个特殊的“二维码”，只不过这个“二维码”上不再是特殊的几何图形，而是一张图片。支付宝“扫一扫”中左边是“二维码”右边是“AR”，有力地支持了我的逻辑。那么按照这个逻辑再来看整个AR红包，可以发现“藏红包”其实就是一个生成“二维码”的过程，“找红包”自然就是扫描“二维码”了。
那么AR红包又与二维码有什么区别？
我们知道二维码是按照一定规律生成的特殊几何图形，可以进行编码解析，即二维码中是内含信息的；而AR红包实际上只是一张图片，不包含信息。这也是为什么支付宝要通过红包地图和用户的方式将AR红包区分开的原因，否则如果两个用户在同一个位置藏了红包，扫描时将无法区分它们。既然如此那为什么还要有AR红包的存在呢？
AR红包相对于二维码的优缺点
优点：
- AR红包藏匿于现实生活中。不同于二维码：“我是一个二维码，你们快来扫我呀O(∩_∩)O~~”，只要你不公开你的红包位置图，没人知道这有红包，当然你也可以让部分人看到。妈妈再也不用担心儿子的私房钱被老婆发现了！
- AR red envelopes come from real life. Therefore, it saves the trouble of posting "QR codes" and replacing "QR codes" , and also reduces the pollution caused by QR codes to the environment. Just imagine: Today's premiere of "King Kong" scans "Empire State Building" to get a movie ticket, and scans "CCTV Big Pants" to watch today's news is not very cool~
- AR red envelopes can be used multiple times in one place. The same "CCTV pants", CCTV scan is a news network, NetEase scan is NetEase headlines, UC scan is UC shock.
- Good user experience. Take a chestnut: Compared with punching the card by scanning the company logo and punching the card by scanning the QR code on the company logo, I choose the former.
shortcoming:
- It does not contain information by itself, and needs other parameters to assist. Without location coordinates and user information, AR red packets can hardly function.
- One cannot step into the same river twice. The places where you hide your red envelopes may change over time and can no longer be reproduced, such as a broken piggy bank, yesterday's sunset, and a young you.
- The algorithm is not perfect. The AR red envelope has just appeared, and there is no unified algorithm. The results of each comparison algorithm vary greatly according to different needs. The recognition effect and efficiency are not perfect, and it cannot be instantly recognized like a QR code. It needs to be further improved.
In the process of realizing the AR red envelope, I deeply felt the lack of knowledge of image processing. This article should be summarized. Some simple opinions will inevitably lead to negligence and omissions.
refer to:
Android
QR code scanning development (1): Implementation ideas and principles