JavaCV's Camera Practice No. 14: Mask Detection

Get into the habit of writing together! This is the 10th day of my participation in the "Nuggets Daily New Plan · April Update Challenge", click to view the details of the event .

Welcome to my GitHub

All original works of Xinchen (including supporting source code) are classified and summarized here: github.com/zq2599/blog…

Overview of this article

• This article is the fourteenth article of the "JavaCV Camera Actual Combat" series. As the title says, today's function is to detect whether the person in the camera is wearing a mask, and mark the detection result in the preview window in real time, as shown in the following figure:

• The whole processing process is as follows. The key to realizing mask detection is to submit the picture to the Baidu AI Open Platform, and then identify the location of the face in the local preview window according to the results returned by the platform, and whether the person is wearing a mask:

Issue notice in advance

• A typical problem of relying on cloud platforms to process business is the limited processing speed
• First of all, if the account you registered on the Baidu AI Open Platform is a personal type, the free interface calls will be limited to two times per second, and if it is an enterprise type account, the limit is ten times
• Secondly, after actual measurement, a face detection interface takes more than 300ms
• In the end, in fact, only two frames can be processed in one second. Such an effect can only be displayed in the preview window, and it can only be a slideshow effect.
• Therefore, this article is only suitable for the display of basic functions and cannot be used as a solution for actual scenarios.

About Baidu AI Open Platform

• In order to use the services of Baidu AI Open Platform, you need to complete some registration and application operations. For details, please refer to "The Simplest Face Detection (Free Call Baidu AI Open Platform Interface)"
• Now, if you have completed the registration and application of Baidu AI Open Platform, then you should have access_token available in your hand, so you can start coding now

Coding: Add dependency library

• This article continues to use the simple-grab-push project created by "One of JavaCV's Camera Practices: Basics"
• The first is to add okhttp and jackson dependencies to pom.xml, which are used for network requests and JSON parsing respectively:

<dependency>
	<groupId>com.squareup.okhttp3</groupId>
	<artifactId>okhttp</artifactId>
    <version>3.10.0</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.11.0</version>
</dependency>
复制代码

Coding: Encapsulate the code for requesting and responding to Baidu AI Open Platform

• 接下来要开发一个服务类，这个服务类封装了所有和百度AI开放平台相关的代码
• 首先，定义web请求的request对象FaceDetectRequest.java：

@Data
public class FaceDetectRequest {
    // 图片信息(总数据大小应小于10M)，图片上传方式根据image_type来判断
    String image;

    // 图片类型
    // BASE64:图片的base64值，base64编码后的图片数据，编码后的图片大小不超过2M；
    // URL:图片的 URL地址( 可能由于网络等原因导致下载图片时间过长)；
    // FACE_TOKEN: 人脸图片的唯一标识，调用人脸检测接口时，会为每个人脸图片赋予一个唯一的FACE_TOKEN，同一张图片多次检测得到的FACE_TOKEN是同一个。
    @JsonProperty("image_type")
    String imageType;

    // 包括age,expression,face_shape,gender,glasses,landmark,landmark150,quality,eye_status,emotion,face_type,mask,spoofing信息
    //逗号分隔. 默认只返回face_token、人脸框、概率和旋转角度
    @JsonProperty("face_field")
    String faceField;

    // 最多处理人脸的数目，默认值为1，根据人脸检测排序类型检测图片中排序第一的人脸（默认为人脸面积最大的人脸），最大值120
    @JsonProperty("max_face_num")
    int maxFaceNum;

    // 人脸的类型
    // LIVE表示生活照：通常为手机、相机拍摄的人像图片、或从网络获取的人像图片等
    // IDCARD表示身份证芯片照：二代身份证内置芯片中的人像照片
    // WATERMARK表示带水印证件照：一般为带水印的小图，如公安网小图
    // CERT表示证件照片：如拍摄的身份证、工卡、护照、学生证等证件图片
    // 默认LIVE
    @JsonProperty("face_type")
    String faceType;

    // 活体控制 检测结果中不符合要求的人脸会被过滤
    // NONE: 不进行控制
    // LOW:较低的活体要求(高通过率 低攻击拒绝率)
    // NORMAL: 一般的活体要求(平衡的攻击拒绝率, 通过率)
    // HIGH: 较高的活体要求(高攻击拒绝率 低通过率)
    // 默认NONE
    @JsonProperty("liveness_control")
    String livenessControl;

    // 人脸检测排序类型
    // 0:代表检测出的人脸按照人脸面积从大到小排列
    // 1:代表检测出的人脸按照距离图片中心从近到远排列
    // 默认为0
    @JsonProperty("face_sort_type")
    int faceSortType;
}
复制代码

• 其次，定义web响应对象FaceDetectResponse.java：

@Data
@ToString
public class FaceDetectResponse implements Serializable {
    // 返回码
    @JsonProperty("error_code")
    String errorCode;
    // 描述信息
    @JsonProperty("error_msg")
    String errorMsg;
    // 返回的具体内容
    Result result;

    @Data
    public static class Result {
        // 人脸数量
        @JsonProperty("face_num")
        private int faceNum;
        // 每个人脸的信息
        @JsonProperty("face_list")
        List<Face> faceList;

        /**
         * @author willzhao
         * @version 1.0
         * @description 检测出来的人脸对象
         * @date 2022/1/1 16:03
         */
        @Data
        public static class Face {
            // 位置
            Location location;
            // 是人脸的置信度
            @JsonProperty("face_probability")
            double face_probability;
            // 口罩
            Mask mask;

            /**
             * @author willzhao
             * @version 1.0
             * @description 人脸在图片中的位置
             * @date 2022/1/1 16:04
             */
            @Data
            public static class Location {
                double left;
                double top;
                double width;
                double height;
                double rotation;
            }

            /**
             * @author willzhao
             * @version 1.0
             * @description 口罩对象
             * @date 2022/1/1 16:11
             */
            @Data
            public static class Mask {
                int type;
                double probability;
            }
        }
    }
}
复制代码

• 然后是服务类BaiduCloudService.java，把请求和响应百度AI开放平台的逻辑全部集中在这里，可见其实很简单：根据图片的base64字符串构造请求对象、发POST请求（path是人脸检测服务）、收到响应后用Jackson反序列化成FaceDetectResponse对象：

public class BaiduCloudService {

    OkHttpClient client = new OkHttpClient();

    static final MediaType JSON = MediaType.parse("application/json; charset=utf-8");

    static final String URL_TEMPLATE = "https://aip.baidubce.com/rest/2.0/face/v3/detect?access_token=%s";

    String token;

    ObjectMapper mapper = new ObjectMapper();

    public BaiduCloudService(String token) {
        this.token = token;

        // 重要：反序列化的时候，字符的字段如果比类的字段多，下面这个设置可以确保反序列化成功
        mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
    }
    
    /**
     * 检测指定的图片
     * @param imageBase64
     * @return
     */
    public FaceDetectResponse detect(String imageBase64) {
        // 请求对象
        FaceDetectRequest faceDetectRequest = new FaceDetectRequest();
        faceDetectRequest.setImageType("BASE64");
        faceDetectRequest.setFaceField("mask");
        faceDetectRequest.setMaxFaceNum(6);
        faceDetectRequest.setFaceType("LIVE");
        faceDetectRequest.setLivenessControl("NONE");
        faceDetectRequest.setFaceSortType(0);
        faceDetectRequest.setImage(imageBase64);

        FaceDetectResponse faceDetectResponse = null;

        try {
            // 用Jackson将请求对象序列化成字符串
            String jsonContent = mapper.writeValueAsString(faceDetectRequest);

            //
            RequestBody requestBody = RequestBody.create(JSON, jsonContent);
            Request request = new Request
                    .Builder()
                    .url(String.format(URL_TEMPLATE, token))
                    .post(requestBody)
                    .build();
            Response response = client.newCall(request).execute();
            String rawRlt = response.body().string();
            faceDetectResponse = mapper.readValue(rawRlt, FaceDetectResponse.class);
        } catch (IOException ioException) {
            ioException.printStackTrace();
        }

        return faceDetectResponse;
    }
}
复制代码

• 服务类写完了，接下来是主程序把整个逻辑串起来

DetectService接口的实现

• 熟悉《JavaCV的摄像头实战》系列的读者应该对DetectService接口不陌生了，为了在整个系列的诸多实战中以统一的风格实现抓取帧-->处理帧-->输出处理结果这样的流程，咱们定义了一个DetectService接口，每种不同帧处理业务按照自己的特点来实现此接口即可（例如人脸检测、年龄检测、性别检测等）
• 先来回顾DetectService接口：

public interface DetectService {

    /**
     * 根据传入的MAT构造相同尺寸的MAT，存放灰度图片用于以后的检测
     * @param src 原始图片的MAT对象
     * @return 相同尺寸的灰度图片的MAT对象
     */
    static Mat buildGrayImage(Mat src) {
        return new Mat(src.rows(), src.cols(), CV_8UC1);
    }

    /**
     * 检测图片，将检测结果用矩形标注在原始图片上
     * @param classifier 分类器
     * @param converter Frame和mat的转换器
     * @param rawFrame 原始视频帧
     * @param grabbedImage 原始视频帧对应的mat
     * @param grayImage 存放灰度图片的mat
     * @return 标注了识别结果的视频帧
     */
    static Frame detect(CascadeClassifier classifier,
                        OpenCVFrameConverter.ToMat converter,
                        Frame rawFrame,
                        Mat grabbedImage,
                        Mat grayImage) {

        // 当前图片转为灰度图片
        cvtColor(grabbedImage, grayImage, CV_BGR2GRAY);

        // 存放检测结果的容器
        RectVector objects = new RectVector();

        // 开始检测
        classifier.detectMultiScale(grayImage, objects);

        // 检测结果总数
        long total = objects.size();

        // 如果没有检测到结果，就用原始帧返回
        if (total<1) {
            return rawFrame;
        }

        // 如果有检测结果，就根据结果的数据构造矩形框，画在原图上
        for (long i = 0; i < total; i++) {
            Rect r = objects.get(i);
            int x = r.x(), y = r.y(), w = r.width(), h = r.height();
            rectangle(grabbedImage, new Point(x, y), new Point(x + w, y + h), Scalar.RED, 1, CV_AA, 0);
        }

        // 释放检测结果资源
        objects.close();

        // 将标注过的图片转为帧，返回
        return converter.convert(grabbedImage);
    }

    /**
     * 初始化操作，例如模型下载
     * @throws Exception
     */
    void init() throws Exception;

    /**
     * 得到原始帧，做识别，添加框选
     * @param frame
     * @return
     */
    Frame convert(Frame frame);

    /**
     * 释放资源
     */
    void releaseOutputResource();
}
复制代码

• 再来看看本次实战中DetectService接口的实现类BaiduCloudDetectService.java，有几处要注意的地方稍后会提到：

@Slf4j
public class BaiduCloudDetectService implements DetectService {

    /**
     * 每一帧原始图片的对象
     */
    private Mat grabbedImage = null;

    /**
     * 百度云的token
     */
    private String token;

    /**
     * 图片的base64字符串
     */
    private String base64Str;

    /**
     * 百度云服务
     */
    private BaiduCloudService baiduCloudService;

    private OpenCVFrameConverter.ToMat openCVConverter = new OpenCVFrameConverter.ToMat();

    private Java2DFrameConverter java2DConverter = new Java2DFrameConverter();

    private OpenCVFrameConverter.ToMat converter = new OpenCVFrameConverter.ToMat();

    private BASE64Encoder encoder = new BASE64Encoder();

    /**
     * 构造方法，在此指定模型文件的下载地址
     * @param token
     */
    public BaiduCloudDetectService(String token) {
        this.token = token;
    }

    /**
     * 百度云服务对象的初始化
     * @throws Exception
     */
    @Override
    public void init() throws Exception {
        baiduCloudService = new BaiduCloudService(token);
    }

    @Override
    public Frame convert(Frame frame) {
        // 将原始帧转成base64字符串
        base64Str = frame2Base64(frame);

        // 记录请求开始的时间
        long startTime = System.currentTimeMillis();

        // 交给百度云进行人脸和口罩检测
        FaceDetectResponse faceDetectResponse = baiduCloudService.detect(base64Str);

        // 如果检测失败，就提前返回了
        if (null==faceDetectResponse
         || null==faceDetectResponse.getErrorCode()
         || !"0".equals(faceDetectResponse.getErrorCode())) {
            String desc = "";
            if (null!=faceDetectResponse) {
                desc = String.format("，错误码[%s]，错误信息[%s]", faceDetectResponse.getErrorCode(), faceDetectResponse.getErrorMsg());
            }

            log.error("检测人脸失败", desc);

            // 提前返回
            return frame;
        }

        log.info("检测耗时[{}]ms，结果：{}", (System.currentTimeMillis()-startTime), faceDetectResponse);

        // 如果拿不到检测结果，就返回原始帧
        if (null==faceDetectResponse.getResult()
        || null==faceDetectResponse.getResult().getFaceList()) {
            log.info("未检测到人脸");
            return frame;
        }

        // 取出百度云的检测结果，后面会逐个处理
        List<FaceDetectResponse.Result.Face> list = faceDetectResponse.getResult().getFaceList();
        FaceDetectResponse.Result.Face face;
        FaceDetectResponse.Result.Face.Location location;
        String desc;
        Scalar color;
        int pos_x;
        int pos_y;

        // 如果有检测结果，就根据结果的数据构造矩形框，画在原图上
        for (int i = 0; i < list.size(); i++) {
            face = list.get(i);

            // 每张人脸的位置
            location = face.getLocation();

            int x = (int)location.getLeft();
            int y = (int)location.getHeight();
            int w = (int)location.getWidth();
            int h = (int)location.getHeight();

            // 口罩字段的type等于1表示带口罩，0表示未带口罩
            if (1==face.getMask().getType()) {
                desc = "Mask";
                color = Scalar.GREEN;
            } else {
                desc = "No mask";
                color = Scalar.RED;
            }

            // 在图片上框出人脸
            rectangle(grabbedImage, new Point(x, y), new Point(x + w, y + h), color, 1, CV_AA, 0);

            // 人脸标注的横坐标
            pos_x = Math.max(x-10, 0);
            // 人脸标注的纵坐标
            pos_y = Math.max(y-10, 0);

            // 给人脸做标注，标注是否佩戴口罩
             putText(grabbedImage, desc, new Point(pos_x, pos_y), FONT_HERSHEY_PLAIN, 1.5, color);
        }

        // 将标注过的图片转为帧，返回
        return converter.convert(grabbedImage);
    }

    /**
     * 程序结束前，释放人脸识别的资源
     */
    @Override
    public void releaseOutputResource() {
        if (null!=grabbedImage) {
            grabbedImage.release();
        }
    }

    private String frame2Base64(Frame frame) {
        grabbedImage = converter.convert(frame);
        BufferedImage bufferedImage = java2DConverter.convert(openCVConverter.convert(grabbedImage));
        ByteArrayOutputStream bStream = new ByteArrayOutputStream();
        try {
            ImageIO.write(bufferedImage, "png", bStream);
        } catch (IOException e) {
            throw new RuntimeException("bugImg读取失败:"+e.getMessage(),e);
        }

        return encoder.encode(bStream.toByteArray());
    }
}
复制代码

• 上述代码有以下几点要注意：

1. 整个BaiduCloudDetectService类，主要是对前面BaiduCloudService类的使用
2. convert方法中，拿到frame实例后会转为base64字符串，用于提交到百度AI开放平台做人脸检测
3. 百度AI开放平台的检测结果中有多个人脸检测结果，这里要逐个处理：取出每个人脸的位置，以此位置在原图画矩形框，然后根据是否戴口罩在人脸上做标记，戴口罩的是绿色标记（包括矩形框），不戴口罩的是红色矩形框

主程序

• 最后是主程序了，还是《JavaCV的摄像头实战》系列的套路，咱们来看看主程序的服务类定义好的框架
• 《JavaCV的摄像头实战之一：基础》创建的simple-grab-push工程中已经准备好了父类AbstractCameraApplication，所以本篇继续使用该工程，创建子类实现那些抽象方法即可
• 编码前先回顾父类的基础结构，如下图，粗体是父类定义的各个方法，红色块都是需要子类来实现抽象方法，所以接下来，咱们以本地窗口预览为目标实现这三个红色方法即可：

• 新建文件PreviewCameraWithBaiduCloud.java，这是AbstractCameraApplication的子类，其代码很简单，接下来按上图顺序依次说明
• 先定义CanvasFrame类型的成员变量previewCanvas，这是展示视频帧的本地窗口：

protected CanvasFrame previewCanvas
复制代码

• 把前面创建的DetectService作为成员变量，后面检测的时候会用到：

    /**
     * 检测工具接口
     */
    private DetectService detectService;
复制代码

• PreviewCameraWithBaiduCloud的构造方法，接受DetectService的实例：

    /**
     * 不同的检测工具，可以通过构造方法传入
     * @param detectService
     */
    public PreviewCameraWithBaiduCloud(DetectService detectService) {
        this.detectService = detectService;
    }
复制代码

• 然后是初始化操作，可见是previewCanvas的实例化和参数设置，还有检测、识别的初始化操作：

    @Override
    protected void initOutput() throws Exception {
        previewCanvas = new CanvasFrame("摄像头预览", CanvasFrame.getDefaultGamma() / grabber.getGamma());
        previewCanvas.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        previewCanvas.setAlwaysOnTop(true);

        // 检测服务的初始化操作
        detectService.init();
    }
复制代码

• 接下来是output方法，定义了拿到每一帧视频数据后做什么事情，这里调用了detectService.convert检测人脸并识别性别，然后在本地窗口显示：

    @Override
    protected void output(Frame frame) {
        // 原始帧先交给检测服务处理，这个处理包括物体检测，再将检测结果标注在原始图片上，
        // 然后转换为帧返回
        Frame detectedFrame = detectService.convert(frame);
        // 预览窗口上显示的帧是标注了检测结果的帧
        previewCanvas.showImage(detectedFrame);
    }
复制代码

• 最后是处理视频的循环结束后，程序退出前要做的事情，先关闭本地窗口，再释放检测服务的资源：

    @Override
    protected void releaseOutputResource() {
        if (null!= previewCanvas) {
            previewCanvas.dispose();
        }

        // 检测工具也要释放资源
        detectService.releaseOutputResource();
    }
复制代码

• 每一帧耗时太多，所以两帧之间就不再额外间隔了：

    @Override
    protected int getInterval() {
        return 0;
    }
复制代码

• 至此，功能已开发完成，再写上main方法，代码如下，请注意token的值是前面在百度AI开放平台取得的access_token：

    public static void main(String[] args) {
        String token = "21.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxx.xxxxxxxxxx.xxxxxx-xxxxxxxx";
        new PreviewCameraWithBaiduCloud(new BaiduCloudDetectService(token)).action(1000);
    }
复制代码

• 至此，代码写完了，准备好摄像头开始验证，群众演员为了免费盒饭已经在寒风中等了很久啦

验证

• 运行PreviewCameraWithBaiduCloud的main方法，请群众演员出现在摄像头前面，此时不戴口罩，可见人脸上是红色字体和矩形框：

• 让群众演员戴上口罩，再次出现在摄像头前面，这次检测到了口罩，显示了绿色标注和矩形框：

• In the actual experience, since there are only two frames per second at most, it is completely a slideshow effect when displayed in the preview window, which is terrible...
• This blog used two photos of the extras, so he took away two lunch boxes, Xin Chen was very distressed...
• So far, the mask detection function based on JavaCV and Baidu AI open platform has been completed. I hope you will continue to pay attention to the "JavaCV Camera Actual Combat" series, and the actual combat will be more exciting after that.

Welcome to the Nuggets: Programmer Xin Chen

On the road of learning, you are not alone, Xinchen Original will accompany you all the way...