ID card recognition (1) - front and back of ID card and head portrait detection

foreword

1. This is a handheld identification project, which belongs to the scope of image recognition. It does not need a handheld identity scanner, but only detects whether there is an ID card in the current video or camera, and then performs related processing.
2. This project is part of a big project I did before. The libraries used in the project include OpenCV, Boost, and the deep learning framework of Caffe. Some related technologies used include face detection and face recognition in image processing. Face verification, digital recognition, Chinese character recognition, and voice recognition may be added in the future when there is time, or related operations corresponding to voiceprint recognition.
3. My project environment is Ubuntu, Qt Creator 5.9.OpenCV3.3, using Caffe to train VGG, VGG_Face, OCR, etc.

ID card detection

1. Resource preparation
(1) I wrote a crawler and downloaded a lot of images with ID cards from the Internet. These identities are public or randomly generated ID cards for demonstration purposes, so there should be no personal privacy Right, if there is any infringement, please private message me, the following is the relevant image of the ID card I downloaded:
insert image description here
(2) use LabelImage to label data

(3) use Caffe SSD's VGGNet training model, how to train and debug parameters, you can see me before About how Caffe_SSD trains the model.
2. Use the model to detect the image
(1) Import the following three files into the project

(2) Write the code for recognition.

void MainWindow::detectionId(Mat &input, string model_file,string model_text_file,string label_file)
{
    vector<String> objNames = readLabels(label_file);

    Ptr<dnn::Importer> importer;
    try
    {
        importer = createCaffeImporter(model_text_file, model_file);
    }
    catch (const cv::Exception &err)
    {
        cerr << err.msg << endl;
    }
    Net net;
    importer->populateNet(net);
    importer.release();

    Mat input_image = preprocess(input);
    Mat blobImage = blobFromImage(input_image);

    net.setInput(blobImage, "data");
    Mat detection = net.forward("detection_out");
    Mat detectionMat(detection.size[2], detection.size[3], CV_32F, detection.ptr<float>());
    float confidence_threshold = 0.2;
    for (int i = 0; i < detectionMat.rows; i++)
    {
        float confidence = detectionMat.at<float>(i, 2);
        if (confidence > confidence_threshold)
        {
            size_t objIndex = (size_t)(detectionMat.at<float>(i, 1));
            float tl_x = detectionMat.at<float>(i, 3) * input.cols;
            float tl_y = detectionMat.at<float>(i, 4) * input.rows;
            float br_x = detectionMat.at<float>(i, 5) * input.cols;
            float br_y = detectionMat.at<float>(i, 6) * input.rows;

            Rect object_box((int)tl_x, (int)tl_y, (int)(br_x - tl_x), (int)(br_y - tl_y));
            rectangle(input, object_box, Scalar(i*10, 0, 255), 2, 8, 0);
            putText(input, format("%s", objNames[objIndex].c_str()), Point(tl_x, tl_y), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(255, 0, 0), 2);
        }
    }

}

Mat MainWindow::getMean(size_t &w, size_t &h)
{
    Mat mean;
    vector<Mat> channels;
    for (int i = 0; i < 3; i++)
    {
        Mat channel(h, w, CV_32F, Scalar(meanValues[i]));
        channels.push_back(channel);
    }
    merge(channels, mean);
    return mean;
}

Mat MainWindow::preprocess(Mat &frame)
{
    Mat preprocessed;
    frame.convertTo(preprocessed, CV_32F);
    cv::resize(preprocessed, preprocessed, Size(width, height));
    Mat mean = getMean(width, height);
    subtract(preprocessed, mean, preprocessed);
    return preprocessed;
}

3. Recognition results
(1) Open the image

insert image description here
(2) Detect the ID card in the current image and the avatar on the ID card.
The reverse side and the head portrait

On the front side, this image has an extra frame.

epilogue

1. This is the ID card detection and recognition I did with the vgg of caffe_ssd for deep learning, and it can also be done with the traditional opencv method. The traditional method can refer to the license plate recognition project.
2. Regarding the source code of the entire project, bugs when running the program, or ideas on how to optimize, you can join this group (487350510) to discuss and learn from each other. 3. ID card edge
detection and extraction can be transferred to this blog .
4. Yolov5 training can be transferred to this blog .
5.yolov5 C++ OpenCV DNN reasoning and the final model are uploaded. If you want to try it, you can turn to this blog .