OpenCV combat (29) - video object tracking

0. Preface

We have learned how to track the motion of points and pixels in an image sequence . But in most applications, it is usually required to track specific moving objects in the video. An object of interest is first identified, which must then be tracked in the video sequence. Tracking an object in video is challenging due to the many visual changes an object undergoes as it evolves in the scene, viewpoint and lighting changes, non-rigid motion, occlusions, etc.
In this section, we describe some OpenCVobject tracking algorithms implemented in the library. We will implement a general framework to easily replace the algorithm. We can contrast this implementation with methods for object tracking based on histograms computed from integral images.

1. Track objects in the video

Object tracking problems usually assume that no prior knowledge about the objects to be tracked is available. Therefore, tracking is initiated by identifying objects in the frame, and tracking must start with object identification. The initial identification of an object is achieved by specifying a bounding box, and the tracker module aims to re-identify this object in subsequent frames.
Thus, the class OpenCVthat defines the object tracking framework cv::Trackerhas two main methods. The first is initthe method used to define the initial object bounding box; the second is updatethe method, which outputs a new bounding box given a new frame. Both methods accept a frame (i.e. cv::Matinstance) and a bounding box (i.e. cv::Rect2Dinstance) as parameters; in the first method, the bounding box is the input parameter, while for the second method the bounding box is the output parameter.

(1) To test the proposed object tracking algorithm, we use the video processing framework introduced in the Video Sequence Processing section. We define a frame handling subclass that VideoProcessorthe class will call when each frame of the image sequence is received, which has the following properties:

class VisualTracker : public FrameProcessor {
    
    
    cv::Ptr<cv::Tracker> tracker;
    cv::Rect2d box;
    bool reset;
    public:
        // 指定要使用的追踪器的构造函数
        VisualTracker(cv::Ptr<cv::Tracker> tracker) : reset(true), tracker(tracker) {
    
    }

(2)reset The property is set whenever the tracker is restarted by specifying the bounding box of the new object true; it is setBoundingBoxthe method used to store the position of the new object:

        // 初始化追踪器，设定边界框
        void setBoundingBox(const cv::Rect2d &bb) {
    
    
            box = bb;
            reset = true;
        }

(3) The callback method for processing each frame simply calls the corresponding method of the tracker and then draws the new computed bounding box on the frame to be displayed:

        // 回调处理方法
        void process(cv::Mat &frame, cv::Mat &output) {
    
    
            if (reset) {
    
    
                reset = false;
                tracker->init(frame, box);
            } else {
    
    
                tracker->update(frame, box);
            }
            // 绘制边界框
            frame.copyTo(output);
            cv::rectangle(output, box, cv::Scalar(255, 255, 255), 2);
        }

(4) To demonstrate how to track objects using VideoProcessorand FrameProcessorinstances, we use OpenCVthe Median Stream Tracker defined in :

int main() {
    
    
    // 创建视频处理实例
    VideoProcessor processor;
    // 生成文件名
    std::vector<std::string> imgs;
    std::string prefix = "test1/test_";
    std::string ext = ".png";
    // 添加用于追踪的图像名称
    for (long i=3040; i<3076; i++) {
    
    
        std::string name(prefix);
        std::ostringstream ss; ss << std::setfill('0') << std::setw(3) << i; name += ss.str();
        name += ext;
        std::cout << name << std::endl;
        imgs.push_back(name);
    }
    // 创建特征追踪器实例
    cv::Ptr<cv::TrackerMedianFlow> ptr = cv::TrackerMedianFlow::create();
    VisualTracker tracker(ptr);
    // VisualTracker tracker(cv::TrackerKCF::createTracker());
    // 打开文件
    processor.setInput(imgs);
    // 设置帧处理器
    processor.setFrameProcessor(&tracker);
    processor.displayOutput("Tracked object");
    // 定义帧率
    processor.setDelay(50);
    // 指定原始对象位置
    cv::Rect bb(1150, 150, 330, 330);
    tracker.setBoundingBox(bb);
    // 开始追踪
    processor.run();
}

(5) The first bounding box identifies a Mr. Wolf in the test image sequence, which is then automatically tracked in subsequent frames:

test frame
(6) But as the video plays, the tracker will inevitably make mistakes. The accumulation of these small errors causes the tracker to slowly drift away from the true target position:

object tracking

(7) Eventually, the tracker will lose track of the object. The ability of a tracker to track objects for a long time is the most important criterion to characterize the performance of an object tracker.

2. The algorithm principle of the median flow tracker

In the previous section, we learned how to use the generic cv::Trackerclass to track objects in an image sequence. We chose to perform tracking using the Median Flow tracker algorithm, which is a simple but effective way of tracking textured objects, as long as the objects are not moving too fast and are not too heavily occluded.
The median flow tracker is based on feature point tracking. First define a grid of points on the object to be tracked, we can optionally use the operators described in Detecting Interest Points to detect points of interest on the object. FASTHowever, using points at predefined locations has a number of advantages:

Save time by avoiding calculating points of interest
Guaranteed that a sufficient number of points are available for tracking
Make sure the points are nicely distributed across the object

The median stream implementation uses 10x10a grid with points by default:

initial grid
Use the feature tracking algorithm described in the section Tracking Feature Points in a VideoLukas-Kanade to track each point of the mesh in the next frame:

Tracked Grid Points
The median flow algorithm estimates the error in tracking these points. For example, the error can be estimated by calculating the sum of the absolute pixel differences in a window around the point's initial position and the tracked position, using the function cv::calcOpticalFlowPyrLKto conveniently calculate and return the error. Another error metric that can be used by the median flow algorithm is the forward-backward error, after tracking the points between the current frame and the next frame, the points are tracked backwards at the new position to check whether they will return to The difference between their original position in the initial image, the newly obtained front-back position and the initial position is the tracking error.
Once the tracking error is computed for each point, only 50%the points with the smallest error are considered, and this set of points is then used to compute the new position of the bounding box in the next image. The points vote for a displacement value, and the median of these displacements is kept. For scale change, the points are considered in pairs, and the ratio of the distances of two points in the initial frame to the next frame is estimated. Again, the median of these ratios is finally applied.
The median flow tracker is a visual object tracker based on feature point tracking. Another solution is a method based on template matching. You can refer to the section on matching local template methodsKernelized Correlation Filter . The kernel correlation filter ( , KCF) algorithm is a classic template-based The matching algorithm OpenCVcan be implemented using cv::TrackerKCFthe class in :

VisualTracker tracker(cv::TrackerKCF::createTracker());

Essentially, the algorithm uses the target object's bounding box as a template to search for a new object location in another image. Usually it can be calculated by using simple correlation, but KCFusing a method based on Fourier transform. In the section on image filtering , we learned that according to signal processing theory, correlating templates on images is equivalent to Simple image multiplication. This can greatly speed up the recognition of matching windows in another image and KCFis therefore one of the best performing trackers.

3. Complete code

videoprocessor.hFor the complete code of the header file ( ), refer to the video sequence processing section. The visualTracker.hcomplete code of the header file ( ) is as follows:

#if !defined FTRACKER
#define FTRACKER

#include <string>
#include <vector>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/features2d.hpp>
#include <opencv2/tracking/tracker.hpp>

#include "videoprocessor.h"

class VisualTracker : public FrameProcessor {
    
    
    cv::Ptr<cv::Tracker> tracker;
    cv::Rect2d box;
    bool reset;
    public:
        // 指定要使用的追踪器的构造函数
        VisualTracker(cv::Ptr<cv::Tracker> tracker) : reset(true), tracker(tracker) {
    
    }
        // 初始化追踪器，设定边界框
        void setBoundingBox(const cv::Rect2d &bb) {
    
    
            box = bb;
            reset = true;
        }
        // 回调处理方法
        void process(cv::Mat &frame, cv::Mat &output) {
    
    
            if (reset) {
    
    
                reset = false;
                tracker->init(frame, box);
            } else {
    
    
                tracker->update(frame, box);
            }
            // 绘制边界框
            frame.copyTo(output);
            cv::rectangle(output, box, cv::Scalar(255, 255, 255), 2);
        }
};

#endif

The complete code of the main function file ( oTracker.cpp) is as follows:

#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <opencv2/video/tracking.hpp>

#include "visualTracker.h"

int main() {
    
    
    // 创建视频处理实例
    VideoProcessor processor;
    // 生成文件名
    std::vector<std::string> imgs;
    std::string prefix = "test1/test_";
    std::string ext = ".png";
    // 添加用于追踪的图像名称
    for (long i=3040; i<3076; i++) {
    
    
        std::string name(prefix);
        std::ostringstream ss; ss << std::setfill('0') << std::setw(3) << i; name += ss.str();
        name += ext;
        std::cout << name << std::endl;
        imgs.push_back(name);
    }
    // 创建特征追踪器实例
    cv::Ptr<cv::TrackerMedianFlow> ptr = cv::TrackerMedianFlow::create();
    VisualTracker tracker(ptr);
    // VisualTracker tracker(cv::TrackerKCF::createTracker());
    // 打开文件
    processor.setInput(imgs);
    // 设置帧处理器
    processor.setFrameProcessor(&tracker);
    processor.displayOutput("Tracked object");
    // 定义帧率
    processor.setDelay(50);
    // 指定原始对象位置
    cv::Rect bb(1150, 150, 330, 330);
    tracker.setBoundingBox(bb);
    // 开始追踪
    processor.run();
    cv::waitKey();
    // 中值追踪器
    cv::Mat image1 = cv::imread("test1/test_3050.png", cv::ImreadModes::IMREAD_GRAYSCALE);
    // 定义网格点
    std::vector<cv::Point2f> grid;
    for (int i = 0; i < 10; i++) {
    
    
        for (int j = 0; j < 10; j++) {
    
    
            cv::Point2f p(bb.x+i*bb.width/10.,bb.y+j*bb.height/10);
            grid.push_back(p);
        }
    }
    // 追踪下一帧
    cv::Mat image2 = cv::imread("test1/test_3051.png", cv::ImreadModes::IMREAD_GRAYSCALE);
    std::vector<cv::Point2f> newPoints;
    std::vector<uchar> status; // 追踪特征点的状态
    std::vector<float> err;    // 追踪误差
    // 追踪点
    cv::calcOpticalFlowPyrLK(image1, image2,    // 2 张连续图像 
                grid,                           // 第一幅图像中的输入点位置 
                newPoints,                      // 第二图像中的输出点位置 
                status,                         // 追踪状态
                err);                           // 追踪误差
    // 绘制点
    for (cv::Point2f p : grid) {
    
    
        cv::circle(image1, p, 1, cv::Scalar(255, 255, 255), -1);
    }
    cv::imshow("Initial points", image1);
    for (cv::Point2f p : newPoints) {
    
    

        cv::circle(image2, p, 1, cv::Scalar(255, 255, 255), -1);
    }
    cv::imshow("Tracked points", image2);
    cv::waitKey();
    return 0;
}

summary

Video object tracking is the process of locating moving objects over time in a video. It has important uses in fields such as smart security. This section introduces some object tracking algorithms implemented in the library and OpenCVimplements a general object tracking framework.

series link