OpenCV 4.x API detailed explanation and C++ examples-motion analysis and object tracking

Section 10 Motion Analysis and Object Tracking

The imgproc module of OpenCV provides basic functions for motion analysis and object tracking. According to these functions, the video can be separated from the foreground and background to achieve the purpose of motion analysis and object tracking.

1、cv::accumulate

---

Accumulate multiple images.

void cv::accumulate(InputArray src,InputOutputArray dst,InputArraymask = noArray())

This function adds src or some of its elements to dst:

$\text{dst}(x,and)\leftarrow \text{dst}(x,and)+\text{src}(x,and)\text{if}\text{mask}(x,and)\ne 0$

This function supports multi-channel images, and each channel is processed independently.

The function cv::accumulate can be used, for example, to collect statistics on the background of the scene viewed by a static camera, and for further foreground and background segmentation.

The parameters are as follows:

parameter name	Parameter Description
src	Input image, supported types are CV_8UC(n) , CV_16UC(n) , CV_32FC(n) or CV_64FC(n) ,
dst	The accumulator image with the same number of channels as the input image, the type is CV_32F or CV_64F.
mask	Optional operating mask.

#include <iostream>
#include <opencv2/opencv.hpp>

using namespace std;

int main()
{
    // 打开摄像头
    cv::VideoCapture cap(0);
    if(!cap.isOpened()){
        cerr << "cannot open camera.\n";
        return EXIT_FAILURE;
    }

    cv::Mat frame;
    int averageCounts = 30;
    int currentCount = 0;
    cv::Mat averageFrame;
    cv::Mat gray;
    cv::Mat frameDelta;
    cv::Mat foreGround,backGround;
    int key = -1;
    while(true){
        cap >> frame;
        if(frame.empty()){
            cerr << "cannot grab frame from camera.\n";
            break;
        }

        cv::cvtColor(frame,gray,cv::COLOR_BGR2GRAY);

        if(currentCount < averageCounts){
            if(averageFrame.empty()){
                averageFrame.create(frame.rows,frame.cols,CV_32FC1);
            }
            // 计算平均图像
            cv::accumulate(gray,averageFrame);
            currentCount++;
        }else{
            // 背景分离
            cv::convertScaleAbs(averageFrame,averageFrame);
            cv::absdiff(gray,averageFrame,frameDelta);

            // 阈值分割
            cv::threshold(frameDelta,foreGround,25,255,cv::THRESH_BINARY_INV);

            cv::imshow("camera",frame);
            cv::imshow("deltaFrame",frameDelta);
            cv::imshow("foreGround",foreGround);

            key = cv::waitKey(10);
            if(key == 27){
                break;
            }
        }
    }
    cv::destroyAllWindows();
    return 0;
}

2、cv::accumulateProduct

---

Add the product of each element of the two input images to the accumulator image.

void cv :: accumulateProduct (InputArray src1, InputArray src2, InputOutputArray etc., InputArray mask = noArray ())

This function adds the product of two images or their selected regions to the accumulator dst:

$\text{dst}(x,and)\leftarrow \text{dst}(x,and)+\text{src1}(x,and)\cdot \text{src2}(x,and)\text{if}\text{mask}(x,and)\ne 0$

This function supports multi-channel images and each channel is processed independently. The parameters are as follows:

parameter name	Parameter name description
src1	The first input image, 1 or 3 channels, 8-bit or 32-bit floating point.
src2	The second input image of the same type and size as src1.
dst	An accumulator image with the same number of channels as the input image as a 32-bit or 64-bit floating point number.
mask	Optional operating mask.

3、cv::accumulateWeighted

---

void cv::accumulateWeighted(InputArray src,InputOutputArray dst,double alpha,InputArray mask = noArray())

This function calculates the weighted sum of the input image src and the accumulator dst, so that dst becomes the running average of the frame sequence:

$\text{dst}(x,and)\leftarrow (1-\text{alpha})\cdot \text{dst}(x,and)+\text{alpha}\cdot \text{src}(x,and)\text{if}\text{mask}(x,and)\ne 0$

In other words, the alpha adjusts the update speed (the accumulator "forgets" the speed of the earlier image). This function supports multi-channel images. Each channel is processed independently.

parameter name	Parameter Description
src	The input image is 1 or 3 channels, 8-bit or 32-bit floating point.
dst	An accumulator image with the same number of channels as the input image, 32-bit or 64-bit floating point.
alpha	Enter the weight of the image.
mask	Optional mask operation.

#include <iostream>
#include <opencv2/opencv.hpp>

using namespace std;

int main()
{
    // 打开摄像头
    cv::VideoCapture cap("videos/vtest.avi",cv::CAP_ANY);
    if(!cap.isOpened()){
        cerr << "cannot open camera.\n";
        return EXIT_FAILURE;
    }

    cv::Mat background;
    cap >> background;
    if(background.empty()){
        cerr << "cannot grab frame.\n";
        return EXIT_FAILURE;
    }
    cv::cvtColor(background,background,cv::COLOR_BGR2GRAY);
    cv::GaussianBlur(background,background,cv::Size(0,0),1.0);

    cv::Mat fback(background.size(),CV_32FC1);
    vector<vector<cv::Point> > contours;
    vector<cv::Vec4i> hierarchy;

    cv::Mat frame,gray,diff;

    int counts = 0;

    while(cap.isOpened()){
        cap >> frame;
        if(frame.empty()){
            cerr << "cannot grab frame\n";
            break;
        }
        // 转换成灰度图像
        cv::cvtColor(frame,gray,cv::COLOR_BGR2GRAY);
        // 高斯模糊
        cv::GaussianBlur(gray,gray,cv::Size(0,0),1.0);

        if(counts < 30){
            // 对图像进行累加,读取前30帧图像，作为背景
            cv::accumulateWeighted(gray,fback,0.8);
            counts ++;
        }else{
             cv::accumulateWeighted(gray,fback,0.2);
        }

        // 分离前景和背景
        cv::absdiff(gray,background,diff);

        // 阈值化处理
        cv::threshold(diff,diff,120,255,cv::THRESH_BINARY);

        cv::imshow("src",frame);
        cv::imshow("diff",diff);

        if(cv::waitKey(10) == 27){
            break;
        }

    }

    return 0;
}

4 、 cv :: phaseCorrelate

---

Used to detect the translation that occurs between two images.

Point2d cv :: phaseCorrelate (InputArray src1, InputArray src2, InputArray window = noArray (), double * response = 0)

This function operation uses the Fourier shift theorem to detect translation in the frequency domain. It can be used for fast image registration and motion estimation. For more information, see http://en.wikipedia.org/wiki/Phase_correlation

Calculate the cross power spectrum of the two provided source arrays. If needed, you can use getOptimalDFTSize to fill the array.

This function implements the following equation:

• First, it applies the Hanning window (see http://en.wikipedia.org/wiki/Hann_function) to each image to eliminate possible edge effects. This window is cached until the array size is changed to speed up processing time.

• Next, it calculates the forward DFT of each source array: G a = F {src 1}, G b = F {src 2} \mathbf{G}_a = \mathcal{F}\{src_1\}, \ ; \mathbf{G}_b = \mathcal{F}\{src_2\}Ga​=F{ src1​},Gb​=F{ src2​} , Where$F$ is the forward DFT.

• Then, it calculates the cross power spectrum of each frequency domain array: $R=\frac{G_a{G}_b^{\ast }}{|{\mathbf{G}}_a{G}_b^{\ast }|}$

• Next, the cross-correlation is converted back to the time domain by inverse DFT: r = F − 1 {R} r = \mathcal{F}^{-1}\{R\}r=F−1{ R}

• Finally, it calculates the peak position and calculates a 5x5 weighted centroid around the peak to achieve sub-pixel accuracy

  ( Δ x , Δ y ) = weightedCentroid { arg ⁡ max ⁡ ( x , y ) { r } } (\Delta x, \Delta y) = \texttt{weightedCentroid} \{\arg \max_{(x, y)}\{r\}\} (Δx,Δy)=weightedCentroid{ argmax( X , and ){ r}}

• If it is not zero, the response parameter is calculated as the sum of the elements of r in the 5x5 center of gravity around the peak position. Normalized to the maximum value of 1 (meaning there is one peak), it will be smaller when there are multiple peaks.

The parameters are as follows:

parameter name	Parameter Description
src1	Source floating point array (CV_32FC1 or CV_64FC1)
src2	Source floating point array (CV_32FC1 or CV_64FC1)
window	Floating point array with windowing coefficients to reduce edge effects (optional).
response	The signal power in the 5x5 centroid around the peak, between 0 and 1 (optional).

#include <stdio.h>
#include <iostream>
#include "opencv2/core.hpp"
#include "opencv2/videoio.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"

using namespace cv;


int main(int argc, char* argv[])
{

  auto src_im = cv::imread(images/balloon.png", 0);
  auto dst_im = cv::imread("images/balloon-2.png", 0);

  cv::Mat src_im_32f, dst_im_32f, hann;

  // src_im.convertTo(src_im_32f, CV_32F);
  // dst_im.convertTo(dst_im_32f, CV_32F);

  cv::Laplacian(src_im, src_im_32f, CV_32F);
  cv::Laplacian(dst_im, dst_im_32f, CV_32F);

  cv::Point2d shift = cv::phaseCorrelate(src_im_32f, dst_im_32f, hann);

  std::cout << shift.x << ", " << shift.y << std::endl;

  return 0;
}

5、cv::createHanningWindow

---

This function calculates two-dimensional Hanning window coefficients.

void cv::createHanningWindow(OutputArray dst,Size winSize,int type)

For more information, see (http://en.wikipedia.org/wiki/Hann_function) and (http://en.wikipedia.org/wiki/Window_function).

The parameters are as follows:

parameter name	Parameter Description
dst	The target array into which Hann coefficients are put
winSize	Window size specification (both width and height must be> 1)
type	Type of array created

// create hanning window of size 100x100 and type CV_32F
Mat hann;
createHanningWindow(hann, Size(100, 100), CV_32F);