The concept of [Optical Flow] was first proposed by Gibson in 1950. It is the instantaneous speed of the pixel motion of the space moving object on the observation imaging plane. It uses the change of the pixels in the image sequence in the time domain and the correlation between adjacent frames to find the existing frame between the previous frame and the current frame. A method for calculating the motion information of objects between adjacent frames.
Generally speaking, optical flow is due to the movement of the foreground object itself in the scene, the movement of the camera, or the combined movement of both. When the human eye observes a moving object, the scene of the object forms a series of continuously changing images on the retina of the human eye, and this series of continuously changing information continuously flows through the retina (ie, the image plane), like a stream of light; , so it is called optical flow.
Optical flow expresses the change of the image, because it contains the information of the movement of the target, so it can be used by the observer to determine the movement of the target. The motion field that cannot be directly obtained is approximated from the picture sequence < motion field, which is actually the movement of objects in the three-dimensional real world; optical flow field is the projection of the motion field on the two-dimensional image plane (human eyes or camera).
In layman's terms, through a sequence of pictures, finding out the motion speed and motion direction of each pixel in each image is the optical flow field. How to find it? Our intuitive understanding must be: the position of point A at frame t is (x 1 , y 1 ), then we will find point A at frame t+1, if its position is (x 2 , y 2 ), then we can determine the motion of point A: (u x , v y ) = (x 2 , y 2 ) - (x 1 ,y1 ).
So how do you know the position of point A at frame t+1? There are many optical flow calculation methods.
Optical flow calculation method
It can be roughly divided into three categories: matching-based methods, frequency-domain methods, and gradient-based methods.
1. Matching-based optical flow calculation methods include feature-based and region
-based methods. 2. Frequency-domain-based methods, also known as energy-based methods, use a speed-adjustable filter bank to output frequency or phase information.
3. Gradient-based methods compute 2D velocity fields (optical flow) using spatiotemporal differentiation of image sequence brightness.
At present, there are two main directions of research on optical flow method.
One is to study the realization of existing algorithms on the basis of the inherent hardware platform, and the
other is to study new algorithms.
The main purpose of optical flow algorithm is to achieve reliable, fast, accurate and robust estimation of optical flow field based on sequence images. However, due to the characteristics of the image sequence target, the illumination in the scene, the change of the light source, the speed of the movement, and the influence of noise, many factors affect the effectiveness of the optical flow algorithm.
Detailed function
1.CalcOpticalFlowPyrLK
Calculate the optical flow of a sparse feature set, using the iterative Lucas-Kanade method in the pyramid
C++ function code
void calcOpticallFlowPyrLK (
InuputArray prevImg,
InputArray prevPts,
InputOutputArraynextPts,
OutputArray err,
Size winSize = Size(21,21),
int maxLevel = 3,
TermCriteriacriteria=TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 0.01),
int flags = 0,
double minEigThreshold = 1e-4
); Detailed function parameters :
| parameter | explain |
|---|---|
| prevImg | The previous frame image or pyramid image with a depth of 8 bits. |
| nextImg | Has the same size and type as prevImg, post-frame image or pyramid. |
| prevPts | Input 2D point vector required to calculate optical flow, point coordinates must be single-precision floating point numbers. |
| nextPts | Output a 2D point vector (also single-precision floating-point coordinates), which contains the new position of the input feature calculated on the next frame of image. |
| status | Output state vector (elements are unsigned char type, uchar), if the stream of the corresponding feature is found, the vector element is set to 1, otherwise, it is 0. |
| err | Output error vector. |
| winSize | Search window size for each pyramid level. |
| maxLevel | The maximum number of pyramid levels; if set to 0, the pyramid is not used (single level); if set to 1, the pyramid has 2 levels, and so on. |
| criteria | Specifies the type of search algorithm convergence iteration |
| minEigTheshold | 算法计算的光流等式的2x2常规矩阵的最小特征值。 |
金字塔Lucas-Kannade算法：
LK算法有三个假设：亮度恒定，即图像场景中目标的像素在帧间运动时外观上保持不变；时间连续或者运动是”小运动“，即图像的运动随时间的变化比较缓慢；空间一致，即一个场景中同一表面上邻近的点具有相似的运动。然而，对于大多数30HZ的摄像机，大而连贯的运动是普遍存在的情况，所以LK光流正因为这个原因在实际中的跟踪效果并不是很好。我们需要大的窗口来捕获大的运动，而大窗口往往违背运动连贯的假设！而图像金字塔可以解决这个问题，于是乎，金字塔Lucas-Kanade就提出来了。
金字塔Lucas-Kanade跟踪方法是：在图像金字塔的最高层计算光流，用得到的运动估计结果作为下一层金字塔的起始点，重复这个过程直到到达金字塔的最底层。这样就将不满足运动的假设可能性降到最小从而实现对更快和更长的运动的跟踪。
2.cvGoodFeaturesToTrack
函数 cvGoodFeaturesToTrack 在图像中寻找具有大特征值的角点。该函数，首先用cvCornerMinEigenVal 计算输入图像的每一个像素点的最小特征值，并将结果存储到变量 eig_image 中。然后进行非最大值抑制（仅保留3x3邻域中的局部最大值）。下一步将最小特征值小于 quality_level?max(eig_image(x,y)) 排除掉。最后，函数确保所有发现的角点之间具有足够的距离，（最强的角点第一个保留，然后检查新的角点与已有角点之间的距离大于 min_distance ）。
C++函数代码
void cvGoodFeaturesToTrack(
const CvArr* image,
CvArr* temp_image,
CvPoint2D32f* corners,
int* corner_count,
double quality_level,
double min_distance,
const CvArr* mask=NULL,
int block_size = NULL,
int use_harris = 0,
double k = 0.4
);函数参数详解：
| 参数 | 解释 |
|---|---|
| image | 输入图像，8-位或浮点32-比特，单通道 |
| eig_image | 临时浮点32-位图像，尺寸与输入图像一致 |
| temp_image | 另外一个临时图像，格式与尺寸与 eig_image 一致 |
| corners | 输出参数，检测到的角点 |
| corner_count | 输出参数，检测到的角点数目 |
| quality_level | 最大最小特征值的乘法因子。定义可接受图像角点的最小质量因子。 |
| min_distance | 限制因子。得到的角点的最小距离。使用 Euclidian 距离 |
| mask | ROI感兴趣区域。函数在ROI中计算角点，如果 mask 为 NULL，则选择整个图像。 |
| block_size | 计算导数的自相关矩阵时指定点的领域，采用小窗口计算的结果比单点（也就是block_size为1）计算的结果要好。 |
| use_harris | 标志位。当use_harris的值为非0，则函数使用Harris的角点定义；若为0，则使用Shi-Tomasi的定义。 |
| k | 当use_harris为k且非0，则k为用于设置Hessian自相关矩阵即对Hessian行列式的相对权重的权重系数 |
实例代码块
本代码参考@浅墨_毛星云的书籍《OpenCV3编程入门》，代码版权归老师所有，仅供学习借鉴只用。
——博主
@--751407505@qq.com
// 程序描述：来自OpenCV安装目录下Samples文件夹中的官方示例程序-利用光流法进行运动目标检测
// 描述：包含程序所使用的头文件和命名空间
#include <opencv2/video/video.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/core/core.hpp>
#include <iostream>
#include <cstdio>
using namespace std;
using namespace cv;
// 描述：声明全局函数
void tracking(Mat &frame, Mat &output);
bool addNewPoints();
bool acceptTrackedPoint(int i);
// 描述：声明全局变量
string window_name = "optical flow tracking";
Mat gray; // 当前图片
Mat gray_prev; // 预测图片
vector<Point2f> points[2]; // point0为特征点的原来位置，point1为特征点的新位置
vector<Point2f> initial; // 初始化跟踪点的位置
vector<Point2f> features; // 检测的特征
int maxCount = 500; // 检测的最大特征数
double qLevel = 0.01; // 特征检测的等级
double minDist = 10.0; // 两特征点之间的最小距离
vector<uchar> status; // 跟踪特征的状态，特征的流发现为1，否则为0
vector<float> err;
//输出相应信息和OpenCV版本-----
static void helpinformation()
{
cout <<"\n\n\t\t\t 光流法跟踪运动目标检测\n"
<<"\n\n\t\t\t 当前使用的OpenCV版本为：" << CV_VERSION
<<"\n\n" ;
}
//main( )函数，程序入口
int main()
{
Mat frame;
Mat result;
//加载使用的视频文件，放在项目程序运行文件下
VideoCapture capture("1.avi");
//显示信息函数
helpinformation();
// 摄像头读取文件开关
if(capture.isOpened())
{
while(true)
{
capture >> frame;
if(!frame.empty())
{
tracking(frame, result);
}
else
{
printf(" --(!) No captured frame -- Break!");
break;
}
int c = waitKey(50);
if( (char)c == 27 )
{
break;
}
}
}
return 0;
}
// parameter: frame 输入的视频帧
// output 有跟踪结果的视频帧
void tracking(Mat &frame, Mat &output)
{
cvtColor(frame, gray, CV_BGR2GRAY);
frame.copyTo(output);
// 添加特征点
if (addNewPoints())
{
goodFeaturesToTrack(gray, features, maxCount, qLevel, minDist);
points[0].insert(points[0].end(), features.begin(), features.end());
initial.insert(initial.end(), features.begin(), features.end());
}
if (gray_prev.empty())
{
gray.copyTo(gray_prev);
}
// l-k光流法运动估计
calcOpticalFlowPyrLK(gray_prev, gray, points[0], points[1], status, err);
// 去掉一些不好的特征点
int k = 0;
for (size_t i=0; i<points[1].size(); i++)
{
if (acceptTrackedPoint(i))
{
initial[k] = initial[i];
points[1][k++] = points[1][i];
}
}
points[1].resize(k);
initial.resize(k);
// 显示特征点和运动轨迹
for (size_t i=0; i<points[1].size(); i++)
{
line(output, initial[i], points[1][i], Scalar(0, 0, 255));
circle(output, points[1][i], 3, Scalar(0, 255, 0), -1);
}
// 把当前跟踪结果作为下一此参考
swap(points[1], points[0]);
swap(gray_prev, gray);
imshow(window_name, output);
}
// 检测新点是否应该被添加
// return: 是否被添加标志
bool addNewPoints()
{
return points[0].size() <= 10;
}
//决定哪些跟踪点被接受
bool acceptTrackedPoint(int i)
{
return status[i] && ((abs(points[0][i].x - points[1][i].x) + abs(points[0][i].y - points[1][i].y)) > 2);
}