Gunnar-Farneback algorithm principle Demo & Farneback optical flow on the abnormal data set UCSD

Gunnar-Farneback algorithm

论文：Two-Frame Motion Estimation Based on Polynomial Expansion

期刊：13th Scandinavian Conference on Image Analysis (SCIA 2003)

作者：Gunnar Farnebäck（Computer Vision Laboratory, Link¨oping University, SE-581 83 Link¨oping, Sweden）

Note: [email protected] http://www.isy.liu.se/cvl/

First, the polynomial expansion

The idea is to expand the polynomial for each pixel in the neighborhood are represented by a polynomial:

$f(x)\sim x^{T}Ax + b^{T}x + c$ （1）

Wherein A is a symmetric matrix, b is a vector, c is a constant. Coefficient is estimated according to the weighted least squares method of signal values in the neighborhood. The weight of two components, known as deterministic and applicability. Deterministic signal value coupled to adjacent areas, according to the point to determine the suitability of points relative weights in the neighborhood of the position of the weight in the neighborhood.

Second, the displacement estimate

Because each neighborhood can be used to approximate a polynomial, so the first analysis of a polynomial change what will happen after translation. Consider the following quadratic polynomial:

$f_{1}(x) = x^{T}A_{1}x + b_{1}^{T}x + c_{1}$ （2）

Overall displacement signal d to obtain units $f_{2}$ :

$f_{2} = f_{1}(x - d)^{T}A_{1}(x - d) + b_{1}^{T}(x - d) + c_{1} = x^{T}A_{2}x + b_{2}^{T} + c_{2}$ （3）

Provided the following quadratic polynomial equation:

$A_{2} = A_{1}$ （4）

$b_{2} = b_{1} - 2A_{1}d$ （5）

$c_{2} = d^{T}A_{1}d - b_{1}^{T}d + c_{1}$ （6）

When $A_{1}$ time is a non-singular matrix, it can be solved $d$ :

$2A_{1}d = -(b_{2} - b_{1})$ （7）

$d = -\frac{1}{2}A_{1}^{-1}(b_{2} - b_{1})$ （8）

In reality, the entire absence of a signal is represented as a polynomial, and global translational these two image signals, the foregoing equation (7) still apply to the actual signal.

Here Global Polynomial Approximation (2) with a local polynomial equation. Then the two images having a polynomial expansion, a first image corresponding coefficients $A_1(x)$ , $b_{1}(x)$ and $c_{1}(x)$ the second image the same way. Ideally, according to equation (4) should have $A_{2} = A_{1}$ , but should actually solving approximation:

$A(x) = \frac{A_1(x) + A_{2}(x)}{2}$ （9）

$\Delta b(x) = -\frac{1}{2}(b_{2}(x) - b_{1}(x))$ （10）

Then $d(x)$ instead of the front of the global variables $d$ :

$A(x)d(x) = \Delta b(x)$ （11）

Let's start to solve the problem neighbor estimation. Calculates the values of each pixel according to the equation (11), but considering the huge amount of computation to do so is clearly not realistic. Assuming that the displacement process is carried out slowly, then it should be reduced as much as possible $x$ neighborhood $I$ search and find the line with equation (11) $d(x)$ :

$$\sum_{\Delta x\epsilon I}^{.} w(\Delta x)\left \| A(x + \Delta x)d(x) - \Delta b(x + \Delta x) \right\|^{2}$ （12）

Here set $w(\Delta x)$ of pixels corresponding weight function, can be obtained according to the method of least squares:

$d (x) = (\ sum wA ^ {T} A) ^ {- 1} \ sum wA ^ {T} \ Delta b$ （13）

Third, the parameters of the displacement field

In order to improve the robustness of the algorithm, for some movement patterns should be established parametric model. The following 2D motion model constructed with eight parameters:

$d_{x}(x, y) = a_{1} + a_{2}x + a_{3}y + a_{7}x^{2} + a_{8}xy$ （15）

$d_{y}(x, y) = a_{4} + a_{5}x + a_{6}y + a_{7}x^{2} + a_{8}xy$

It can be expressed as the formula:

$d = S_{P}$ （16）

$S = \binom{1,x,y,0,0,0,x^{2},xy}{0,0,0,1,x,y,xy,y^{2}}$ （17）

$P = (a_{1},a_{2},a_{3},a_{4},a_{5},a_{6},a_{7},a_{8})^{T}$ （18）

Into the equation (12), to obtain a weighted least squares problem:

$\sum_{i}^{.}w_{i}\left \| A_{i}S_{i}P - \Delta b_{i} \right \|^{2}$ （19）

Now with the $i$ coordinates of the pixel neighborhood to the index, with the method of least squares can be drawn:

$P = (\sum_{i}^{.}w_{i}S_{i}^{T}A_{i}^{T}A_{i}S_{i})^{-1}\sum_{i}^{.}w_{i}S_{i}^{T}A_{i}^{T}\Delta b_{i}$ （20）

Here were calculated as before $S_{i}^{T}A_{i}^{T}A_{i}S_{i}$ and $S_{i}^{T}A_{i}^{T}\Delta b_{i}$ then used $w$ to make a weighted average of the displacement is obtained $d(x)$ .

Fourth, the use of a priori information

So far there is still a problem that two local polynomial signals at the same coordinates in addition to the displacement is the same. Because of this expansion is a partial polynomial model, it will vary with the spatial displacement, it can introduce errors (11), and the error will increase with increasing displacement. So here incorporated priori displacement information, i.e., a first comparison signal $x$ of a polynomial expansion and a second signal in $x + \tilde{d}(x)$ the polynomial expansion wherein $\tilde{d}(x)$ priori displacement rounded up integer value, so that only needs to be calculated the true value based on a priori estimates of displacement.

Alternatively the above (9) and (10) as:

$A(x) = \frac{A_{1}(x) + A_{2}(\tilde{x})}{2}$ （21）

$\Delta b(x) = -\frac{1}{2}(b_{2}(\tilde{x}) - b_{1}(x)) + A(x)\tilde{d(x)}$ （22）

among them,

$\tilde{x} = x + \tilde{d}(x)$ （23）

Fifth, iterative and multi-scale displacement estimates

Displacement field using a priori in the algorithm may be closed loop and advantages of iterations. Have a good a priori estimate means that the relative displacement is smaller, which in turn can improve the accuracy of displacement estimation. Here consider two different methods, iterative and multi-scale displacement of an estimated displacement estimation.

In both methods, the iterative estimation step of displacement, as a next step prior displacement. The first step is usually a priori displacement field is initialized to zero, unless there is a clear message. In the first method, the same polynomial expansion coefficients in all iterations, and only needs to be calculated once. This is a problem that, if the first iteration of displacement is too large (relative to the prior displacement), displacement output can not be expected to improve, iterative loses its meaning. By analyzing on the coarse scale, we can reduce the problem of excessive displacement. This means that we have high applicability to a polynomial expansion. As a result, the estimation algorithm can handle large displacement, but the accuracy is decreased.

Multi-scale displacement estimation method, starting from a coarser scale, to get a rough but reasonable estimates of displacement, then get more and more accurate estimates by stepwise refinement scale. Such a disadvantage is the need to recalculate the polynomial expansion coefficients for each scale, but this cost may be reduced by subsampling between scaling.

Demo Farneback optical flow method in the abnormal data sets UCSD

Experimental environment: Win10 | Python 3.7.3 | OpenCV 4.1.0

Experiment code:

def draw_flow(im, flow, step=16):
    # 在间隔分开的像素采样点处绘制光流
    h, w = im.shape[:2]
    y, x = mgrid[step/2:h:step, step/2:w:step].reshape(2, -1).astype(int)
    fx, fy = flow[y, x].T
    # 创建线的终点
    lines = vstack([x, y, x+fx, y+fy]).T.reshape(-1, 2, 2)
    lines = int32(lines)
    # 创建图像并绘制
    vis = cv2.cvtColor(im, cv2.COLOR_GRAY2BGR)
    for (x1, y1), (x2, y2) in lines:
      cv2.line(vis, (x1, y1), (x2, y2), (0, 255, 0), 1)
      cv2.circle(vis, (x1, y1), 1, (0, 255, 0), -1)
    return vis

# 省略N行代码......

# 提取第一帧
first_rgbframe = cv2.imread(frame_path + frames[0], 1)
del frames[0]
prev_gray = cv2.cvtColor(first_rgbframe, cv2.COLOR_BGR2GRAY)

for frame_i in frames:
  rgbframe = cv2.imread(frame_path + frame_i, 1)
  gray = cv2.cvtColor(rgbframe, cv2.COLOR_BGR2GRAY)
  # 计算流
  flow = cv2.calcOpticalFlowFarneback(prev_gray, gray, None, 0.5, 3, 15, 3, 5, 1.2, 0)
  prev_gray = gray

  drawImg = draw_flow(gray, flow)
  cv2.imwrite(frameFB_path + frame_i, drawImg)
  # 画出流矢量
  cv2.imshow('Optical flow', drawImg)
  if cv2.waitKey(10) == 27:
     break
  time.sleep(0.1)

The results:

UCSDped1\Test\Test014

UCSDped2\Test\Test005

The above are the two Test segment UCSD anomaly dataset in, you can see where the crowd or bicycle / vehicle passes, the green dot will be accompanied by linkage.

appendix:

Dataset link: UCSD Anomaly Detection Dataset

OpenCN - calcOpticalFlowFarneback () parameters explained:

prev: single channel before an input image 8

next: After a single-channel input image 8

flow: calculated input image coincides with the size and type of the optical flow in FIG CV_32FC2

pyr_scale: Construction image scaling factor (<1) of the pyramid; = coefficient such as 0.5 is typical pyramid model, not on the next level is half the size of an image of

levels: the pyramid layers; the stratum level of 1 indicates that no additional level of direct input picture

winsize: averaging window size; the larger window for the less sensitive image noise can be more quick capture operation, but the operation will produce a more missing field

iterations: iterations each pyramid hierarchy algorithm

poly_n: the size of the pixels in the neighborhood, the neighborhood to do polynomial expansion; the larger the neighborhood, the more smooth the image, the algorithm is more robust and more blurred stadium, a standard reference value 5 or 7

poly_sigma: standard smoothing polynomial expansion as a function of basic Gaussian derivative of the difference; when poly_n = 5, may be set to 1.1 poly_sigma; when poly_n = 7, may be provided poly_sigma = 1.5

flags: can be set to one of two flags

OPTFLOW_USE_INITIAL_FLOW: FIG optical flow using the input as an initial value of optical flow estimation

OPTFLOW_FARNEBACK_GAUSSIAN: winsize × winsize filter using a Gaussian filter instead of the same size cassette optical flow estimation; Typically, this option provides more accurate than filter cartridge z stream at a lower speed; generally, a Gaussian window should the winsize set a larger value in order to achieve the same level of robustness.

OpenCV original official explanation: click to jump to the official documents

References:

1. Gunnar Farnebäck. Two-Frame Motion Estimation Based on Polynomial Expansion[C]// 13th Scandinavian Conference on Image Analysis (SCIA 2003). Springer-Verlag, 2003.

2. OpenCV official documents

VG-Lin

Released four original articles · won praise 13 · views 1579

Private letter concerns