Optical flow method

Optical Flow (Optical Flow) is a technique in the field of computer vision, which is used to describe the motion pattern of object points between different frames in an image sequence. The basic idea of the optical flow method is to infer the displacement or velocity of object points between adjacent image frames by analyzing the brightness changes of pixels in the image. Optical flow is of great significance in many applications, such as motion analysis, object tracking, 3D reconstruction, virtual reality, etc.

The principle of optical flow method is based on some assumptions:

Continuity assumption: In a short period of time, the movement of object points is continuous, that is, there will be no sudden changes in the movement between adjacent frames.

Brightness consistency assumption: In adjacent image frames, the brightness of an object point remains unchanged, that is, the pixel value of a point in different frames should be consistent.

Based on these assumptions, the optical flow method attempts to represent the movement of object points in adjacent frames by calculating the displacement vector (optical flow vector) of each pixel. The size of the optical flow vector represents the movement speed of the object point, and the direction represents the direction of movement.

When calculating optical flow, there are several different methods:

Luminance change-based methods: These methods exploit the brightness variation between adjacent frames to infer optical flow. The basic idea is to minimize the grayscale difference of pixels between two frames.

Correlation-based methods: This type of method determines optical flow by calculating the cross-correlation of pixels in adjacent images. They are more powerful at finding matches in brightness patterns.

Energy optimization-based methods: These methods transform the optical flow estimation problem into an energy minimization problem and solve it using various optimization techniques to find the optimal optical flow field.

Machine learning-based methods: In recent years, deep learning technology has also made significant progress in optical flow estimation. Using techniques such as convolutional neural networks, complex optical flow patterns can be learned from images.

Although the optical flow method is simple in theory, it faces many challenges in practical applications, such as occlusion, texture discontinuity, illumination changes, etc. Therefore, it is very important to choose an appropriate optical flow estimation method depending on the application scenario.

The following are several representative papers on optical flow method:

1. Lucas-Kanade optical flow method (sparse optical flow method):

Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In IJCAI (Vol. 81, No. 1, pp. 674-679).

        The main content of this paper involves image registration technology in the field of image processing, especially its application in stereo vision applications. Image registration is the alignment of two or more images for comparison, analysis, or merging . In stereo vision, registration of two or more images can be used to generate depth maps, three-dimensional models, and other stereo vision-related information.

        The main contribution of the paper is the introduction of an iterative image registration technique. The goal of this technique is to find points in one image that match corresponding points in another image, thereby achieving registration between the two images. In order to achieve this goal, the author proposed an iterative optimization method based on local image brightness consistency, which can search the position of the corresponding point in the image and gradually improve the accuracy of matching through continuous iterative optimization.

        This paper has aroused widespread interest in the field of computer vision because image registration is a key step in many computer vision tasks, including stereo vision, object tracking, image stitching, etc. The iterative registration technology proposed in the paper provided a basic framework for subsequent research and applications. Many subsequent image registration methods were inspired by this paper.

        Overall, in this paper we introduce an important image registration technique that has had a profound impact in the field of computer vision and provides a key method for applications such as stereo vision.

2. Horn-Schunck optical flow method (dense optical flow method):

Horn, B. K., & Schunck, B. G. (1981). Determining optical flow. Artificial intelligence, 17(1-3), 185-203.

        This paper focuses on the determination of optical flow, which refers to the movement pattern of points on the surface of an object in an image between consecutive image frames. Determining optical flow is critical for understanding object motion in image sequences as well as other computer vision tasks. The goal of the paper is to propose a method that can calculate the optical flow vector of an object point from a continuous image sequence.

        In this paper, Horn and Schunck introduce an optical flow calculation method based on the brightness constancy assumption. This assumption is based on the observation that points on an object surface should remain unchanged in grayscale between adjacent image frames . The author transforms this problem into an energy minimization problem and uses optimization to find the optical flow field that minimizes energy.

        This method is based on a variational optimization technique. The author formalizes the brightness constancy assumption as an energy function, and then uses the variational method to solve the minimum value of this energy function to obtain an estimate of the optical flow field . This method is an iterative process that gradually reduces energy by continuously adjusting the optical flow field.

        The contribution of this paper is to introduce a method to solve the optical flow estimation problem and formalize the problem as an optimization problem, thus providing a basis for subsequent research. This approach, although based on simplifying assumptions, provides an inspiring framework for the optical flow estimation problem in the field of computer vision.

        Overall, Horn and Schunck proposed a classic optical flow estimation method in this paper, which laid the foundation for later computer vision research and is of great significance for understanding object motion patterns in image sequences.

3. Farneback optical flow method:

Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363-370). Springer.

The paper introduces a two-frame motion estimation method based on frequency domain and polynomial expansion.

        This paper focuses on the problem of motion estimation between dual-frame images, that is, inferring the motion of points on the surface of an object from two consecutive images. This is of great significance for many computer vision applications, such as optical flow estimation, target tracking, etc. The goal of the paper is to propose a method that can estimate the pixel-level motion field from two frames of images.

        In this paper, the author Farnebäck introduces a motion estimation method based on polynomial expansion . This method is based on a key observation: the motion pattern of an object between two frames can be approximated as a polynomial. The author uses this idea to model the pixel value changes between two frames of images through polynomial expansion , and determines the coefficients of the polynomial through optimization to obtain pixel-level motion estimation.

        This method has some advantages over traditional methods based on brightness changes or correlations , especially in the presence of weak textures or occlusions. It can handle non-uniform motion and complex scenes in images better.

        The contribution of this paper is to introduce a novel two-frame motion estimation method based on polynomial expansion, which provides a new idea for the field of computer vision. This method performs well in some cases and provides an effective tool for processing images of complex scenes and motion patterns.

        Overall, Farnebäck proposes an innovative two-frame motion estimation method in this paper, which brings a new perspective to the motion estimation problem in the field of computer vision and has practical application value in specific situations.

4 FlowNet: End-to-end depth optical flow estimation:

Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., ... & Brox, T. (2015). FlowNet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision (ICCV) (pp. 2758-2766).

This paper introduces the method of using convolutional neural networks (CNN) to learn optical flow.

        Optical flow refers to the movement pattern of object points in an image between different image frames. It is very important for many tasks in computer vision, such as target tracking, 3D reconstruction, and image synthesis. Traditional optical flow estimation methods are usually based on complex mathematical models and calculations, while in recent years, deep learning technology, especially convolutional neural networks, has made significant progress in the field of image processing.

        The main goal of this paper is to learn optical flow estimation using convolutional neural networks. The authors propose a network structure called FlowNet, which is an end-to-end convolutional neural network that can directly learn optical flow from two images . FlowNet is designed to take full advantage of the advantages of convolutional neural networks in image feature extraction and learning representations.

        FlowNet contains two main branches, one for processing two input images. Each branch contains multiple convolutional and pooling layers, as well as specific connection layers, for generating optical flow fields. The entire network optimizes the optical flow estimation in the training data through supervised learning.

        The contribution of this paper is to introduce a method for optical flow estimation using convolutional neural networks, avoiding the complex manual feature engineering and mathematical modeling in traditional methods. FlowNet achieved excellent optical flow estimation results at the time, proving the potential of deep learning in the field of optical flow estimation.

        In general, the paper "FlowNet: Learning Optical Flow with Convolutional Networks" introduces deep learning methods to the field of optical flow estimation, provides new ideas for computer vision research, and lays the foundation for subsequent research.

5. RAFT: Structured optical flow estimation:

Teed, Z., & Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 336-352).

代码： GitHub - princeton-vl/RAFTContribute to princeton-vl/RAFT development by creating an account on GitHub.https://github.com/princeton-vl/RAFT

"Raft: Recurrent All-Pairs Field Transforms for Optical Flow" is a paper published by Zachary Teed and Jia Deng at the European Conference on Computer Vision (ECCV) in 2020. This paper introduces a method for optical flow estimation, called Raft, which utilizes cyclic all-pair field transformations to achieve optical flow estimation.

The main goal of this paper is to propose a novel method to solve the optical flow estimation problem, namely the Raft method. The Raft method employs a cyclic all-pair field transformation to estimate optical flow by operating on all point pairs. This method is different from traditional local methods in that it can capture a larger range of motion information on a global scale .

The core idea of the Raft method is to gradually optimize the optical flow estimation by applying the optical flow transformation of each pair of points to the entire image through stepwise iterations. This full pair field transformation method can capture the interaction between different points, thereby improving the accuracy of optical flow estimation.

The contribution of the paper is the introduction of an optical flow estimation method based on cyclic full pair field transformation, which captures motion information in a global range and has good performance. The Raft method achieves excellent results on multiple optical flow estimation datasets, demonstrating its effectiveness.

Overall, the paper "Raft: Recurrent All-Pairs Field Transforms for Optical Flow" introduces an innovative method to the field of optical flow estimation, provides a new perspective for solving motion estimation problems, and contributes to computer vision research and Application brings new revelations.

Papers related to optical flow method - LK optical flow method, HS optical flow method, Farneback optical flow method, FlowNet: end-to-end depth optical flow estimation, RAFT: structured optical flow estimation