[Disclaimer: This literature review of academic programs from winter camps million students in Jiangsu Province. I agree and indicate the need to reprint Source]
The first written literature review, no experience, thanks to the guidance of many teachers. Read before writing many papers and blog, on a formula to derive the optical flow but do not have a lot of talk about how specific, according to the formula for those who want the code line and hand comrades, reading these formulas some difficulty, in this article, I do these formulas some transformation, from pure mathematics and in a manner consistent and intuitive programming practice to introduce optical flow algorithm, to facilitate understanding. For questions about any part of the text, you can directly ask questions in the comments.
Foreign optical flow technology, there are many leading technologies such as energy and phase based on the calculation method of optical flow field, but I really do not understand Caishuxueqian, want to know about my coaching Daniel, thank you.
table of Contents
2 optical flow past and present
2.1 optical flow concepts, basic principles and assumptions
2.2 Several mainstream optical flow algorithm
2.2.1 Horn-Schunck optical flow method [3]
2.2.2 Lucas-Kanade optical flow method [4]
2.2.3 block matching algorithm [5]
2.3 Application of Optical Flow
2.3.2 Positioning optical flow
2.4 The current technical challenges and some solutions
Past, present and development trend of Optical Flow
Abstract : optical flow field of computer vision is to extract a video image or continuous optical flow algorithm. To characterize the surface of the optical flow field moving image is grayscale. In the ideal case, the optical flow field and the two-dimensional image of the stadium match, the correlation between the change in the optical flow method using a pixel in the image sequence and the time domain between adjacent frames to find the one with the current frame correspondence relation exists, to calculate the motion information of the object between adjacent frames, and thus can be used for estimation of object motion analysis of the situation. Thus, optical flow method is very useful in the field of computer vision and pattern recognition, it can be used for motion detection, object tracking, visual positioning direction.
Keywords: optical flow; computer vision; target track; visual positioning.
Abstarct: Optical flow is an algorithm for extracting optical flow field from video or continuous images in the field of computer vision. Optical flow field is used to characterize the surface motion of gray pattern image . Ideally, the optical flow field coincides with the two-dimensional motion field in the image. Optical flow method uses the change of pixels in the image sequence in time domain and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thus calculating the motion information of objects between adjacent frames, so it can be used to estimate and analyze the motion of objects. Therefore, optical flow method is useful in the field of pattern recognition and computer vision. It can be used in motion detection, target tracking, visual positioning and other directions.
Key words: optical flow; computer vision; target tracking; visual positioning.
1 development trend of optical flow
Originated in computer vision can be traced back to a very early time, but until the 20th century, the late 70's, when's enough to improve the performance of computer processing, such as image when such a large-scale data, computer vision it has gained official attention and development [1 ].
Optical flow method is in the field of computer vision algorithm for extracting a successive image in a video or optical flow, with the continuous development of computer manufacturing technology, computer operator force is increased year by year. The complexity of some of the original larger image processing algorithm begins moving toward real-time, optical flow can be used as an optical flow field approximation algorithm stadium under certain circumstances, it may be applied to moving target tracking, and other fields. Therefore, attention and scholars optical flow is increasing. Analysis tool by degree of concern Chinese HowNet. Researchers can see the light flow method of attention increased year by year (Figure 1.1).
Chart 1.1 Chinese HowNet (CNKI) statistics 1954-2018 year "optical flow" of the results of academic concern under the direction of
Light streams through reading, comparison and classification of relevant literature, as of December 31, 2018, the author retrieved article on optical flow algorithm, mainly in the Horn-Schunck optical flow algorithm, Lucas-Kanade optical flow algorithm and block Research and improvement matching algorithm based on. In addition there are some attempts to use energy-based methods or methods based on phase I hope that this literature review, detailed as described in the following:
Concept (1) of the optical flow, basic principles and assumptions;
(2) several mainstream optical flow algorithm;
(3) Application of Optical Flow
(4) current technical challenges and some solutions
(5) the development direction
2 optical flow past and present
2.1 optical flow concepts, basic principles and assumptions
Temporal variation of the light flow study using the image pixel intensity data and the sequence of determining the respective correlation pixel position "movement" [2], a simple method that the purpose is to find the optical flow of each pixel in the grayscale moving speed of the continuous images.
Achieve optical flow method of claim meet several basic assumptions:
(1) constant luminance between adjacent frames;
(2) neighboring video frame time to take successive frames, or movement of the object between adjacent frames Comparative "minute";
(3) retaining spatial consistency; i.e., the same sub-pixels having the same motion picture.
Suppose (1) is a condition for establishing an optical flow constraint equations, this assumption is very intuitive description, when a moving object, the object brightness and texture should remain substantially constant.
Suppose (2) is to ensure that the gray value function of time on Cartesian coordinates continuously differentiable, the optical flow constraint is the next requirement is a necessary condition for equation derived.
Hypothesis (3) is a condition intuitive, under normal circumstances, the target object when the motion picture, the target object is used to represent a pixel region necessarily move together, thus belong to the same velocity vector of the inter pixel size of an object and should be the same direction, so that the optical flow constraint equation is assumed to obtain a plurality of constraints can formula, thereby solving a motion vector into equations overdetermined.
1 can be seen as a three-dimensional array of video
The optical flow method, or a video set each time a successive image pixel gray value of each treated as a three-dimensional matrix configuration (FIG. 1). By a discrete function 
is completed and the time at any pixel location mapping to a corresponding gray value, for example: 
represents the tone value of the pixel row y in column x time.
The assumption (a), after a time of a pixel, the gradation value is not changed, namely:
(2-1)
Because 
small enough to ignore the high order endless hours to obtain the approximate equation:
(2-2)
Wherein 
the gradation value of the partial derivative of x, is the 
gray value of the partial derivative of y, 
for the gradation value of the partial derivative of t, 
is the instantaneous speed of the pixels in the X direction, 
for the pixels in the Y direction of the instantaneous velocity .
By the formula (2-1), (2-2) can be obtained:
(2-3)
式(2-3)就是光流约束方程，
为当前位置的光流矢量,整张二维图像上所有像素的光流矢量的集合就是光流场。
在一般情况下，光流场和运动场的运动趋势是类似的，所以我们可以利用光流场来实现对运动物体进行分析。当然，也存在一些特例，比如光源固定情况的均匀球体，在该球体旋转时光流场不会有变化[3]，因此，若要通过光流场来推测图像中的运动场，必须确保运动物体的运动趋势可以通过其对应灰度值的变化反映出来。
光流法按照不同的实现方式可以分为：基于梯度的方法、基于匹配的方法、基于能量的方法、基于相位的方法等，其中基于梯度的方法和基于匹配的方法由于计算量能满足一般的实时处理而被广泛使用。
2.2 几种主流光流算法
2.2.1 Horn-Schunck光流法[3]
Horn-Schunck算法属于基于梯度的光流算法。主题思想是全局平滑假设，即运动物体内部的光流场是相同的，因此物体内部光流场的梯度应该为零，也就是说物体内部的光流场应当是平滑的，故得到一个约束方程：
(2-4)
根据式（2-4），我们可以用来衡量一个像素点周围光流场的平滑度了。Horn-Schunck算法用来表示光流场的不平滑度（数值越大，说明越不平滑）
到目前为止，使用的还是基于连续函数的分析方法，但是在数字图像中，必须离散化才能使用。因此为了方便处理，令：
(2-5)
其中sum(M)表示M矩阵的元素和，A·B表示A矩阵与B矩阵的点积。
由于图像不可避免的存在噪声，根据式（2-3）引入：
(2-6)
来表示图像灰度的变化率。
要找到最接近真实的光流场分布的结果，Horn-Schunck算法设计了一个总误差参数：
(2-7)
式中为权重参数，用来调节图像灰度的变化率和光流场平滑度在指引光流场正确梯度下降时所占比重。当取得最小值时，认为计算得到的光流场是最优的。由式（2-5）、（2-6）、（2-7）可得到光流场的迭代公式：
(2-8)
2.2.2 Lucas-Kanade光流法[4]
Lucas-Kanade算法也是用了类似Horn-Schunck算法的光流场平滑假设，Lucas-Kanade算法认为一个像素周围的相邻像素的光流场应该和中心的像素光流场一致。因此根据式（2-3）可以得到一系列等式：
(2-9)
光流场的总误差可表示为：
(2-10)
用最小二乘法求解式（2-10）得：
(2-11)
其中
(2-12)
当不可逆时无法求解式（2-11）
否则可得该点的光流矢量为：
(2-13)
2.2.3 块匹配算法[5]
块匹配算法的思路是把某一帧某一个像素为中心的图块在下一帧中找到最匹配的图块，两帧中的图块中心点的位移就是这一点的光流位移（图 2）。块匹配算法可以根据实际要求和设备算力灵活配置成稠密光流或稀疏光流。不足在于这种方法的鲁棒性较差，对于图片发生的畸变和镜头旋转难以处理。
图 2 块匹配示意图
2.3 光流法的应用
光流法的应用主要有两类，一类利用光流场中局部光流的变化在静态背景中识别、跟踪运动物体；另一类利用光流场的整体变化来推测视频拍摄设备的移动信息，主要运用在视觉定位中。
2.3.1 目标跟踪
由于光流场和运动场在多数场合是类似的，因此可以通过对光流场的分析在估测图像中物体的运动状态。
在安防领域，运用光流法可以判断监控区域内是否有运动行人，控制监控的工作模式和存储模式，以达到降低功耗和节约存储的作用。
在人脸表情识别中。可以运用光流法检测面部肌肉的微小变化用于分析人脸微表情。
在一些图像处理的算法中，算法关于视屏尺寸的复杂度较大，但在图像中有些部分并不包含所需要的信息，需要通过设置ROI（感兴趣区域，region of interest，下同）来减少计算时间，这时就可以用光流法在视频中找到运动区域进行进一步处理。
图3.1是OpenCV中 Farneback光流的效果，可以看到移动的螺丝刀是图片中唯一检测到到的目标，颜色在色度环上位置显示了物体运动的方向。
图3.2是笔者曾经设计的一个检测血管血液流速的demo,高亮区域为流速明显的区域，其核心算法也使用了光流。
图 3.1 检测运动物体
图 3.2 检测血管流速
2.3.2 光流定位
光流定位是通过光流场的变化来推测相机的移动速度，一般用于飞行器的视觉辅助定位。比如通过挂在在四旋翼飞行器下方的相机，相机拍摄正下方的连续图像，经过处理即可得到光流场的分布，若此时全画面光流场都出现一个方向的移动，就可以推断四旋翼与地面存在相对移动，如果四旋翼需要悬停，只需根据光流相机计算得到的移动速度实现位置闭环即可。通过对速度的积分，还可以得到四旋翼的总位移，这也是视觉里程计的一个基本思路。
2.4 当前技术挑战与一些解决方案
2.4.1 算法复杂度
光流算法在目前任然受到算法复杂度的限制，精度和速度依然是一对矛盾，因此光流的实时性应用往往需要高性能的计算设备，但是在光流定位方面，已经出现了许多嵌入式的解决方案：一种是光流定位芯片，如优象科技的光流芯片；另一种是以FPGA芯片或ARM Cortex-A系列芯片为核心的模块或处理单元，如大疆的精灵系列无人机所使用的光流定位方案。
2.4.2 鲁棒性
光流算法的另一个局限是对于图像的畸变和非刚体的运动做计算时鲁棒性较差，这种问题的根源是光流算法的平滑假设无法处理邻近像素发生偏移时产生的误差。目前也有一些改进方法，比如通过其他的传感器判断相机的姿态，对图像预先做仿射变换以确保图像视角的稳定[6]
2.4.3 孔径问题
光流算法还有一个困扰研究者已久的问题——孔径问题：当图片的纹理信息较少的时候，算法会找到许多匹配的像素，从宏观看，光流场在这些地方就是静止的或者是一些噪声。引起孔径问题的原因是出现了一个没有纹理或纹理不明显的像素区域，而且区域大于光流算法检测光流的近邻像素区，就像相机的镜头盖着一张白纸，白纸在移动但是对于相机来说，只能得到全白的画面，无法推断前景的移动。为了解决这个问题，目前最有效的办法是采用图像金字塔，这样图像的尺寸变小了，但是光流检测的邻近像素区尺寸没有变，这样对于原图片来说，近邻像素区变大了，近邻像素区大于无纹理区的尺寸，这样就避免了“孔径问题”，同时，图像金字塔的引入还提高了大幅度位移的检测精度[6]。
图 4 图像金字塔
2.5 发展方向
2.5.1 更多特征
光流场是基于像素亮度变化得到的，可以看做是图像的亮度特征。光流算法的计算的最基本假设就是亮度不变。但是在实际情况下,这个假设很难满足。光照，反光或阴影都会影响算法的匹配。这时出现了一种基于图像角点特征的光流场算法，这种办法可以在一定程度上抵抗亮度变化导致的干扰。笔者自己也写过基于极值点特征的光流算法，效果相当不错。由此笔者想到，图像有无穷多的特征，是不是其中的任意一种或多种都可以应用于光流场的计算？现在用于处理的图像都是灰度图，而当前的相机大多是彩色的，色彩的特征能否用于光流，也许可以实现色度流，饱和度流？这些笔者没有尝试过，也不确定目前是否有相关的研究，不过笔者相信，基于多种特征的“光”流算法，是有研究意义的。
2.5.2 更多维度
光流定位是一种已经投入市场的技术，大疆、pixhawk的飞控都支持光流（模块）的输入。不过目前的光流只能做到检测水平方向的位移速度，而实际上高度和旋转的变化信息也可以通过光流场得出。笔者正在研究这一方面，希望能得到期待的结果。
2.5.3 更快速度
算法速度几乎是任何一种算法不变的追求，速度越快就有可能在低算力的嵌入式平台上运行,在现行算法中，关于梯度下降的光流算法有很多优化方案，代价是牺牲了一定的精度。如何损失最小的精度并获得更高的运算速度，也是一个值得研究的方向。
2.5.4 更具鲁棒性
由于大量算法应用了平滑假设，导致这类算法对于画面的旋转和畸变几乎没有抵抗力，虽然有些算研究者用一些预处理算法把图片的旋转和畸变消除后在进行光流处理，但是预处理消耗了一定的算力，如果能提出一种能够直接抵抗画面的旋转和畸变的光流算法，意义相当巨大。
3 光流技术的未来
光流法发展至今，研究者提出了多种计算光流的算法，也有许多基于光流法思路的改进算法，基于光流定位的产品也已经开始投入市场。但是，光流法作为一种需要大量计算的算法，在嵌入式设备上的运行速度和精度还是受嵌入式硬件的算力影响很大，如何提高精度和优化算法，将是未来十年内研究者的主要研究方向。
4 参考文献
- Zouxy09.光流Optical Flow介绍与OpenCV实现[EB/OL].https://blog.csdn.net/zouxy09/article/details/8683859,2013-03-17/2019-03-02
- ChenLee_1.光流法简单介绍[EB/OL].https://blog.csdn.net/carson2005/article/details/7581642,2012-05-18/2019-03-02
- Horn B K P, Schunck B G. Determining optical flow[J]. Artificial intelligence, 1981, 17(1-3): 185-203.
- Baker S, Matthews I. Lucas-kanade 20 years on: A unifying framework[J]. International journal of computer vision, 2004, 56(3): 221-255.
- Davis C Q, Karul Z Z, Freeman D M. Equivalence of subpixel motion estimators based on optical flow and block matching[C]//Proceedings of International Symposium on Computer Vision-ISCV. IEEE, 1995: 7-12.
- Ranjan A, Black M J. Optical flow estimation using a spatial pyramid network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4161-4170.
- 曹远宁. 四旋翼无人机的运动目标识别与跟踪[D].东北大学,2015.
- 李奇. 基于深度神经网络的面部表情识别研究与实现[D].南京邮电大学,2018.
- Mémin E, Pérez P. Dense estimation and object-based segmentation of the optical flow with robust techniques[J]. IEEE Transactions on Image Processing, 1998, 7(5): 703-719.
- Verri A, Straforini M, Torre V. Computational aspects of motion perception in natural and artificial vision systems[J]. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 1992, 337(1282): 429-443.
- 张艳艳. 基于光流算法的运动目标检测应用研究[D].西安石油大学,2018.
- 李成美,白宏阳,郭宏伟,梁华驹.一种改进光流法的运动目标检测及跟踪算法[J].仪器仪表学报,2018,39(05):249-256.
- 袁如月. 基于FPGA的无人机光流定位系统研究[D].哈尔滨工业大学,2017.
- 马龙,王鲁平,李飚,陈小天.基于像素筛选技术的光流估计方法[J].吉林大学学报(工学版),2012,42(04):979-984.
