opencv study notes 9 - background modeling + optical flow estimation
Background modeling
frame difference method
Since the target in the scene is moving, the position of the target image in different image frames is different. This type of algorithm performs a difference operation on two consecutive frames of images in time. The pixels corresponding to different frames are subtracted to determine the absolute value of the grayscale difference. When the absolute value exceeds a certain threshold, it can be determined to be a moving target, thereby achieving the goal. detection function.
The frame difference method is very simple, but it will introduce noise and hole problems
Gaussian Mixture Model
Before foreground detection, the background is trained first, and a Gaussian mixture model is used to simulate each background in the image. The number of Gaussian mixtures for each background can be adaptive. Then in the testing phase, GMM matching is performed on the new pixels. If the pixel value can match one of the Gaussians, it is considered to be the background, otherwise it is considered to be the foreground. Since the GMM model is continuously updated and learned throughout the process, it has a certain degree of robustness to dynamic backgrounds. Finally, good results were achieved by performing foreground detection on a dynamic background with swaying branches.
The changes in pixels in the video should conform to Gaussian distribution
The actual distribution of the background should be a mixture of multiple Gaussian distributions, and each Gaussian model can also have weights
Mixed Gaussian model learning method
-
1. First initialize each Gaussian model matrix parameter.
-
2. Take T frame data images in the video to train the Gaussian mixture model. After getting the first pixel, use it as the first Gaussian distribution.
-
3. When the pixel value comes later, compare it with the mean value of the previous Gaussian. If the value of the pixel point is within 3 times the variance of its model mean value, it belongs to the distribution, and its parameters are updated.
-
4. If the next coming pixel does not satisfy the current Gaussian distribution, use it to create a new Gaussian distribution.
Gaussian Mixture Model Testing Method
In the testing phase, the value of the new pixel is compared with each mean value in the Gaussian mixture model. If the difference is between 2 times the variance, it is considered to be the background, otherwise it is considered to be the foreground. Assign the foreground value to 255 and the background value to 0. This forms a foreground binary image.
import numpy as np
import cv2
#经典的测试视频
cap = cv2.VideoCapture('test.avi')
#形态学操作需要使用，用于去除噪音
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(3,3))
#创建混合高斯模型用于背景建模
fgbg = cv2.createBackgroundSubtractorMOG2()
while(True):
ret, frame = cap.read()
fgmask = fgbg.apply(frame)
#形态学开运算去噪点
fgmask = cv2.morphologyEx(fgmask, cv2.MORPH_OPEN, kernel)
#寻找视频中的轮廓
im, contours, hierarchy = cv2.findContours(fgmask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for c in contours:
#计算各轮廓的周长
perimeter = cv2.arcLength(c,True)
if perimeter > 188:
#找到一个直矩形（不会旋转）
x,y,w,h = cv2.boundingRect(c)
#画出这个矩形
cv2.rectangle(frame,(x,y),(x+w,y+h),(0,255,0),2)
cv2.imshow('frame',frame)
cv2.imshow('fgmask', fgmask)
k = cv2.waitKey(150) & 0xff
if k == 27:
break
cap.release()
cv2.destroyAllWindows()
Optical flow estimation
Optical flow is the "instantaneous velocity" of the pixel movement of a spatially moving object on the observation imaging plane. Based on the velocity vector characteristics of each pixel, the image can be dynamically analyzed, such as target tracking.
-
Constant brightness: the brightness of the same point will not change as time changes.
-
Small movement: Changes in time will not cause drastic changes in position. Only in the case of small movements can the grayscale changes caused by unit position changes between previous and subsequent frames be used to approximate the partial derivative of grayscale to position.
-
Spatial consistency: adjacent points on a scene are also adjacent points when projected onto the image, and the adjacent points have the same speed. Because there is only one basic equation constraint of the optical flow method, and the required speed in the x and y directions, there are two unknown variables. Therefore, n multiple equations need to be solved in succession.
Lucas-Kanade algorithm
How to solve a system of equations? It seems that one pixel is not enough at all. What other characteristics are there during the movement of objects? (Corner points are reversible, so feature points use corner points)
cv2.calcOpticalFlowPyrLK():
parameter:
-
prevImage previous frame image
-
nextImage current frame image
-
prevPts feature point vector to be tracked
-
winSize the size of the search window
-
maxLevel The maximum number of pyramid levels
return:
-
nextPts output tracking feature point vector
-
status Whether the feature point is found, the status if found is 1, if not found, the status is 0
import numpy as np
import cv2
cap = cv2.VideoCapture('test.avi')
# 角点检测所需参数
feature_params = dict( maxCorners = 100,
qualityLevel = 0.3,
minDistance = 7)
# lucas kanade参数
lk_params = dict( winSize = (15,15),
maxLevel = 2)
# 随机颜色条
color = np.random.randint(0,255,(100,3))
# 拿到第一帧图像
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
# 返回所有检测特征点，需要输入图像，角点最大数量（效率），品质因子（特征值越大的越好，来筛选）
# 距离相当于这区间有比这个角点强的，就不要这个弱的了
p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)
# 创建一个mask
mask = np.zeros_like(old_frame)
while(True):
ret,frame = cap.read()
frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# 需要传入前一帧和当前图像以及前一帧检测到的角点
p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
# st=1表示
good_new = p1[st==1]
good_old = p0[st==1]
# 绘制轨迹
for i,(new,old) in enumerate(zip(good_new,good_old)):
a,b = new.ravel()
c,d = old.ravel()
mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
img = cv2.add(frame,mask)
cv2.imshow('frame',img)
k = cv2.waitKey(150) & 0xff
if k == 27:
break
# 更新
old_gray = frame_gray.copy()
p0 = good_new.reshape(-1,1,2)
cv2.destroyAllWindows()
cap.release()
