Chapter 18: Video Processing

Video is an important source of information, a type of information that is often processed during visual processing.
A video is made up of a series of images called frames, which are taken from the video at regular intervals.
The speed of acquiring (playing) frames is called the frame rate, and its unit is usually expressed in "frames per second", which represents the number of frames that appear within 1 second, and the corresponding English is FPS (Frames Per Second). If independent frames are extracted from the video, they can be processed by image processing methods to achieve the purpose of video processing.

OpenCV provides cv2.VideoCapture class and cv2.VideoWriter class to support processing various types of video files. In different operating systems, the file types they support may be different, but video files in AVI format are supported in various operating systems.

cv2.VideoCapture: mainly used to read and process camera or video information
cv2.VideoWriter: mainly used to write video

1. VideoCapture class:

OpenCV provides the cv2.VideoCapture class to process video. It can handle both video files and camera information.

1. Introduction to related methods in the VideoCapture class:

Common methods of the cv2.VideoCapture class include initialization, opening, frame capture, release, property setting, etc. These functions are briefly introduced below.

(1) Initialization:

The construction method of the cv2.VideoCapture class in OpenCV is cv2.VideoCapture(), which is used to open the camera and complete the initialization of the camera or initialize the video file object.

The syntax of this function is:

Capture object = cv2.VideoCapture("object that needs to be initialized")
- When the parameter passed in the cv2.VideoCapture() construction method is "the ID number of the camera", the camera object is initialized.
  
  Note that this parameter is the ID number of the camera device (camera). Its default value is -1, which means randomly selecting a camera; if there are multiple cameras, use the number "0" to represent the first camera, use the number "1" to represent the second camera, and so on. So, if there is only one camera, you can use either "0" or "-1" as the camera ID number. On some platforms, if the value of this parameter is "-1", OpenCV will pop up a window to let the user manually select the desired camera.
- When the "video file name" is passed in the cv2.VideoCapture() construction method, the video object is initialized.
  
  Note: It can be the url path of the video.
- Capture object: return value, instantiated video or camera object. cv2.VideoCapture() construction method

Special Note: After the video is processed, remember to release the camera or video object.

(2) Initialization judgment method:

cv2.VideoCapture.open()
cv2.VideoCapture.isOpened(): Determine whether the current camera or video file is initialized successfully:

In general, use the cv2.VideoCapture() function to complete the initialization of the camera or video file. Sometimes, in order to prevent initialization errors, you can use the function cv2.VideoCapture.isOpened() to check whether the initialization is successful. The syntax of this function is:

retval=cv2.VideoCapture.isOpened()
- If successful, the return value retval is True.
- If unsuccessful, the return value retval is False.

If the camera or video file initialization fails, you can use the function cv2.VideoCapture.open() to open the camera or video file. The syntax of this function is:

retval=cv2.VideoCapture.open(index)
- index is the camera ID number or video file name.
- retval is the return value, when the camera (or video file) is successfully opened, the return value is True.

(3) Capture frame:

After the camera (video file) is initialized successfully, frame information can be captured from the camera. The function cv2.VideoCapture.read() is used to capture the frame.

The syntax of this function is:

retval,image=cv2.VideoCapture.read()
- image: is the returned captured frame, or empty if no frame was captured.
- retval: Indicates whether the capture is successful, if successful, the value is True, and if unsuccessful, it is False.

(4) release:

Turn off the camera when it is not needed. The function cv2.VideoCapture.release() is used to close the camera. The syntax of this function is:

None=cv2.VideoCapture.release()

For example, there is currently an object cap of the VideoCapture class. To release it, you can use the statement:

cap.release()

(5) Property setting:

Sometimes, we need to get the properties of cv2.VideoCapture class object, or change the properties of this class object.

Obtain object properties: cv2.VideoCapture.get() The syntax format of this function is:

retval=cv2.VideoCapture.get(propId)
- propId: Corresponds to the properties of the cv2.VideoCapture class object.

For example, if there is a cv2.VideoCapture class object cvc, then:

Through cvc.get(cv2.CAP_PROP_FRAME_WIDTH), the width of the current frame object can be obtained.
Through cvc.get(cv2.CAP_PROP_FRAME_HEIGHT), the height of the current frame object can be obtained.

Set object properties: cv2.VideoCapture.set() The syntax of this function is:

retval=cv2.VideoCapture.set(propId,value)
- propId: corresponds to the attribute of the cv2.VideoCapture class object
- value: The value corresponding to the attribute propid.

For example, if there is a cv2.VideoCapture class object cvc, then:

The statement ret=cvc.set(cv2.CAP_PROP_FRAME_WIDTH,640) sets the width of the current frame object to 640 pixels.
The statement ret=cvc.set(cv2.CAP_PROP_FRAME_HEIGHT,480) sets the height of the current frame object to 480 pixels.

The attribute values and meanings of the cv2.VideoCapture class object are shown in the table.

(6) Capture multi-camera (video file) data:

cv2.VideoCapture.grab(): captures the next frame in the video.
cv2.VideoCapture.retrieve(): decodes and returns the frame captured by grab().

Generally, if you need to read the video data of a camera, the easiest way is to use the function cv2.VideoCapture.read(). However, if you need to synchronize the video data of a group or a multihead camera (such as a stereo camera or Kinect), this function will not be able to do it. The function cv2.VideoCapture.read() can be understood as being composed of the function cv2.VideoCapture.grab() and the function cv2.VideoCapture.retrieve(). The function cv2.VideoCapture.grab() is used to point to the next frame, and the function cv2.VideoCapture.retrieve() is used to decode and return a frame. Therefore, you can use the functions cv2.VideoCapture.grab() and function cv2.VideoCapture.retrieve() to obtain data from multiple cameras.

The function cv2.VideoCapture.grab() is used to point to the next frame, and its syntax format is:

retval=cv2.VideoCapture.grab()

If the function successfully points to the next frame, the return value retval is True.

The function cv2.VideoCapture.retrieve() is used to decode and return the video frame captured by the function v2.VideoCapture.grab(). The syntax of this function is:

retval,image=cv2.VideoCapture.retrieve()
- image is the returned video frame, or an empty image if unsuccessful.
- retval is a Boolean value, if not successful, returns False; otherwise, returns True.

For a group of cameras, the following code can be used to capture video frames from different cameras:

success0=cameraCapture0.grab()
success1=cameraCapture1.grab()
if success0 and success1:
	frame0=cameraCapture0.retrieve()
	frame1=cameraCapture1.retrieve()

2. Capture camera video:

Example: Use the cv2.VideoCapture class to capture camera video.

import cv2
import numpy as np

cap = cv2.VideoCapture(0)

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1260)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

while cap.isOpened():
    ret, frame = cap.read()
    cv2.imshow('frame', frame)
    c = cv2.waitKey(1)
    print(c)
    if c == 97:
        break

cap.release()
cv2.destroyAllWindows()

Note: waitKey function

The function cv2.waitKey() is used to wait for the key. When the user presses the keyboard, the statement will be executed and the return value will be obtained. Its syntax format is:

retval=cv2.waitKey（[delay]）
- retval: Indicates the return value. If no key is pressed, it returns −1; if a key is pressed, it returns the ASCII code of the key.
- delay: Indicates the time to wait for the keyboard trigger, the unit is ms. When the value is negative or zero, it means waiting infinitely. This value defaults to 0.

3. Play the video file:

You can set the duration (stay) time of each frame when playing the video by setting the parameter value in the function cv2.waitKey(). If the parameter value in the function cv2.waitKey():

If it is smaller, it means that each frame stays for a shorter time, and the video playback speed will be faster.
If it is larger, it means that each frame stays for a longer time, and the video playback speed will be slower.

import cv2

cap = cv2.VideoCapture(
    'https://vd3.bdstatic.com/mda-ijurqhuzpqac4fm8/sc/mda-ijurqhuzpqac4fm8.mp4?v_from_s=hkapp-haokan-suzhou&auth_key=1639325601-0-0-8ddbd7ccf330e4d61fbbd0dd792b3eb9&bcevod_channel=searchbox_feed&pd=1&pt=3&abtest=&klogid=2601737365')

while cap.isOpened():

    ret, frame = cap.read()
    
    # 设置每一帧的图像大小
    frame = cv2.resize(frame, (640, 320))
    cv2.imshow('frame', frame)
    c = cv2.waitKey(100)
    if c == 97:
        break

cap.release()
cv2.destroyAllWindows()

Note: video file name can be url

Two, VideoWriter class

The cv2.VideoWriter class in OpenCV can save the image sequence as a video file, modify various attributes of the video, and complete the conversion of the video type.

1. Class function introduction:

Commonly used member methods of the cv2.VideoWriter class include: construction method, write method, etc. This section briefly introduces these two commonly used methods.

(1) Construction method:

OpenCV provides a constructor for the cv2.VideoWriter class, which is used to implement initialization. The syntax of this function is:

< VideoWriter object > =cv2.VideoWriter（filename,fourcc,fps,frameSize[,isColor]）
- filename: Specify the storage path and file name of the output target video. If the specified filename already exists, it will be overwritten.
- fourcc: Indicates the video encoding/decoding type (format). Use the function cv2.VideoWriter_fourcc() in OpenCV to specify the video encoding format. cv2.VideoWriter_fourcc() has 4 character parameters. These 4 character parameters constitute the "4-word token" of the codec, and each codec has one such token. A few commonly used tags are listed below.
  - cv2.VideoWriter_fourcc('I','4','2','0') indicates the uncompressed YUV color coding format, and the chroma subsampling is 4:2:0. This encoding format has good compatibility, but the generated file is relatively large, and the file extension is .avi.
  - cv2.VideoWriter_fourcc('P','I','M','I') indicates the MPEG-1 encoding type, and the extension of the generated file is .avi.
  - cv2.VideoWriter_fourcc('X','V','I','D') indicates the MPEG-4 encoding type. If you want the video size to be average, you can choose this combination of parameters. The file extension generated by this combination is .avi.
  - cv2.VideoWriter_fourcc ('T', 'H', 'E', 'O') indicates the Ogg Vorbis encoding type, and the file extension is .ogv.
  - cv2.VideoWriter_fourcc('F','L','V','I') means Flash video, and the extension of the generated file is .flv.
  For more character parameter combinations, you can check on the website http://www.fourcc.org. If the parameter fourcc is "-1", a dialog box will pop up when the program is running, as shown in the figure. In this dialog box, users can choose the appropriate compression program and compression quality according to their needs.
- fps: is the frame rate.
- frameSize is the length and width of each frame.
- isColor indicates whether it is a color image.

For example, the following statement completes the initialization of the cv2.VideoWriter class:

fourcc=cv2.VideoWriter_fourcc（*'XVID'）
out=cv2.VideoWriter（'output.avi',fourcc,20,（1024,768））

If you want to set the encoding/decoding format through the dialog box shown in the figure, you can use the statement:

fourcc=-1
out=cv2.VideoWriter（'output.avi',fourcc,20,（1024,768）)

(2) write method:

The function cv2.VideoWriter.write() in the cv2.VideoWriter class is used to write the next frame of video. The syntax of this function is:

None=cv2.VideoWriter.write（image）
- image: is the video frame to write. Usually, the color image format is required to be in BGR mode.

When calling this function, just pass the video frame to be written into this function.

For example, there is a video frame as frame, and to write it into the cv2.VideoWriter class object named out in the above example, use the statement:

out.write（frame）

The above statement will pass the frame into the out object named output.avi.

(3) release:

When the cv2.VideoWriter class object is not needed, it needs to be released. The function cv2.VideoWriter.release() is used to release this type of object. The syntax of this function is:

None=cv2.VideoWriter.release（）

For example, there is currently an object out of the cv2.VideoWriter class, which can be released with the following statement:

out.release（）

2. Save the video:

Saving a video includes multiple steps such as creating an object, writing the video, and releasing the object. The following is a brief introduction to each step.

create object

Before creating an object, the parameters need to be set first.
- Set the specific path to be saved, for example: filename=''out.avi''.
- Use cv2.VideoWriter_fourcc() to determine the encoding/decoding type, for example:
  - fourcc=cv2.VideoWriter_fourcc（ * ‘XVID’）。
- Determine the frame rate of the video, for example: fps=20.
- Determine the length and width of the video, for example: size=(640,480).
Then use the above parameters to create an object. For example:

out=cv2.VideoWriter（filename ,fourcc ,fps ,size）
write video

Use the function cv2.VideoWriter.write() to write the read video frame frame in the created object out. The code used is: out.write(frame)
release object

After the write is done, release the object out. The code is:

out.release（）

import cv2

cap = cv2.VideoCapture(0)
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('output1.avi', fourcc, 50, (int(cap.get(3)), int(cap.get(4))))

while cap.isOpened():
    rst, frame = cap.read()
    if rst is True:
        out.write(frame)
        cv2.imshow('frame', frame)
        if cv2.waitKey(1) == 97:
            break
    else:
        break

cap.release()
out.release()
cv2.destroyAllWindows()

Note: cv2.VideoWriter('output.avi',fourcc, 20.0, (fwidth,fheight)) The fwidth and fheight in should be consistent with the frame width and frame height of your original video

3. Basic application of video operation:

A video is composed of video frames, and the purpose of video processing can be achieved by extracting video frames from the video and processing them using image processing methods.

Example: Extract Canny edge detection results for a video.

import cv2

cap = cv2.VideoCapture(
    'https://vd3.bdstatic.com/mda-ijurqhuzpqac4fm8/sc/mda-ijurqhuzpqac4fm8.mp4?v_from_s=hkapp-haokan-suzhou&auth_key=1639325601-0-0-8ddbd7ccf330e4d61fbbd0dd792b3eb9&bcevod_channel=searchbox_feed&pd=1&pt=3&abtest=&klogid=2601737365')

while cap.isOpened():

    ret, frame = cap.read()
    frame = cv2.resize(frame, (640, 320))
    frame = cv2.Canny(frame, 100, 150)
    cv2.imshow('frame', frame)
    c = cv2.waitKey(100)
    if c == 97:
        break

cap.release()
cv2.destroyAllWindows()