Computer Vision (2)-HSV color separation and target tracking

  HSV is a color space created by ARSmith in 1978 based on the intuitive characteristics of colors, also called the hexagonal cone model. The color parameters in this model are: hue (H), saturation (S), and lightness (V) . HSV can perceive colors more accurately than the traditional RGB color space, and still keep the calculation simple.
  The basic steps of HSV color separation are: convert the HSV representation, set the target threshold, set the mask, and filter the target color.

---

---

1. RGB color model

  The RGB color mode is a color standard in the industry. It obtains a variety of colors by changing the three color channels of red (R ), green (G ), and blue (B) and superimposing them with each other. Yes, RGB is the color representing the three channels of red, green, and blue. This standard includes almost all the colors that human vision can perceive, and it is one of the most widely used color systems.

Two, HSV color model

1. Color model

  HSV is a color space created by ARSmith in 1978 based on the intuitive characteristics of colors, also called the hexagonal cone model. The color parameters in this model are: hue (H), saturation (S), and lightness (V).
   Hue H : Measured by an angle, the value range is 0°~360°, starting from red and counting in a counterclockwise direction, red is 0°, green is 120°, and blue is 240°. Their complementary colors are: yellow is 60°, cyan is 180°, and purple is 360°.
   Saturation S : Saturation S indicates how close the color is to the spectral color. A color can be seen as the result of mixing a certain spectral color with white. Among them, the greater the proportion of the spectral color, the higher the color close to the spectral color, the higher the saturation of the color, the higher the saturation, and the darker and brighter the color. The white light component of the spectral color is 0, and the saturation is the highest. Usually the value range is 0%~100%, the larger the value, the more saturated the color.
   Brightness V : Brightness indicates the brightness of the color. For the color of the light source, the value is related to the brightness of the luminous body; for the color of an object, this value is related to the transmittance or reflectance of the object. Usually the value ranges from 0% (black) to 100% (white).

2. Conversion algorithm

  Let (r, g, b) be the red, green, and blue coordinates of a color, and their values ​​are real numbers between 0 and 1. Let max be equivalent to the largest of r, g, and b. Let min be equal to the smallest of these values. To find the (h, s, v) value in the HSV space, where h ∈ [0, 360) is the hue angle of the angle, and s, v ∈ [0,1] is the saturation and brightness of



  HSV to the user Said to be an intuitive color model. We can start with a pure color, that is, specify the color angle H, and let V=S=1, and then we can get the color we need by adding black and white to it. Increasing black can reduce V without changing S, and increasing white can reduce S without changing V. For example, to get dark blue, V=0.4 S=1 H=210 degrees. To get light blue, V=1 S=0.4 H=210 degrees.
  Generally speaking, the human eye can distinguish 128 different colors, 130 color saturations, and 23 shades. If we use 16Bit to represent HSV, we can use 7 bits to store H, 4 bits to store S, and 5 bits to store V, that is, 745 or 655 can meet our needs.
  Because HSV is a relatively intuitive color model, it is widely used in many image editing tools, such as Photoshop (called HSB in Photoshop), etc., but this also determines that it is not suitable for use in lighting models. Many light Hybrid calculations, light intensity calculations, etc. cannot be directly implemented using HSV.
  By the way, another intuitive color model is the HSL model, in which the first two parameters are the same as HSV, and L represents brightness. Its three-dimensional representation is a double pyramid.
  The specific values ​​of each color are as follows:

Third, the mask

  The mask concept in digital image processing is borrowed from the process of PCB plate making. In semiconductor manufacturing, many chip process steps use photolithography technology. The graphic "negative" used for these steps is called a mask, and its role is: in silicon An opaque pattern template is concealed in the selected area on the chip, and then the following corrosion or diffusion only affects the area outside the selected area.
  Image masking is similar to it, using selected images, graphics or objects to cover the processed image to control the image processing area or process .
  In digital image processing, the mask is a dimensional matrix array and sometimes multi-value images are used. The image mask is mainly used to:
  ① Extract the region of interest, and multiply the image to be processed with the pre-made region of interest mask. Get the impression For the image of the region of interest, the value of the image within the region of interest remains unchanged, while the value of the image outside the region is all 0 .
  ②Shielding function , using a mask to shield certain areas on the image so that they do not participate in the processing or calculation of processing parameters. Or only the shielded area is processed or counted.
  ③Structural feature extraction , using similarity variables or image matching methods to detect and extract structural features similar to the mask in the image.
  ④The production of special shape images .

Four, HSV color separation code implementation (python)

1. Introduce the library

import numpy as np
import cv2 

2. Import the original video

cap = cv2.VideoCapture('green.mp4') #打开同一目录下的视频
while(cap.isOpened()):
    ret, frame = cap.read() #frame保存视频每一帧
    if ret==True: #当读取成功时

        cv2.imshow('frame',frame)
        if cv2.waitKey(10) & 0xFF == ord('q'):
            break
    else:
        break

cap.release()
cv2.destroyAllWindows()

To facilitate browsing, all videos in this article are converted to gif format and uploaded.

3. Convert the original video to hsv color space

Conversion to hsv color space requires the use of function cv2.cvtColor(input_image, flag)

parameter			description			return value
input_image			The picture that needs to be converted			Image matrix after color space conversion
flag			Type of conversion

Type of conversion:

Types of	description
cv2.COLOR_BGR2GRAY	BGR -> Gray
cv2.COLOR_BGR2RGB	BGR -> RGB
cv2.COLOR_BGR2HSV	BGR -> HSV

The specific code is:

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV) 

The result after conversion is:

4. Get the mask

To calculate the mask, you first need to determine the thresholds of the colors first, and then use the cv2.inRang function to set the mask, leaving only the green part. The matrix expansion in the fourth row is just to visualize this step into a video, which is not necessary in actual operation.

lower = np.array([35, 43, 46])
upper = np.array([77, 255, 255])   #设定绿色的hsv阈值
mask2 = cv2.inRange(hsv, lower, upper)#设置掩模 只保留绿色部分
mask=np.stack([mask2] * 3, axis=2) #矩阵拓展

The mask result obtained is:

5. Filter the target color

After obtaining the mask, use the cv2.bitwise_and function to "AND" the mask with the original image to filter out the green color, and get the final result.

res = cv2.bitwise_and(frame, frame, mask = mask2) 

The filtered result is:

6. Complete code

import cv2

cap = cv2.VideoCapture('green.mp4') #打开原视频
fourcc = cv2.VideoWriter_fourcc(*'mp4v') #设置输出视频格式
fps =cap.get(cv2.CAP_PROP_FPS) #设置输出视频帧数
size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))) #视频尺寸

out2 = cv2.VideoWriter('green_hsv.mp4',fourcc, fps, size) #设置输出hsv视频
out3 = cv2.VideoWriter('green_mask.mp4',fourcc, fps, size) #设置输出mask视频
out4 = cv2.VideoWriter('green_res.mp4',fourcc, fps, size) #设置输出最终过滤视频

while(cap.isOpened()):
    ret, frame = cap.read()
    if ret==True:
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)  #rgb通道难以分离颜色 需要先转化到hsv色彩空间

        lower = np.array([35, 43, 46])
        upper = np.array([77, 255, 255])   #设定绿色的hsv阈值
    
        mask2 = cv2.inRange(hsv, lower, upper)#设置掩模 只保留绿色部分
        res = cv2.bitwise_and(frame, frame, mask = mask2 #利用掩模与原图像做“与”操作 过滤出绿色
        mask=np.stack([mask2] * 3, axis=2) #mask矩阵拓展

        out2.write(hsv) #保存hsv视频到本地
        out3.write(mask) #保存mask视频到本地
        out4.write(res) #保存最终视频到本地
        
        cv2.imshow('frame',frame) #显示原视频
        cv2.imshow('hsv',hsv) #显示hsv视频
        cv2.imshow('mask',mask) #显示mask视频
        cv2.imshow('res',res) #显示最终视频
        
        if cv2.waitKey(10) & 0xFF == ord('q'):
            break
    else:
        break

cap.release()
out.release()
out2.release()
out3.release()
out4.release()
cv2.destroyAllWindows()

Five, target tracking

  Through the hsv color separation, we can identify the green objects in the video, so we can further use this method to track the green objects in the video. The specific ideas are as follows:
  1. Use the open operation in morphology to remove the green noise in the video. Open detailed calculations .
  2. According to the two-dimensional (0-255) matrix obtained by the mask, the range of the object is obtained.
  3. Draw a rectangular box according to the scope of the object.

The code involved is as follows:

kernel = np.ones((10,10), np.uint8) #设置开运算所需核
opening = cv2.morphologyEx(mask2, cv2.MORPH_OPEN, kernel)  # 对得到的mask进行开运算
rectangle = np.where(opening == 255) #找出开运算后矩阵中为255的部分，即物体范围
cv2.rectangle(frame, (min(rectangle[1]), min(rectangle[0])), (max(rectangle[1]), max(rectangle[0])), (0, 0, 255), 3) #根据每一帧中物体的左上角坐标以及右下角坐标绘制矩形框

The final result is: