OpenCV~feature extraction

Here are a few ways to extract color, shape and texture features from images

How to extract features from images? The first time I heard the term "feature extraction" was in a machine learning video tutorial on YouTube, which clearly explained how we extract features in large datasets.

Quite simply, the columns of a dataset are the features. However, when I came across the subject of computer vision, I was taken aback when I heard that we were going to extract features from images. Do you start going through each column of the image and take out each pixel?

After a while, I understood the meaning of feature extraction in computer vision. Feature extraction is part of the dimensionality reduction process, where an initial set of raw data is partitioned and reduced to more manageable groups.

In simple terms, for an image, each pixel is a piece of data, and what image processing does is to extract useful information from the image, thereby reducing the amount of data but retaining the pixels that describe the characteristics of the image.

All image processing does is extract useful information from an image, reducing the amount of data but preserving the pixels that describe the image's features.

In this article, let's explore several methods for extracting color, shape, and texture features from images. These methods are based on experience working with images, if there are any mistakes feel free to add or correct them!

1. Color

Color spaces are automatically the first place to explore whenever you work on an image project. Understanding the color space in which the image environment is set is critical to extracting the correct features.

Using OpenCV, we can convert the color space of an image to one of several options provided such as HSV, LAB, Grayscale, YCrCb, CMYK, etc. A simple breakdown of each color space:

a. HSV (Hue Saturation Value)

Hue : Describes the dominant wavelength and is the channel of the specified color
Saturation : Describes the hue/purity/hue of a color
Value : Describes the intensity of the color

import cv2  
from google.colab.patches import cv2\_imshow  
  
image = cv2.imread\(image\_file\)  
hsv\_image = cv2.cvtColor\(image, cv2.COLOR\_BGR2HSV\)  
cv2\_imshow\(hsv\_image\)

RGB and HSV color spaces

b. LAB

L : Describes the lightness of a color, used interchangeably with intensity
A : Range of color components, from green to magenta
B : color components from blue to yellow

import cv2  
from google.colab.patches import cv2\_imshow  
  
image = cv2.imread\(image\_file\)  
lab\_image = cv2.cvtColor\(image, cv2.COLOR\_BGR2LAB\)  
cv2\_imshow\(lab\_image\)

RGB and LAB color spaces

YCrCb

Y : Brightness obtained from RGB color space after gamma correction
Cr : Describes the distance of the red (R) component from brightness
Cb : Describes the distance between the blue (B) component and the brightness

import cv2  
from google.colab.patches import cv2\_imshow  
  
image = cv2.imread\(image\_file\)  
ycrcb\_image = cv2.cvtColor\(image, cv2.COLOR\_BGR2YCrCb\)  
cv2\_imshow\(ycrcb\_image\)

RGB and YCrCb color spaces

The importance of these color spaces is sometimes underestimated. In order to obtain relevant information from images, these color spaces provide an opportunity to identify whether features appear more different in each image. The craziest thing about color spaces is that we can perform addition/subtraction with different color spaces and you will be amazed by the results!

Another useful function for exploring the color space of an image is simply to use *numpy.mean()*, which gives the mean of each channel in the color space in the image dataset. This is especially useful if we want to see which channel in the color space dominates the dataset.

import cv2  
from google.colab.patches import cv2\_imshow  
import numpy as np  
import plotly.figure\_factory as ff  
  
# Check the distribution of red values   
red\_values = \[\]  
for i in range\(len\(images\)\):  
  red\_value = np.mean\(images\[i\]\[:, :, 0\]\)  
  red\_values.append\(red\_value\)  
  
# Check the distribution of green values   
green\_values = \[\]  
for i in range\(len\(images\)\):  
  green\_value = np.mean\(images\[i\]\[:, :, 1\]\)  
  green\_values.append\(green\_value\)  
  
# Check the distribution of blue values   
blue\_values = \[\]  
for i in range\(len\(images\)\):  
  blue\_value = np.mean\(images\[i\]\[:, :, 2\]\)  
  blue\_values.append\(blue\_value\)  
    
# Plotting the histogram  
fig = ff.create\_distplot\(\[red\_values, green\_values, blue\_values\], group\_labels=\["R", "G", "B"\], colors=\['red', 'green', 'blue'\]\)  
fig.update\_layout\(showlegend=True, template="simple\_white"\)  
fig.update\_layout\(title\_text='Distribution of channel values across images in RGB'\)  
fig.data\[0\].marker.line.color = 'rgb\(0, 0, 0\)'  
fig.data\[0\].marker.line.width = 0.5  
fig.data\[1\].marker.line.color = 'rgb\(0, 0, 0\)'  
fig.data\[1\].marker.line.width = 0.5  
fig.data\[2\].marker.line.color = 'rgb\(0, 0, 0\)'  
fig.data\[2\].marker.line.width = 0.5  
fig

Once we've identified or explored enough of the image's color space, and determined we're only interested in a single channel, we can use *cv2.inRange()* to mask out unwanted pixels. This is especially useful in the HSV color space.

import cv2  
from google.colab.patches import cv2\_imshow  
  
# Reading the original image  
image\_spot = cv2.imread\(image\_file\)  
cv2\_imshow\(image\_spot\)  
  
# Converting it to HSV color space  
hsv\_image\_spot = cv2.cvtColor\(image\_spot, cv2.COLOR\_BGR2HSV\)  
cv2\_imshow\(hsv\_image\_spot\)  
  
# Setting the black pixel mask and perform bitwise\_and to get only the black pixels  
mask = cv2.inRange\(hsv\_image\_spot, \(0, 0, 0\), \(180, 255, 40\)\)  
masked = cv2.bitwise\_and\(hsv\_image\_spot, hsv\_image\_spot, mask=mask\)  
cv2\_imshow\(masked\)

RGB vs HSV vs Masked images use cv2.inRange() to retrieve black points

Sometimes, we can even use *cv2.kmeans()_ to quantize the color of an image, essentially reducing the color to a few neat pixels. Depending on our goal, we can use _cv2.inRange()* to retrieve the target pixel. Usually, this function works like a charm in identifying important parts of an image, and I always check this function out before moving on to other color feature extraction methods.

import cv2  
from google.colab.patches import cv2\_imshow  
  
image\_spot\_reshaped = image\_spot.reshape\(\(image\_spot.shape\[0\] \* image\_spot.shape\[1\], 3\)\)  
  
# convert to np.float32  
Z = np.float32\(image\_spot\_reshaped\)  
# define criteria, number of clusters\(K\) and apply kmeans\(\)  
criteria = \(cv2.TERM\_CRITERIA\_EPS + cv2.TERM\_CRITERIA\_MAX\_ITER, 10, 1.0\)  
K = 2  
ret, label, center = cv2.kmeans\(Z, K, None, criteria, 10, cv2.KMEANS\_RANDOM\_CENTERS\)  
# Now convert back into uint8, and make original image  
center = np.uint8\(center\)  
res = center\[label.flatten\(\)\]  
res2 = res.reshape\(\(image\_spot.shape\)\)  
  
cv2\_imshow\(res2\)

Use cv2.kmeans() for color quantization (K=2)

2. Shape

Once we have fully explored the color features, we may at some point want to extract the shapes in the image.

For example, your task is to distinguish different types of wine glasses. Colors may not matter here, but shapes can tell us a lot about them.

Again, what I would do is convert the image to a different color space and see if any color spaces make the edges or shapes of objects stand out more. Then, we can use *cv2.findContours()* to retrieve all contours in the image. From here, we will examine all properties of the contour of interest.

Ideally, once we are able to extract the correct attributes that define the shape of the contours, we will apply this to all the images in the dataset, and the extracted digits will become our new non-image dataset. See how we can reduce the amount of data to just one column of shape features and still be able to explain our wine glass image?

Let's explore the many attributes we can extract from contours using OpenCV. As it has been shown before, I will provide the link here for reference:

https://docs.opencv.org/3.4/dd/d49/tutorial_py_contour_features.html

moment
contour area
contour perimeter
contour approximation
convex hull
convexity detection
rectangular border
Minimum circumscribed circle
fit ellipse
Fitting a straight line

In many cases, it was found that both cv2.HoughCircles() and cv2.SimpleBlobDetector() did not give accurate results when detecting circles. One of the reasons may be that the circles in the preprocessed image are not obvious enough. However, cv2.SimpleBlobDetector() still provides some handy built-in filters like inertia, convexity, circularity and area to retrieve circles as accurately as possible.

3. Texture

At some point, we may want to extract texture features since we have exhausted color and shape features. Both the gray level co-occurrence matrix (GLCM) and the local binary pattern (LBP) are texture features that I have used, but other texture features that you commonly use can also be commented below, I would like to know!

a. GLCM

It is difficult to understand the concept of GLCM specifically in terms of images. Statistically speaking, GLCM is a texture inspection method that considers the spatial relationship of pixels. It works by counting how often pairs of pixels with certain values and certain spatial relationships occur in an image, creating a GLCM, and then extracting statistical measures from this matrix.

An easy-to-use package that includes GLCM functionality is the _scikit-image_ package. In GLCM, we can also derive some statistics that describe more about the texture, such as:

Contrast : Measures the local variation of the grayscale co-occurrence matrix.
Correlation : Measures the joint probability of occurrence of a specified pair of pixels.
Square : Provides the sum of squares of the elements in the GLCM. Also known as uniformity or angular second moment.
Homogeneity : Measures how close the distribution of elements in a GLCM is to the diagonal of the GLCM.

import cv2  
from google.colab.patches import cv2\_imshow  
  
image\_spot = cv2.imread\(image\_file\)  
gray = cv2.cvtColor\(image\_spot, cv2.COLOR\_BGR2GRAY\)  
  
# Find the GLCM  
import skimage.feature as feature  
  
# Param:  
# source image  
# List of pixel pair distance offsets - here 1 in each direction  
# List of pixel pair angles in radians  
graycom = feature.greycomatrix\(gray, \[1\], \[0, np.pi/4, np.pi/2, 3\*np.pi/4\], levels=256\)  
  
# Find the GLCM properties  
contrast = feature.greycoprops\(graycom, 'contrast'\)  
dissimilarity = feature.greycoprops\(graycom, 'dissimilarity'\)  
homogeneity = feature.greycoprops\(graycom, 'homogeneity'\)  
energy = feature.greycoprops\(graycom, 'energy'\)  
correlation = feature.greycoprops\(graycom, 'correlation'\)  
ASM = feature.greycoprops\(graycom, 'ASM'\)  
  
print\("Contrast: \{\}".format\(contrast\)\)  
print\("Dissimilarity: \{\}".format\(dissimilarity\)\)  
print\("Homogeneity: \{\}".format\(homogeneity\)\)  
print\("Energy: \{\}".format\(energy\)\)  
print\("Correlation: \{\}".format\(correlation\)\)  
print\("ASM: \{\}".format\(ASM\)\)

Features obtained from Gray Level Co-occurrence Matrix (GLCM)

b. LBP

As there are already many articles explaining native binary mode, here to save your time and share the reference link here:

https://www.pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/
https://towardsdatascience.com/face-recognition-how-lbph-works-90ec258c3d6b

In a nutshell, LBP is a texture operator that labels pixels of an image by thresholding surrounding pixels and representing them with binary numbers. To our surprise, the LBP returned a grayscale image that clearly showed the textures in the image. Here, we try to break down the operations inside LBP based on our understanding:

For each central pixel, we try to compare with the surrounding pixels, and give them a label if the central pixel is larger or smaller than the surrounding pixels. As a result, we have 8 labels around, and by maintaining a consistent pattern of clockwise or counterclockwise throughout the image, we arrange them in a 2d array and convert them to binary numbers.

Such a matrix appears after we perform operations on each pixel of the entire image.

From here, we can see that the resulting matrix has the same shape as our original image, and we are able to draw and display the LBP as if it were an image. whaosoft aiot http://143ai.com

import cv2  
from google.colab.patches import cv2\_imshow  
  
class LocalBinaryPatterns:  
  def \_\_init\_\_\(self, numPoints, radius\):  
    self.numPoints = numPoints  
    self.radius = radius  
  
  def describe\(self, image, eps = 1e-7\):  
    lbp = feature.local\_binary\_pattern\(image, self.numPoints, self.radius, method="uniform"\)  
    \(hist, \_\) = np.histogram\(lbp.ravel\(\), bins=np.arange\(0, self.numPoints+3\), range=\(0, self.numPoints + 2\)\)  
  
    # Normalize the histogram  
    hist = hist.astype\('float'\)  
    hist /= \(hist.sum\(\) + eps\)  
  
    return hist, lbp  
  
image = cv2.imread\(image\_file\)  
gray = cv2.cvtColor\(image, cv2.COLOR\_BGR2GRAY\)  
desc = LocalBinaryPatterns\(24, 8\)  
hist, lbp = desc.describe\(gray\)  
print\("Histogram of Local Binary Pattern value: \{\}".format\(hist\)\)  
  
contrast = contrast.flatten\(\)  
dissimilarity = dissimilarity.flatten\(\)  
homogeneity = homogeneity.flatten\(\)  
energy = energy.flatten\(\)  
correlation = correlation.flatten\(\)  
ASM = ASM.flatten\(\)  
hist = hist.flatten\(\)  
  
features = np.concatenate\(\(contrast, dissimilarity, homogeneity, energy, correlation, ASM, hist\), axis=0\)   
cv2\_imshow\(gray\)  
cv2\_imshow\(lbp\)

Grayscale image and LBP representation

Similarly, we can store LBP in a histogram and treat it as a feature, which we can feed into a classifier for classification. Adrian Rosebrock of PyImageSearch did an amazing example of this!

I don't have much experience with texture features, but am interested in digging into it after gathering more information and trying to implement them in my project.

in conclusion

To sum up, in this article, the experience of three features that have been used in previous projects is shared, mainly color, shape and texture features. Along with code and results, trying to explain why I took each step. Hopefully you were able to learn something from image features, starting with color, shape, and texture.