In a world flooded with images, the ability to measure and quantify similarities between images has become a critical task. Whether used for image retrieval, content recommendation or visual search, image similarity methods play a crucial role in modern applications.
Fortunately, Python provides a wealth of tools and libraries that make it easy for developers and researchers to explore and implement these methods. In this blog, we will take a deep dive into various image similarity techniques and demonstrate how to implement them using Python.
Understanding image similarities
Image similarity can be viewed as a numerical representation of how similar two images are in terms of visual content. Images can be similar in multiple dimensions such as color, shape, texture, and composition. To quantify these similarities, various mathematical and computational methods are employed, allowing us to compare and classify images efficiently.
Popular image similarity methods
Histogram-based method
Feature-based methods
Deep learning based methods
Histogram-based method
A histogram captures the distribution of pixel values in an image. By comparing the histograms of two images, their similarity can be measured.
Histogram intersection and histogram correlation are metrics commonly used for this purpose. Python's OpenCV library provides tools for calculating and comparing histograms.
Here we will demonstrate a simple example using a histogram-based approach and the opencv-python library:
import cv2
# Load images
image1 = cv2.imread(image1)
image2 = cv2.imread(image2)
hist_img1 = cv2.calcHist([image1], [0, 1, 2], None, [256, 256, 256], [0, 256, 0, 256, 0, 256])
hist_img1[255, 255, 255] = 0 #ignore all white pixels
cv2.normalize(hist_img1, hist_img1, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX)
hist_img2 = cv2.calcHist([image2], [0, 1, 2], None, [256, 256, 256], [0, 256, 0, 256, 0, 256])
hist_img2[255, 255, 255] = 0 #ignore all white pixels
cv2.normalize(hist_img2, hist_img2, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX)
# Find the metric value
metric_val = cv2.compareHist(hist_img1, hist_img2, cv2.HISTCMP_CORREL)
print(f"Similarity Score: ", round(metric_val, 2))
# Similarity Score: 0.94Structural Similarity Index (SSIM)
SSIM is a widely used metric that evaluates the structural similarity between two images. It takes into account brightness, contrast and structure, giving a score between -1 (dissimilar) and 1 (same). The scikit-image library in Python provides an implementation of SSIM. Here we will demonstrate a simple example using SSIM and the scikit-image library:
import cv2
from skimage import metrics
# Load images
image1 = cv2.imread(image1)
image2 = cv2.imread(image2)
image2 = cv2.resize(image2, (image1.shape[1], image1.shape[0]), interpolation = cv2.INTER_AREA)
print(image1.shape, image2.shape)
# Convert images to grayscale
image1_gray = cv2.cvtColor(image1, cv2.COLOR_BGR2GRAY)
image2_gray = cv2.cvtColor(image2, cv2.COLOR_BGR2GRAY)
# Calculate SSIM
ssim_score = metrics.structural_similarity(image1_gray, image2_gray, full=True)
print(f"SSIM Score: ", round(ssim_score[0], 2))
# SSIM Score: 0.38The main disadvantage of the SSIM method compared to the histogram method is that the images must have the same dimensions. Even if the similarity score is low, we can remove background and transparency from the image to improve the similarity score.
Feature-based methods
These methods extract salient features from images, such as edges, corners, or key points. Techniques such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF) identify unique points in images that can then be compared between images. SIFT and SURF can be done using the opencv-python library.
Deep learning based methods
Deep learning has revolutionized the image similarity task. Deep features can be extracted from images using pre-trained convolutional neural networks (CNNs) such as ResNet, VGG and Inception.
openAI's CLIP (Contrastive Language-Image Pretraining) is an impressive multi-modal zero-shot image classifier that achieves impressive results in multiple domains without the need for fine-tuning. It applies recent advances in large-scale transformers (such as GPT-3) to the vision domain.
We can fine-tune these models using the torch, open_clip and sentence_transformers libraries, training them with our own image and text data. ScrapeHero can help prepare your own image datasets to train these models. Its web crawler service can crawl complex websites and provide high-quality data.
Here we will demonstrate a simple example using a CLIP-based pre-trained model and the torch, open_clip and sentence_transformers libraries:
!pip install git+https://github.com/openai/CLIP.git
!pip install open_clip_torch
!pip install sentence_transformers
import torch
import open_clip
import cv2
from sentence_transformers import util
from PIL import Image
# image processing model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-16-plus-240', pretrained="laion400m_e32")
model.to(device)
def imageEncoder(img):
img1 = Image.fromarray(img).convert('RGB')
img1 = preprocess(img1).unsqueeze(0).to(device)
img1 = model.encode_image(img1)
return img1
def generateScore(image1, image2):
test_img = cv2.imread(image1, cv2.IMREAD_UNCHANGED)
data_img = cv2.imread(image2, cv2.IMREAD_UNCHANGED)
img1 = imageEncoder(test_img)
img2 = imageEncoder(data_img)
cos_scores = util.pytorch_cos_sim(img1, img2)
score = round(float(cos_scores[0][0])*100, 2)
return score
print(f"similarity Score: ", round(generateScore(image1, image2), 2))
#similarity Score: 76.77The similarity between images can then be calculated based on the cosine similarity or Euclidean distance of these feature vectors. To improve accuracy, we can preprocess the image.
Application areas
The main applications of image similarity technology include e-commerce product matching, image retrieval, object recognition and face recognition. For example, in image retrieval, image similarity can be used to find images that are similar to a query image. Image similarity can be used in object recognition to match a given object against a known database. Image similarity algorithms can be used to identify individuals by comparing their faces to a database.
in conclusion
In today's vision-driven world, the ability to measure image similarity is an important part of many applications. This blog introduces you to various image similarity methods from simple histogram-based methods to complex deep learning techniques. You can also explore Siamese networks, a class of neural networks designed for one-shot learning and image similarity tasks.
Python, with its rich ecosystem of libraries such as scikit-image, opencv-python, TensorFlow, and PyTorch, enables developers and researchers to implement these methods efficiently. By experimenting with these techniques, the door will open to creating innovative applications that harness the power of image similarity.
· END ·
HAPPY LIFE
This article is for learning and communication only. If there is any infringement, please contact the author to delete it.