With OpenCV, you can do almost every computer vision task you can think of. Real-life problems require the simultaneous use of many computer vision algorithms and modules to achieve the desired results. So, you just need to know which OpenCV modules and functions to use to get what you want.
Let's take a look at the features that are available out of the box in OpenCV.
1 Built-in data structures and input/output
One of the greatest advantages of OpenCV is that it provides many built-in primitives to handle operations related to image processing and computer vision. If you have to program from scratch, you have to define Image, Point, Rectangle, etc. These are the basis of almost any computer vision algorithm.
OpenCV comes with all these basic structures, they are included in the core module. Another advantage is that these structures are already optimized for speed and memory, so you don't have to worry about their implementation details.
The imgcodecs module handles reading and writing of image files. When you operate on an input image and create an output image, you can save it as a .jpg or .png file with simple commands.
When using a camcorder, you will be dealing with a large number of video files. The videoio module handles all operations related to the input and output of video files. You can easily capture video from your webcam, or read video files in many different formats. You can even save many frames as a video file by setting properties like frames per second, frame size, etc.
2 Image processing operations
When writing computer vision algorithms, there are many basic image processing operations that you will use over and over again. Most of these functions are in the imgproc module. You can perform operations such as image filtering, morphological operations, geometric transformations, color conversions, image rendering, histograms, shape analysis, motion analysis, feature detection, and more.
Let's look at Figure 1-3.
The image on the right is a rotated version of the image on the left, and we can achieve this transformation with one line of code in OpenCV.
There is another module called ximgproc that contains advanced image processing algorithms that can be used for processing such as edge detection for structured forests, domain transform filters, adaptive manifold filters, etc.
3 GUI
OpenCV provides a module called highgui that can be used to handle all advanced user interface operations. Assuming you're working on a problem and want to check how the image looks before moving on, this module has the ability to create windows to display images and videos.
It has a wait function that waits for you to press a key on the keyboard before going to the next step. There is also a function to detect mouse events, which is very useful when developing interactive applications.
Using these functions, you can draw rectangles on those input windows, and then process according to the selected area, take Figure 1-4 as an example.
As you can see, we draw a green rectangle on the window. Once the coordinates of this rectangle are obtained, the area can be manipulated individually.
4 Video Analysis
Video analysis includes tasks such as analyzing motion between consecutive frames in a video, tracking different objects in a video, creating video surveillance models, etc. OpenCV provides a module called video that handles all these tasks.
There is also a module called videostab that handles video stabilization. Video stabilization is very important because when you're shooting video with a handheld camera, there's usually a lot of jitter to correct. All modern devices use video stabilization to process video before presenting it to the end user.
5 3D reconstruction
3D reconstruction is an important topic in computer vision. Given a set of 2D images, we can use correlation algorithms to reconstruct a 3D scene. In the calib3d module, OpenCV provides algorithms that find the relationship between various objects in these 2D images and calculate their 3D positions.
This module also handles camera calibration, which is crucial for estimating the parameters of a camera. These parameters define how the camera sees the scene in front of it. We need to know these parameters to design the algorithm, otherwise we may get unexpected results.
Please see Figure 1-5.
As we can see here, the same object is captured from multiple locations. Our job is to reconstruct the original object using these 2D images.
.6 Feature extraction
As we discussed earlier, the human visual system tends to extract the main features from a given scene, and then remember it, which facilitates subsequent retrieval. In order to imitate this, people began to design various feature extractors to extract these feature points from a given image. Popular algorithms include Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Features From Accelerated Segment Test (FAST).
An OpenCV module called features2d provides functions to detect and extract all these features. Another module called xfeatures2d provides more feature extractors, some of which are still experimental. You can try to use them if you have the chance.
There is also a module called bioinspired that provides algorithms for biologically inspired computer vision models.
7 Object Detection
Object detection refers to detecting the location of an object in a given image. This process is independent of the object type. If you design a chair detector, it won't tell you whether the chair in a given image is red with a high back or blue with a low back, it will only tell you where the chair is.
Detecting the location of objects is a critical step in many computer vision systems.
Take Figure 1-6 as an example.
If you run a chair detector on this image, it will place a green box around all the chairs, but it won't tell you what the chairs look like.
Object detection used to be a computationally intensive task due to the varying number of calculations required to perform detection at various scales. To solve this problem, Paul Viola and Michael Jones proposed a nice algorithm in their seminal 2001 paper (https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr -01.pdf ), which presents a method for quickly designing object detectors for any object.
OpenCV comes with modules called objdetect and xobjdetect, which provide a framework for designing object detectors, and you can use them to develop detectors for any object, such as sunglasses, boots, etc.
8 Machine Learning
Machine learning algorithms are widely used to build computer vision systems for object recognition, image classification, face detection, visual search, and more.
OpenCV provides a module called ml, which bundles many machine learning algorithms, including Bayesian classifier (Bayes classifier), k-nearest neighbor (KNN for short), support vector machine (support vector machine, SVM for short), decision tree (decision tree), neural network (neural network), etc.
It also has a module called the Fast Approximate Nearest Neighbor Search Library (FLANN), which contains algorithms for fast nearest neighbor searches in large datasets.
9 Computational Photography
Computational photography refers to the use of advanced image processing techniques to improve images captured by cameras. Computational photography does not focus on optical processes and methods of image capture, but uses software to manipulate visual data. Its applications include high dynamic range imaging, panoramic images, image fill light and light field cameras, etc.
Take Figure 1-7 as an example.
Check out these vivid colors! This is an example of a high dynamic range image, which cannot be achieved using traditional image capture techniques. This image had to be created by capturing the same scene in multiple exposures, registering these images with each other, and blending them well.
The photo and xphoto modules contain various algorithms providing algorithms related to computational photography. There is also a module called stitching which provides algorithms for creating panoramic images.
10 Shape Analysis
The concept of shape is crucial in computer vision. We analyze visual data by recognizing various shapes in images. In fact, this is an important step in many algorithms.
Say you're trying to recognize a specific logo in an image. You know it can come in all shapes, orientations and sizes. A good way to start is to quantify the shape characteristics of an object.
The shape module provides all the algorithms for extracting different shapes, measuring the similarity between them, transforming object shapes, etc.
11 Optical flow algorithm
Optical flow algorithms are used to track features in consecutive frames in a video. Suppose you want to track a specific object in a video. Running a feature extractor on every frame is very computationally expensive and the process can be slow. So you just extract features from the current frame and track those features in successive frames.
Optical flow algorithms are heavily used in video-based computer vision applications. The optflow module contains all the algorithms needed to perform optical flow operations. There is also a module called tracking that contains more algorithms that can be used to track features.
12 Face and Object Recognition
Face recognition refers to identifying a person in a given image. This is different from face detection, where only the location of a face in a given image needs to be identified.
If you want to build a practical biometric system that can recognize the person in front of the camera, you first need to run a face detector to identify the location of the face, and then run a separate face recognizer to identify who the person is. There is an OpenCV module called face for dealing with face recognition.
As we discussed earlier, computer vision attempts to model algorithms the way humans perceive visual data. Therefore, it will be helpful to find salient regions and objects in an image, which can help us in different applications such as object recognition, object detection and tracking, etc. A module called saliency is designed for this purpose. It provides algorithms that can detect salient regions in still images and videos.
13 surface matching
There are a growing number of devices capable of capturing the 3D structure of objects around us, capable of capturing depth information as well as regular 2D color images. Therefore, it is very important for us to build algorithms that can understand and process 3D objects.
A good example of a device that captures depth information and visual data is the Kinect, which is now able to recognize incoming 3D objects and match them to models in a database. If we have a system that can recognize and locate objects, it can be used in many different applications.
A module called surface_matching contains algorithms for 3D object recognition, as well as pose estimation algorithms using 3D features.
14 Text Detection and Recognition
Recognizing text in a given scene and recognizing its content is becoming more and more important, with applications ranging from license plate recognition, identifying road signs for self-driving cars, book scanning to digitize content, etc.
A module named text contains various algorithms dealing with text detection and recognition.
15 Deep Learning
Deep learning has had a big impact on computer vision and image recognition, and it has higher accuracy than other machine learning and artificial intelligence algorithms. Deep learning is not a new concept; it was proposed around 1986, but was revolutionized around 2012 when new GPU hardware was developed for parallel computing and Convolutional Neural Network (CNN) implementations. Optimization, along with other techniques, make it possible to train complex neural network architectures in a reasonable amount of time.
Deep learning can be applied to a variety of use cases such as image recognition, object detection, speech recognition, and natural language processing. OpenCV has been implementing deep learning algorithms since version 3.4, and in the latest version, importers for several important frameworks such as TensorFlow and Caffe have been added.