Take a look Opencv: Mat format

I. Overview

We have many ways to acquire digital images from the real world: digital cameras, scanners, computed tomography and magnetic resonance imaging, to name a few. In any case, what we (humans) see are images. However, when converting this to a digital device, what we record is the numerical value of each point in the image.

For example, in the image above, you can see that a car's rearview mirror is nothing more than a matrix containing the intensity values ​​of all pixels. The way we obtain and store pixel values ​​may change depending on our needs, but ultimately all images in the computer world may be reduced to a matrix of numbers and other information describing the matrix itself. OpenCV is a computer vision library whose main focus is on processing and manipulating this information. So the first thing we need to be familiar with is how OpenCV stores and processes images.

2. Mat format

OpenCV has been around since 2001. At that time, the library was built around a C interface, and to store the image in memory they used a C structure called IplImage. This is what you'll see in most old tutorials and educational materials. The problem with this is that it exposes all the shortcomings of the C language. The biggest problem is manual memory management. It is built on the assumption that the user is responsible for handling memory allocation and deallocation. While this isn't a problem for smaller programs, once the codebase grows it will be more difficult to deal with all of these issues instead of focusing on solving our development goals.

Fortunately, C++ came along and introduced the concept of classes, which provides convenience to users through automatic memory management (more or less). The good news is that C++ is fully compatible with C, so changes will not create compatibility issues. Therefore, OpenCV 2.0 introduces a new C++ interface that provides a new way of doing things, which means you don't need to fiddle with memory management, making your code cleaner (less to write, more to implement). The main disadvantage of the C++ interface is that many embedded development systems currently only support the C language. So unless we're targeting an embedded platform, there's no point in using the old approach (unless we're a masochistic programmer and are asking for trouble).

The first thing we need to know about Mat is that we no longer need to manually allocate its memory and release it as soon as it is no longer needed. While it is still possible to do so, most OpenCV functions will automatically allocate their output data. If we pass an already existing Mat object that already has the required space allocated for the matrix, then it will be reused. In other words, we only use the memory required to perform the task at any time.

Mat is basically a class with two data parts: Matrix header (contains things like matrix size, methods for storage, Information such as the address of the storage matrix) and a pointer to a matrix containing pixel values ​​​​like (taking any dimension according to the selected storage method).

The size of the matrix header is constant, but the size of the matrix itself may vary from image to image, and is usually several orders of magnitude larger.

In fact, it is equivalent to a tiny data structure library.

OpenCV is an image processing library. It contains a large number of image processing functions. To solve computational challenges, most of the time we end up using multiple functions of the library. Therefore, it is a common practice to pass images to functions. We should not forget that we are talking about image processing algorithms, which tend to require a lot of calculations. The last thing we want to do is slow down the program further by making unnecessary copies of potentially larger images.

To solve this problem, OpenCV uses a reference counting system. The idea is that each Mat object has its own header, but a matrix can be shared between two Mat objects by having their matrix pointers pointing to the same address. Furthermore, the copy operator only copies the header file and the pointer to the large matrix, not the data itself. Many programming frameworks use this principle, just like the Qt library, which is copy-on-write.

Mat A, C; // creates just the header parts
A = imread(argv[1], IMREAD_COLOR); // here we'll know the method used (allocate matrix)
Mat B(A); // Use the copy constructor
C = A; // Assignment operator

Finally, all of the above objects point to the same single data matrix, and modifications using any one of them will also affect all the others. In fact, different objects simply provide different access methods for the same underlying data. However, their title parts are different. The really interesting part is that we can create headers that only reference part of the complete data. For example, to create a region of interest (ROI) in an image, just create a new title with new boundaries:

Mat D (A, Rect(10, 10, 100, 100) ); // using a rectangle
Mat E = A(Range::all(), Range(1,3)); // using row and column boundaries

Now you may ask, if the matrix itself may belong to multiple Mat objects, who is responsible for cleaning it when it is no longer needed. The short answer is: the last object to use it. This is handled using a reference counting mechanism. Whenever someone copies the header of a Mat object, the matrix's counter is incremented. This counter is decremented every time a header is cleared. When the counter reaches zero, the matrix is ​​released. Sometimes you also want to copy the matrix itself, so OpenCV provides the cv::Mat::clone() and cv::Mat::copyTo() functions.

Mat F = A.clone();
Mat G;
A.copyTo(G);

Modifying F or G now does not affect the matrix pointed to by A's head. What you need to remember is:

• Output image allocation for OpenCV functions is automatic (unless specified otherwise).
• We don't need to think about using OpenCV's c++ interface for memory management.
• The assignment operator and copy constructor only copy the header file.
• The underlying matrix of an image can be copied using the cv::Mat::clone() and cv::Mat::copyTo() functions.

3. Storage method

This is about how pixel values ​​are stored. We can choose the color space and the data type used. Color space refers to how we combine color components to encode a given color. The simplest one is grayscale and the colors we can use are black and white. These combinations allow us to create many shades of gray.

For colorful ways, we have more methods to choose from. Each of them breaks it down into three or four basic components, the combination of which we can use to create other components. The most popular one is RGB, mainly because this is also how our eyes construct colors. Its base colors are red, green and blue. To encode the transparency of a color, a fourth element is sometimes added: alpha (a).

However, there are many other color systems, each with their own advantages:

• RGB is the most common since our eyes use something similar, but keep in mind that the OpenCV standard display system uses the BGR color space (red and blue channels swap places) to combine colors.
• HSV and HLS break down colors into their hue, saturation and value/brightness components, which is a more natural way for us to describe colors. For example, we can ignore the last component to make our algorithm less sensitive to the lighting conditions of the input image.
• The popular JPEG image format uses YCrCb.
• CIE Lab* is a perceptually consistent color space, if you need to measure one given color to another Color distance, it will come in handy.

Each building component has its own valid domain. This results in the data type used. How a component is stored defines our control over its domain. The smallest data type possible is char, which means one byte or 8 bits. It can be unsigned (so it can store values ​​from 0 to 255) or signed (values ​​from -127 to +127). While in the case of three components this already provides 16 million possible colors to represent (just like in the case of RGB), we can do this by using float (4 bytes = 32 bits) or double for each component (8 bytes = 64 bits) data type for finer control. However, keep in mind that increasing the size of the component also increases the size of the entire image in memory.

4. Display the creation of Mat object

In the Loading, Modifying, and Saving Images tutorial, we have learned how to use the cv::imwrite() function to write a matrix to an image file. However, for debugging purposes, it is much more convenient to see the actual values. We can do this using Mat's << operator. Note that this only works for 2D matrices.

Although Mat works well as an image container, it is also a general matrix class. Therefore, multidimensional matrices can be created and manipulated. We can create Mat objects in various ways:

1. Constructor of Mat

 Mat M(2,2, CV_8UC3, Scalar(0,0,255));
 cout << "M = " << endl << " " << M << endl << endl;

For 2D and multi-channel images, we first define their size: number of rows and columns.

We then need to specify the data type used to store the elements and the number of channels per matrix point. To do this, we constructed multiple definitions based on the following conventions:

• CV_[number of bits per item][signed or unsigned][type prefix]C[channel number]

For example, CV_8UC3 means we use an 8-bit long unsigned char type, with three of these types per pixel to form three channels. There are predefined types, up to four channels. A scalar is a four-element short vector. Specifying it, we can initialize all matrix points with custom values. If you need more types, you can create the types using the macro above, setting the channel number in brackets as shown below.

2. Use C/c++ arrays and initialize them through the constructor

 int sz[3] = {
    
    2,2,2};
 Mat L(3,sz, CV_8UC(1), Scalar::all(0));

The above example shows how to create a matrix in more than two dimensions. Specify its dimensions, then pass a pointer containing the size of each dimension, leaving the rest unchanged.

3. Use cv::Mat::create function

 M.create(4,4, CV_8UC(2));
 cout << "M = "<< endl << " " << M << endl << endl;

Matrix values ​​cannot be initialized with this construct. It reallocates matrix data memory only if the new size does not fit the old size.

4. Use MATLAB style initializers cv::Mat::zero, cv::Mat::ones, cv::Mat::eye.

Specify the size and data type to use:

 Mat E = Mat::eye(4, 4, CV_64F);
 cout << "E = " << endl << " " << E << endl << endl;
 Mat O = Mat::ones(2, 2, CV_32F);
 cout << "O = " << endl << " " << O << endl << endl;
 Mat Z = Mat::zeros(3,3, CV_8UC1);
 cout << "Z = " << endl << " " << Z << endl << endl;

5. For small matrices, we can use comma-separated initializers or initializer lists (the last case requires c++11 support):

 Mat C = (Mat_<double>(3,3) << 0, -1, 0, -1, 5, -1, 0, -1, 0);
 cout << "C = " << endl << " " << C << endl << endl;
 C = (Mat_<double>({
    
    0, -1, 0, -1, 5, -1, 0, -1, 0})).reshape(3);
 cout << "C = " << endl << " " << C << endl << endl;

6. Create a new header for the existing Mat object and cv::Mat::clone or cv::Mat::copyTo.

 Mat RowClone = C.row(1).clone();
 cout << "RowClone = " << endl << " " << RowClone << endl << endl;

Please note

We can fill the matrix with random values ​​using the cv::randu() function. You need to give the random values ​​a lower and upper bound:

 Mat R = Mat(3, 2, CV_8UC3);
 randu(R, Scalar::all(0), Scalar::all(255));

5. Output format

In the example above, we can see the default formatting options. However, OpenCV allows you to format your matrix output:

Default
cout << "R(default)= " << endl << R << endl << endl ;

Python
cout << "R (python) = " << endl << format(R, Formatter::FMT_PYTHON) << endl << endl;

逗号分隔值(CSV)
cout << "R (csv) = " << endl << format(R, Formatter::FMT_CSV) << endl << endl;

Numpy
cout << "R (numpy) = " << endl << format(R, Formatter::FMT_NUMPY) << endl << endl;

C
cout << "R © = " << endl << format(R, Formatter::FMT_C) << endl << endl;

6. Output of other common projects

OpenCV also provides support for the output of other common OpenCV data structures through the << operator:

1. 2D point:

 Point2f P(5, 1);
 cout << "Point (2D) = " << P << endl << endl;

2. 3D points:

 Point3f P3f(2, 6, 7);
 cout << "Point (3D) = " << P3f << endl << endl;

3. std::vector via cv::Mat

 vector<float> v;
 v.push_back( (float)CV_PI); v.push_back(2); v.push_back(3.01f);
 cout << "Vector of floats via Mat = " << Mat(v) << endl << endl;

4. std::vector of points

 vector<Point2f> vPoints(20);
 for (size_t i = 0; i < vPoints.size(); ++i)
 vPoints[i] = Point2f((float)(i * 5), (float)(i % 7));
 cout << "A vector of 2D Points = " << vPoints << endl << endl;