Camera Engineer Says Camera - Data Format RAW, RGB(1)

Camera data format-RAW, RGB

Overview:
This article mainly talks about the basic knowledge related to Camera output data format. Through this article, you can understand the selection and principle of camera output format in camera projects.

Let’s start with the basic principles of camera operation

The camera is an optical device containing a photosensitive chip. As shown below is the composition of a typical camera component. Its overall structure is similar to the imaging principle of a convex lens. When the camera shutter is pressed, the light passes through the lens group and the filter, and the collected light is irradiated on the photosensitive matrix of the CMOS photosensitive chip, and the photosensitive chip records the lighting conditions at this time.

Camera composition (picture from Internet wiki)

There are a series of photosensitive components in the photosensitive chip. Each photosensitive element can be referred to as a pixel. Its main function is to sample the light in the field of view at the moment the camera shutter is pressed, and then quantize the sampled light to form a quantized value. Each small square in the picture below is a pixel in a photosensitive chip. These pixels are gathered together to sample the lighting conditions of an area, and then record the image within the field of view, and the photo is recorded. .

Return to the essence-several methods of expressing image color

RGB three primary colors

The physics teacher told us. All colors in nature can be obtained by combining the three primary colors of red, green, and blue:

Three primary colors (picture from Internet wiki)

Therefore, as long as the light we collect records the R\G\B values ​​within the field of view, we can restore the color of the actual object corresponding to the pixel.

How to record the RGB value of each pixel?

The birth of RAW RGB format

This requires the use of a component in the above-mentioned camera composition - a filter. The physics teacher told us that a lens of a given color can only pass through lenses of the same color. For example, a red lens can only transmit red light:

(The red lens can only pass red light - the picture comes from the Internet wiki)

Therefore, a different color filter is placed in front of each pixel. That is, each pixel in the illustration can only pass through one color. For example, the pixel at coordinates (0, 0) can only record red, and the pixel at coordinates (0, 1) can only record green:

In order to restore the actual color at coordinates (0,0), it requires three values ​​​​of R\G\B, but there is only a red filter at coordinates (0,0), and only the red data here can be obtained. what to do?
Therefore, when generating the color value of the pixel, it needs to borrow the green of the pixel at coordinates (0, 1) and the blue at coordinates (1, 1), thus combining the three values ​​​​of R, G, and B. (Don’t worry, this kind of borrowing has been verified. Although there is a small error in restoring the color of the point, it is completely fine).

Here we introduce the definition of the first data format-RAW RGB. RAW RGB refers to the sampling data formed after filtering by the above-mentioned filters, that is, each pixel only outputs a value corresponding to one color. This original R\G\B data is called RAW RGB format. At this time, the RAW RGB value corresponding to each pixel is:

• Coordinates (0, 0): R00 (suffix is ​​the horizontal and vertical coordinates)
• Coordinates (0, 1): G01
• Coordinates (1, 1): B11

RAW8 VS RAW10

If the above R00 is represented by an 8-bit value, it is RAW8. Similarly, if the above R00 is represented by a 10-bit value, it is RAW10.
There are also different standards for the value range of 8-bit data.

• If it is 0~255, it is the full range standard.
• If it is 16 ~240, it is the limited range standard.

True Color-Introduction of RGB888 and BGR888 formats

After such raw data RAW-RGB is generated, the camera can combine it through software or hardware to restore the color of the pixel. Typical sampling is that each R, G or B value is represented by 8 bits of data. This leads to the RGB888 format, that is, each pixel outputs its corresponding three primary color values ​​(borrowing the colors of other pixels). At this time, the corresponding value of the coordinate (0, 0) pixel is:

• The R\G\B value used at pixel coordinates (0, 0) is: R00+G01+B11.
  Of course, you can change the order of R\G\B data, for example: B11+G01+R00, which becomes data in BGR888 format.
  Some professional tools can extract the values ​​​​of the three components R, G, and B from the image:
  

(Separate R, G, B from the image - the picture comes from the Internet wiki)

RGB format with smaller data volume-RGB565, RGB555

In the RGB888 format, the data size corresponding to one pixel is 8bit + 8bit + 8bit, which can form 2^8*3 colors. Although the color is more lossless, the data volume is too large.
Allow a little error to reduce the size of the data.
The low bits of each pixel have less impact on the overall data than the high bits. For example, if the last two bits of the 8-bit bit are discarded, the error in the overall data of the 8-bit bit is just addition or subtraction {0, 4}. Based on this rule, the RGB565 format is created. Its data is cropped compared with RGB888, but the overall data volume is reduced from 24bits/pixel to 16bits/pixel. When the background uses this cropped data, it can restore the 24bits RGB888 value according to the agreed cropping rules:

Bit images corresponding to RGB888 and RGB565 (pictures from the Internet wiki)

The reason why green has one more bit than other colors is because some images are mainly viewed by the human eye. Studies have shown that the human eye is more sensitive to green, so more green is retained. Coincidentally, the pictures in some application scenarios are not visible to the human eye. Maybe the machine directly processes the data, or the scene itself does not have many green objects, so that the data can be cropped more evenly, such as this RGB555 Format, so that all three colors occupy 5 bits, and the highest bit defaults:

No matter which way, when we apply these RGB data types, we can restore the corresponding R, G, and B colors according to the agreed rules. For example, according to the storage method of RGB555, the values ​​of R, G, and B components can be extracted. can use:

R = color & 0x7C00, (获取高字节的5个bit) 
G = color & 0x03E0, (获取中间5个bit)
B = color & 0x001F, (获取低字节5个bit)

For another example, RGB565 is obtained by converting the 8bit R, G, and B components in RGB888:

short int rgb565_pixel; 
rgb565_pixel = ((R >> 3) << 11) | ((G >> 2) << 5) | (B >> 3);

RGB format expressed in bits

In addition to the above-mentioned RGB888, RGB555, and RGB565 methods of describing colors according to their corresponding Bit bits, there are also some countries that like to use the total number of digits to represent the corresponding RGB value (the world is such a big place, please give us a format name that you like) That’s OK too).

RGB24：

RGB24, one pixel is represented by 24 bits == 3 bytes, and the R, G, and B components are represented by 8 bits respectively. The typical one is the above-mentioned RGB888.

RGB32

Some processors require 32bits alignment when processing data to process image data quickly. Hence RGB32:

RGB4、RGB16

In some shooting scenes, there are not many colors that need to be recognized. The method of expressing colors can reduce the amount of data by defaulting some color bits. These types of methods work similarly to the representation methods described above. For specific representation methods, please consult the camera manufacturer for further confirmation.

When 3D effects or shadows become important-RGBA

In some game design or image processing, color formats with object outlines are used. The typical one is ARGB format, where:

ARGB data composition (picture from Internet wiki)

To be precise, the ARGB format is the ARGB8888 format (also known as ARGB32). ARGB is the abbreviation of Alpha, Red, Green, and Blue respectively. Because it contains 4 pieces of information near the pixel, it is also called a 4-channel representation method. Among them, alpha means the outline and transparency of the object. Only alpha images are grayscale images (examples of grayscale images will be discussed below), which have additional uses in some 3D games or image processing. For example, the difference between using RGB888 and using ARGB8888 images:

Comparison of ARGB data and RGB data (picture from Internet wiki)

Compared with RGB888, in the ARGB8888 picture, the shadow of the tree is shown below the small tree. Is 3D here yet, folks?

Small question: After understanding ARGB8888, I believe everyone should know what RGBA32 represents. Cowhide!

Summarize

1) This article starts with the basic principles of camera operation. The Camera sensor collects data of different colors through filters and generates RAW data.
2) Data in RAW format can also be divided into RAW8, RAW10, RAW12 and other types according to how many bits are used to represent each color.
3) RGB data is obtained by combining RAW data. According to the combination order, data in RGB, BGR and other formats can be combined.
4) Some application scenarios require small data bandwidth, or there are not many color types that need to be collected. In this case, RGB565, RGB555 and other formats with smaller data volumes can be used.
5) Some RGB format data named after the total data length include RGB24, RGB16, RGB32, etc.
6) Some special application scenarios also have Alpha values, so there are types of RGB data such as ARGB and RGBA.
(It’s not easy to code, thank you for likes or favorites, and unauthorized reprinting is prohibited)