1. The hardware structure of Sensor
1. The structure of each pixel
There is a microlens at the top of each pixel to increase light transmission. If there is a lens, there is a problem of CRA. Light beyond a certain angle cannot be collected, and it needs to be matched with the lens.
In all camera products, the cmos sensor is the core device of the camera. It is located between the lens and the ISP processor, and converts the optical signal into a digital signal (electrical signal) that the ISP can process.
2. Vertical structure of sensor
Light passing through the circuit will be reflected, causing interference between each pixel. The back-illuminated structure (the photosensitive layer is on top of the circuit) is not affected by the circuit.
BI = Back illuminated – Corresponding to the front-illuminated
back-illuminated CIS, the structure is changed, and the position of the metal line and the photodiode are interchanged, so that the light passes through the upper lens, filter, and photodiode in sequence-reducing the metal line for The interference of wire tubes increases the amount of incoming light and reduces NOISE, and has obvious effects on scenes with insufficient light (commercialized in 2009)
3. CFA(color filter array)
Each pixel is covered with a color filter to perceive the brightness of each color. In order to be able to distinguish colors, a layer of filter film is designed on the silicon photosensitive area. The filter film above each pixel can pass through one of the three wavelengths of red, green and blue, and filter out the other two filters
. Light sheet CFA
Bayer format: the green component is half of the total number of pixels, and the red and blue components are a quarter of the total number of pixels.
3.1 MONO sensor
Due to the loss of brightness caused by light filtering, there is a kind of MONO sensor. Without the treatment of filters, the brightness will be greatly improved (some articles say that it can be increased by four times), but this can not perceive the color, so it is also called black and white. sensor.
3.2 Spectral response curve
The figure below is the spectral response curve of the sensor.
This is the response curve of the cone cells of the human eye to the spectrum
Response curve of cone cells of human eye to spectrum
Due to the different sensitivity curves of human eye and sensor to spectrum, the same object perceives different colors. So colors need a conversion relationship. The CCM (color correction matrix) in the ISP was born, using a 3x3 matrix to calibrate the RGB color perceived by the sensor to the RGB color we see.
3.3 Discoloration of traffic lights
Here is a question often encountered by surveillance cameras. When shooting red lights, they often shoot yellow and white lights. Why is this?
Because the spectrum of the red light is between 780 and 620nm, it can be seen from the spectral response curve that the red pixel is the most sensitive, then the green pixel, and finally the blue pixel.
When the sensitivity is relatively small, the brightness of red pixels is much greater than that of green and blue, and it must appear red. When the sensitivity is relatively high, the green pixels can also have high brightness, while the red pixels have already been saturated and are not increasing. At this time, the color presented is red + green = yellow. When the sensitivity is high, RGB is saturated, and the addition of red, green and blue is white. Solution:
Reduce exposure and replace with blue glass IRCUT: For red light above 650nm, the sensitivity of green pixels is also very strong. Switching to blue glass IR can effectively reduce the interference of green pixels. At the same time, it can also completely filter out red light with wavelengths above 630nm, reducing the problem of overexposure. 3.4 Demosaic Since each pixel of the sensor can only perceive one color, interpolation is required to obtain the RGB value of each pixel. This process is demosaic (Demosiac).
demosiac
4. micro lens:
1. Microlens in the sensor: A tiny lens is installed above the photosensitive area of the pixel, so the photosensitive area is no longer determined by the opening area of the photosensitive sheet, but by the surface area of the microlens. In this way, while taking into account the size of a single pixel, the aperture ratio is increased in terms of specifications, and the sensitivity is greatly improved.
4.1.1 Principle and function of Micro lens
The real size of each "pixel" is inside the red frame. The
red square marked Pixel in the picture is the real photosensitive area in CMOS. The surrounding T1, including VSS, are the positions of auxiliary transistors or interfaces. Compared with the blue frame And the red frame will find that in the area where a pixel is located, a large part of the area cannot be used for light sensing, which is a huge waste. The light that hits the area between the red and blue boxes in the picture can be focused into the deep well of the pixel by using the micro lens.
in:
Lens CRA: The angle from the mirror center to the imaging surface.
Sensor CRA: The angle of the maximum light that the micro lens can receive.
4.1.2. The influence of the microlens on the image:
The micro lens plays the role of collecting light, and the micro lens is not located directly above the pixel in each field of view. If the CRA (chief ray angle) value of the lens does not match the micro lens CRA value of the sensor, it will cause shading Severe or serious problems with imaging color shading.
5. IR-CUT
Everyone knows that the camera has an IR-CUT, so why add an IR-CUT filter? IR-CUT is an infrared filter that can filter out infrared light.
From the spectral sensitivity curve of IMX290, it can be seen that the sensor can perform photoelectric conversion at a wavelength of 1000, while the human eye can only reach 700nm, which means that the "visible light" of the sensor is different from that of the human eye, and the range is far larger The human eye is larger.
So what problems will it cause? The biggest problem is that the colors are different, because the human eye feels the response in the visible light range, but the sensor feels the response in the range of 400-1000. Therefore, we need to add an IR-cut to make the sensor feel the same band as the human eye.
There is another reason: when light enters the lens, due to the difference in refractive index, visible light and infrared light will be imaged at different positions on the target surface, and IR-CUT can reduce the artifacts produced by infrared light.
5.1 Expansion: RGB-IR sensor
Add a sensor, this sensor does not use IR-CUT, in addition to RGB three kinds of CFA, but also adds a CFA that only passes infrared light. From the spectrogram, we can know that the green pixels in the ordinary sensor can not only perceive green, but also perceive the light in the infrared band, which is actually green + IR (infrared light). The RGB-IR sensor has an IR pixel that only perceives infrared light, and subtracting the two can obtain the green color of visible light.
RGB-IR format
This replaces the IR-CUT with digital filtering. What I think is more applicable is that turning on the infrared light in the night scene can not only perceive infrared, but also perceive color, which is a black light technology to be developed.
6. Black level
Black level (Black Level
Correction) is also the lowest point of black. In terms of 8bit data, it refers to the video signal level without a line of light output on a calibrated display device. Define the corresponding signal level when the image data is 0.
The reason for the black level:
The information collected by the CMOS sensor undergoes a series of conversions to generate the original RAW format data. Taking 8bit data as an example, the effective value of a single pixel is 0-255, but the accuracy of the actual AD chip (analog-to-digital conversion chip) may not be able to convert a small part of the voltage value. A fixed offset is added before, so that the output pixel value is between 5 (non-fixed) and 255. The purpose is to fully preserve the details of the dark part, and of course some bright part details will be lost at the same time, because for the image In other words, our attention is more inclined to the dark area, and there will be many gain modules (LSC, AWB, Gamma, etc.) behind the ISP, so a little loss in the bright area is acceptable. There will be dark current in the circuit of the sensor itself, so that when there is no light, the pixel unit will also have a certain output voltage. The dark current is related to the exposure time and gain, and it is different in different positions. Therefore, when the gain increases, the gain of the circuit increases, and the dark current also increases, so many ISPs will choose to subtract different black level values under different gains. ISP
Black Level Correction in the pipeline:
Take several black images with different gains, and calculate the corresponding black level value for each gain. isp will subtract this black level value after the raw image. This is generally the first step for an ISP.
7. Point
Due to the process of the sensor chip, there are some defective pixels, which may be darker or brighter than adjacent pixels. Bad point correction in the ISP pipeline:
After a pixel is determined to be a bad point, a value is calculated based on the values of adjacent pixels to replace the bad point.
2. Exposure of Sensor
The exposure time and shutter time of the sensor are a concept, both of which are the photosensitive time of the sensor. The longer the exposure (shutter) time, the brighter the image. When the shutter time is controlled and the desired brightness is still not achieved, it is necessary to adjust the sensitivity (ISO) of the sensor. The sensitivity of the CMOS
sensor is its gain. The gain is amplified by analog or digital methods, which will inevitably amplify the noise, so the shutter is generally given priority when shooting. When shooting fast-moving objects or shaking hands, it is easy to produce motion blur, so you need to reduce the shutter time.
There are two exposure methods of the sensor, rolling shutter exposure and global exposure.
1. Rolling shutter
The sensor exposure starts to expose each row in turn, so the initial exposure time of each row in the image is inconsistent, and most sensors currently belong to this category. The picture below shows the phenomenon of rolling shutter exposure shooting high-speed moving objects.
Rolling shutter exposure distortion
This is the sensor exposure and output diagram of IMX290
The yellow area is the time when the sensor is sensitive to light, the blue area is the output time of each line of sensor data, and the red area is the time when the exposure starts. XHS is the smallest unit of time for each operation. In terms of time sequence, an image is exposed from the first row, after an XHS, it is exposed from the second row, and so on. After the exposure of the first line of the image is finished, the output is performed, and the time to output the data is one XHS, and so on.
Problems with Rolling Exposure
Horizontal stripes under fluorescent light
Fluorescent lamps are strobe lights, which are related to the local power supply frequency. If the local power supply is a 50Hz sine wave (domestic), the fluorescent lamp will flicker at a frequency of 100Hz. The initial exposure time of each row of the sensor is different, which leads to different brightness of each row when exposed. Hence the horizontal stripes
Solution:
The control exposure time is an integer multiple of 1/100, so that the exposure time of each row is an integer multiple of a period, and the brightness remains consistent. Controlling the frame rate to 25/50 frames cannot solve the problem of horizontal stripes, but it can fix the horizontal stripes in the same position in each frame of image. 1/25 is an integer multiple of 1/100, which can make the start time of exposure of each row in different images differ by four periods. It is guaranteed that the brightness of the same line in each image is consistent. There will be distortion when shooting fast-moving objects.
Because the exposure start time of each line is different, when the object moves quickly, the position of the object captured by each line is different, resulting in the deformation of the object.
Solution:
Increase the speed of image output (it is also possible to increase the frame rate, which essentially changes the output speed) and adjust the time and angle of the capture. For example, a vehicle in the distance moves fewer pixels per frame, so you can capture a vehicle a little further away. Sensors using global exposure
2. Global shutter
The start exposure and end exposure time of each row are the same, which is currently only used in electric police equipment and is expensive. Because the electric police equipment has flashing lights, the flashing lights are on for a short time. If you use a rolling shutter exposure sensor, the light will go out after a few lines of exposure for an image, so you must use a global exposure sensor.