ISP-AF related-focus area selection-sharpness evaluation

1. Lens related

Lens type

Zoom type:

Fixed focus, manual zoom, automatic zoom

aperture:

Fixed aperture, manual aperture, automatic aperture

Field of view:

Fisheye lens, super wide-angle lens, wide-angle lens, standard lens, telephoto lens, super telephoto lens (from large to small)

aperture:

Super starlight lens, starlight lens, universal lens

Interface Type:

M12, $\Phi$14, C, CS, Nikon F mount, Sony A mount, Canon E mount, etc.

Form function:

Spherical lens, aspheric lens, pinhole lens, fisheye lens

Autofocus refers to the process of controlling the movement of a stage or lens through a motor to change the image of an object from blurry to clearer.

zoom

Zoom usually refers to lengthening or shortening the focal length by moving the position of the lens element in the lens, also called ZOOM. Zoom can currently be divided into two types: optical zoom and digital zoom. Optical zoom does not sacrifice clarity, while digital zoom significantly sacrifices clarity.

optical zoom:

Optical zoom is to change the position of the focal point by moving the relative position of the lens inside the lens, change the length of the focal length of the lens, and change the viewing angle of the lens, so as to realize the enlargement and reduction of the image. If the position of the object being photographed remains unchanged, the focal length of the lens will be proportional to the magnification of the object. If there is one word to describe optical zoom, it is "telescope". (The imaging surface moves horizontally)

Digital zoom:

Digital zoom is to increase the area of each pixel in the picture through the processor of the camera phone, so as to achieve the purpose of zooming in, just like we forcefully enlarge the pixels of the image in image processing software such as ACDSEE, but this process It is carried out on the mobile phone and a part of the pixels on the original SENSOR are enlarged by interpolation. Different from optical zoom, digital zoom changes in the vertical direction of the SENSOR to give people a zoom effect. Since the focal length does not change during the entire process, the image quality will gradually decrease as the magnification ratio increases. (Scaling in the vertical direction of the imaging surface)

Digital zoom can also be divided into two types: interpolation algorithm zoom and pseudo digital zoom:

Interpolation algorithm zoom: Interpolation operation is performed on the image to expand the size of the image to the required specification. This algorithm is not ideal in terms of its effect, especially when it is used on a mobile phone, the camera on the mobile phone itself gets The data will have a lot of noise, and if interpolated again, the resulting image will be almost unusable.

Pseudo-digital zoom: When the camera is not in the maximum resolution format, for example, when a 1.3 million-pixel sensor uses a 640x480 specification to take pictures, the sersor is still set to work at a resolution of 1280x960, and then obtained by collecting the central part of the image A 640x480 photo makes the size of the object photographed appear to be doubled on the mobile phone. This method requires almost no additional algorithm support and has no impact on image quality. The disadvantage is that it can only be used in small size situations.

Focus(focus)

focus concept

When a convex lens is used as a camera lens, the clearest image generally does not fall on the focal point, or in other words, the distance (image distance) from the clearest image to the optical center is generally not equal to the focal length, but slightly greater than the focal length. The specific distance is related to the distance between the object being photographed and the lens (object distance). The larger the object distance, the smaller the image distance (but in fact it is always greater than the focal length).

Focusing actually adjusts the position of the entire lens (rather than the lens within the lens) to control the image distance so that the image is the clearest. The distance between the object to be photographed and the camera (lens) is not always the same. For example, if you want to photograph a person, sometimes, if you want to photograph the whole body, you should stay far away; If you want to get a clear image, you must change the distance from the photosensitive surface to the optical center of the lens as the object distance varies.

Due to the limitation of the human eye's resolution ability, the amount of blur within a certain range will not affect the observed imaging effect. For clear imaging, the maximum degree of blur perception of the human retina is the allowed blur amount s, and the value of s is 0. Between 03.0.04mm.

Focus classification

Focus can be divided into manual focus, auto focus and multi-point focus:

Manual focus is a focusing method that adjusts the camera lens by manually turning the focus ring to make the photos taken clear.

Autofocus allows the camera to automatically adjust the focusing distance of the lens based on the distance to the subject.

Multi-point focusing, also called area focusing, can be used when the focus center is not set in the center of the picture. Common multi-point focusing is 5-point, 7-point and 9-point focusing.

2. Focus area selection

If the window is too small, it is easy to lose important details of the image; if the window is too large, it will increase the interference of the background area and also increase the amount of calculation.

The focus window should be located in the main target area of the image. Commonly used window selection methods are mainly divided into two categories: static and dynamic. Static windowing methods include: center area windowing method, golden section cross area windowing method and inverted T-shaped windowing method, etc. Dynamic windowing methods include: Gaussian non-uniform sampling windowing method, first-order moment windowing method and artificial windowing method Eye visual salience mechanism windowing method, etc. The former usually selects a fixed area as the focus window according to a certain feature, and the size of the focus window is determined by the actual target image; the latter is to statistically analyze the information distribution of the image, give an optimal estimation area of the main scene, and then take a single or multiple windows as focus areas.

1. Center window method

The center windowing method believes that the target of interest to the human eye is always located in the center of the image, so the center part is used as the focus area. Calculate the image sharpness value in the focus area, and make the objects in the focus area the clearest according to the auto focus adjustment method to complete the auto focus. This focus area selection method is suitable for most situations, but if the object of interest deviates from the center of the image, the camera will not be able to focus on the object of interest, and the focus quality will be severely degraded)

The assumption of the center window method is that the main target is located in the center of the image. When the center of the image is a solid color background, it will cause focus failure. The length and width of the window rectangle are 1/4 of the length and width of the image, and the size of the center area is average. It is 1/16 of the image, and the calculation amount is reduced by 93.75%.

2. Window taking in inverted T-shaped area

When observing an image, the human eye usually pays attention to the middle and lower parts of the image for the first time. When composing a photograph, people usually place the subject in the middle and lower part of the entire scene. Therefore, the area in the middle and lower part of the image can be selected as the focus area, as shown in Figure 2.5. This method improves the success rate of covering the area of interest, but at the same time it introduces more background information, resulting in a corresponding increase in the amount of calculation of the image sharpness value.

3. Golden multi-point window selection method

In actual scenes, there are often multiple different objects whose textures and distances from the camera are different. The multi-point windowing method divides the image into several areas according to certain rules, and selects the area closer to the target of interest as the focus area of the image. The focus area selection is flexible and versatile, and the usage scenarios are richer.

In the optical imaging system, there is only one correct focal plane, and only at the focal plane position can a clear object image be obtained, and at other positions, the imaging of the target point will have spots. Considering the minimum angular resolution of the human eye, when the spot diameter is smaller than a certain value, it can be identified as a clear image point.

There are edges in the image, and whether an image is focused or not is related to the high-frequency components of the image edge information. When it is fully focused, the image is clear, the edge information is large, and the focus evaluation function value is the largest; when the image is out of focus, the image is blurred and the edge information Less, the value of the focus evaluation function is small.

Image focus evaluation function:

characteristic

1) Unbiasedness:

The peak value of the sharpness evaluation function curve should correspond to the clearest focus position, and the evaluation values of different defocus degrees also correspond to different defocus image acquisition positions.

2) Unimodality:

In the process from being away from the quasi-focus plane to the positive focus plane and then away from the quasi-focal plane, the focus evaluation function curve increases first and then decreases correspondingly, and the value is the largest at the exact focus position, which should meet the requirement of unimodality.

3) Sensitivity:

The steepness of the evaluation function curve reflects the sensitivity during the autofocus process. The flatter the curve, the lower the sensitivity. A higher sensitivity is desired during the autofocus process, and the curve should have a certain steepness.

4) Noise immunity:

Whether the curve can still maintain good shape characteristics under noise interference.

5) Real-time:

We hope that the auto-focusing process is a relatively short time. In order to meet the real-time requirements of the auto-focusing system, the designed algorithm should not be too complicated and the amount of calculation should not be too large.

Evaluation function

1)SMD function

The SMD function is called the sum of gray differences function. It calculates the image focus extreme point based on the sum of the differences between the adjacent left pixel point and the upper pixel point of a certain pixel point and this point.

$F_{SMD}=\sum_{i,j}(\lvert f(i,j)-f(i,j-1) \rvert +\lvert f(i,j)-f(i-1,j) \rvert )$

The SMD function algorithm is simple, fast, and has strong anti-interference ability, but its accuracy is not high.

2)Brenner function

Calculate the grayscale difference between adjacent k unit pixel points, and then obtain the square of the difference

$F_{Brenner}=\sum_{i,j}\lvert f(i+k,j)-f(i,j) \rvert ^2$

The Brenner function has low complexity and simple calculation process.

3) Tenengrad function

$F_{Tenengrad}=\sum_{i,j}(\lvert G_{i}^{2}(i,j)+G_{j}^{2}(i,j) \rvert )$

Among them, $G_{i}(i,j)$ 和 $G_{j}(i,j)$ is the first-order sobel operator difference in the i, j direction

4) EOG function square gradient

$F_{EOG}=\sum_{i,j}(\lvert f(i,j)-f(i,j-1) \rvert ^2 +\lvert f(i,j)-f(i-1,j) \rvert ^2)$

5) Robert operator gradient function

$F_{Robert}=\sum_{i,j}(\lvert f(i,j)-f(i+1,j+1) \rvert +\lvert f(i+1,j)-f(i,j+1) \rvert )$

The Roberts gradient function is calculated based on the sum of the pixel differences between all intersecting pixels in the focus area to calculate the focus point of the image, improving the SMD function, and the anti-interference performance is stronger than that of the SMD function

6) sobel operator gradient function

$F_{Sobel}=\sum_{i,j}\lvert G_{i}(i,j)+G_{j}(i,j) \rvert$

Among them, $G_{i}(i,j)$ 和 $G_{j}(i,j)$ is the first-order sobel operator difference in the i, j direction

7)Laplace operator function

$F_{Laplace}=\sum_{i,j}\lvert f(i-1,j)-f(i+1,j) + f(i,j-1)-f(i,j+1) - 4f(i,j) \rvert ^2$

Reference article: <Simulation Analysis of Auto-Focus Image Sharpness Evaluation Function Based on Gradient Calculation>
<Research on Auto-Focus System>

Combined with the previous calculation of sharpness, add some AF content