[] Multisensor fusion distinctions of Xuecheng multisensor fusion study notes (a) - a pinhole camera model

Pinhole camera model

Pinhole camera imaging model

Pinhole camera (pinhole camera) physical model as follows:

the photosensitive surface of the left side is called an image plane , and the pinhole (pinhole) referred to the camera center . The distance between the center of the camera image plane is called the focal lengthf .
Points on the three-dimensional world of the target object $P$ using the light passing through the projection center of the camera can be mapped to a point on the image plane $P^{'}$ , As shown in FIG.

Wherein, the mapping relationship may be expressed by the following equation:

Using these equations, we can and the three-dimensional position of the camera focal length of the actual object in space, calculates the actual two-dimensional position of the object on the imaging plane. However, please note that the coordinates obtained $x'$ And $Y '$ Only "measurement coordinate" rather than true pixel position.

One problem with a pinhole camera is insufficient to generate an image on the photosensitive sensor of light passing through the pinhole. To increase the light can be achieved simply by expanding a pinhole, as shown below. However, this would target object light and other light rays reflected by the reflective portion overlap each other, resulting in image blur effect: i.e., the brighter the pinhole opening larger images, but at the same time, the object on the imaging plane blur more serious .
One way to solve this problem is to use a lens , it is possible to capture a plurality of light rays emitted from the target point with the object. Next we look at the lens and iris.

Lens and aperture

An appropriate size and location of the lens (a lens) can be a point on the target object $P_1$All refract light emitted to a point on the image plane ${P_1}'$ On. However, the light passing through the center of the refractive lens does not occur, they move in a straight line until it intersects with the imaging plane.

Other object distance closer or more distant point, such as $P_2$, The concentration phenomenon occurs not focus on the image plane, as a set of rays emitted from them can not be concentrated in one point by the lens, but converge on a circle of a certain radius. This vague tact is often called Circle of Confusion (COF). To reduce blurring, you can use a diaphragm , which is open and a lens with a center of a typical size adjustable, placed directly behind the lens. The following figure illustrates the principle:

Pincushion and barrel distortion

By reducing the diameter of the aperture of the lens through the outer edge of the light is blocked, thereby reducing the size of the COF on the imaging plane. Obviously, smaller aperture blur can be reduced, but at the cost of reduced light sensitivity. The larger the aperture, the more light is converged to the imaging region, a better signal to noise ratio produced brighter images, the disadvantage is possible to increase the degree of image blurring.
So how do we calculate the position of space objects in an image of it? Given space in a three-dimensional point, which may be similar to the pinhole camera through the lens of two-dimensional position on the imaging plane manner calculated. In fact, the lens due to the different types of lenses and cause image distortion. Closest to the distortion of reality we call "radial distortion." This is because the focus lens is non-uniform in diameter caused. Thus, the amplification effect depends on the camera's lens center (optical axis) distance between the light through the lens. If the magnification is increased , distortion effect produced is called " pincushion distortion (Pincushion Distortion) "; if the magnification reduction , it is called " barrel distortion (Barrel Distortion) ." When using a wide-angle lens barrel distortion usually it occurs. In the following figures, it illustrates how the two types of distortion.

Image calibration and calibration concepts

When information is extracted from the camera images, many applications need to obtain a target object from an image (such as a vehicle) spatial location. For this purpose, it is necessary to remove or at least mitigate the effects of lens distortion. This process is called calibration (Calibration) . For each camera lens configuration, a calibration procedure must be performed, so that it can calculate the distortion parameters. This is usually accomplished by a known set of objects photographed pictures, such as planar checkerboard pattern. From these known geometry can be deduced that all the lens and the image sensor parameters. Process eliminate distortion in the image from the camera is called calibration (Rectification) . The following figure shows the calibration image correction apparatus of the present course. You can easily see both sides of the straight line are a serious distortion.
However, details of the distortion correction, we are no longer here-depth discussion. However, when using your camera settings, if accurate measurement of space or object reconstruction is the goal, you must perform a calibration procedure.

Three-dimensional coordinate system to the coordinate system of conversion pixels

As described above, in the three-dimensional space, the image point of the projection plane does not correspond directly to the actual figures we can see in the image, the digital image is actually composed of thousands of pixels (pixel) components. For discrete pixel image is how to express understanding, we need to look closely for the camera model again. As shown below, the camera center points in space $O$ positions, and has its own coordinate system, the coordinate axes are $i$、 $j$、 $k$ , which $k$ -axis points to the imaging plane. $k$ location point axis intersecting the imaging plane $C'$ , Point $C'$ Called the principal point (Principal Point), it represents the center of the image coordinate system.

The point in space $P$ mapped to the image plane after the first step is obtained by subtracting the principal point coordinates, so that the discrete images to have lower left corner of the image plane center own coordinate system, as shown in FIG.

The second step is to convert from a conversion process measurement coordinate system to the pixel coordinate system. We can use the parameters provided by the calibration process $k$ and $l$ mapping formula to achieve the conversion from the measured values of the pixels, and these parameters can easily be integrated into shown in the following figure. Note: In the image coordinate system, the coordinate origin at the upper left corner thereof, and $Y$ -axis positive direction pointing straight downward, $x$ -axis positive direction toward the front-right.
In subsequent chapters, we will LIDAR 3D point cloud is mapped to the camera image. We will achieve this goal by the above formula. In addition, the focal length $f$ and $k$ and $l$ the product (also referred to as $\alpha$ and $\beta$ ) will be used in a calibration matrix, which greatly simplifies the mapping operation.

One final calibration image should be noted: in many applications (e.g., feature tracking), the processing of the original image is significant, because it avoids the calibration image and the calibration image after pixel conversion is not completely fall into discrete pixels calculating an error caused by the center process. In this case, it is recommended locating feature in the original image are unmodified, using the above equation and the results obtained coordinate transformation. When using a set of weight training weights depth study based on an image calibrated prior to import images into the network makes sense - if we use the original image, distortion (for example, caused by the use of fisheye lens) will lead to detection errors because these networks usually use images without distortion generated training.

And Bayer array image sensor (Bayer Pattern)

In this section, you will have to understand how the specific wavelength of light is converted into a digital storage medium may be a color pixel.

When the camera captures an image, light travels through the lens and onto the image sensor. Such a sensor by the photosensitive elements, which records the amount of light shone on their body, and converts it into a corresponding number of electrons. The stronger the light, the more electrons generated. After completion of the exposure time, the generated electrons are converted into a voltage, and finally through a digital to analog (A / D) converter into a discrete digital number.

Currently, there are two main techniques of image - CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). Both techniques are electrons into a voltage, and a natural color blind, because they can not distinguish between different wavelengths generated electrons. In order to show the color of each pixel is placed before the fine filter medium (also referred to as a microlens), the filtering means allowing only specific wavelengths of light through. One common method is mapped to the color of the wavelength filter means is arranged in the RGB (red, green, blue) array, allowing only three primary colors (red, green, blue) separately through, thus can be obtained three separate images - each image corresponding to one primary color.

By various combinations, RGB values may be generated in the vast majority of colors visible to the human eye. When each discrete color values are coded into 8 bits (i.e., 256 values), you can create a total of 16.7 million different colors by using "the RGB filters" principle. Wherein, the most common arrangement, "the RGB filter" approach is the Bayer array (Bayer the Pattern) , having alternating red, green, and cyan are alternately two types of filter combination. Because the human eye is more sensitive to green than red or blue, the number of green filter is twice the red Bayer array filter or blue filter. In computer vision applications, when processing a color image, the RGB three layers can all be used. If the processing capacity is limited, different channels may be combined into a grayscale image. It is worth mentioning that we can use OpenCV computer vision library in a cvtColorway to achieve an RGB image to grayscale image conversion, conversion principles and specific instructions, refer to the official documentation OpenCV: https://docs.opencv.org/ 3.1.0 / de / d25 / imgproc_color_conversions.html

CCD vs. CMOS

In the CCD sensor, each of the image elements in the collected electrons from the chip transport through one or several output nodes. Then, the charge is converted into a voltage level, sent as an analog signal after the buffer. This signal is then amplified by the signal amplifier outside the sensor and outside the sensor by the A / D converter converts the amplified signal into discrete digital. Initially, the CCD and CMOS technology has compared many advantages, such as higher photosensitivity and lower noise. However, in recent years, these differences almost disappeared. The main disadvantage of CCD is the high production cost, high power consumption (approximately 100 times higher than CMOS), which often cause the camera to heat problems.

Initially CMOS sensor for machine vision applications, but because of its poor light sensitivity, resulting in poor image quality. However, with the development of modern CMOS sensor, the image quality and photosensitivity have improved significantly. CMOS technology has the following advantages: Unlike CCD, CMOS chip integrated amplifier and A / D converter, which brings a significant cost advantage. For CCD, these components are located outside the sensor chip. CMOS sensor also has a faster data reading speed, lower power consumption, higher noise immunity and smaller system size. In automotive applications, because of these advantages, almost all cameras use CMOS sensors. Most camera settings sequence of images used in this course can be found here: http://www.cvlibs.net/datasets/kitti/setup.php