In daily work, Ruibo's technical colleagues will receive inquiries from customers about the problem of model spots every now and then. As a result, Ruibo's internal technical assessment questions also derived a question: "Please describe the circumstances under which the model is prone to appear. Piebalds and Approximate Solutions”. It can be seen that for oblique photography real-life models, the speckle problem is always a lingering "shadow" on the model results and is one of the culprits that affects the look and feel of the model.
Many customers want to completely solve the above problems, but they suffer from the fact that it takes a lot of effort inside and outside the industry - choosing a suitable working period or performing color balancing on a large number of photos, etc. However, these methods reduce project efficiency and are not effective enough.
Is there any way to solve this kind of problem from the source? So today, Ruibo will share with you the latest technology: based on the Kalman filter smooth exposure model, which partially solves the mottling problem of the model;
▲ Patterns will seriously affect the look and feel of the model
MAIN CAUSE OF MODEL SPECKLE
The main causes of model spots
01Weather
(light and shadow) changes
Senior aerial survey veterans all know that the best weather for oblique photography is either a cloudless sunny day or a cloudy day with sufficient brightness, because in these two weathers, the changes in the sun's light are minimal, especially on cloudy days when there is no light direction. The resulting shadows and just the right clouds can not only ensure sufficient light intensity, but also act like a huge uniform light mask to ensure that the light intensity and light direction in all directions on the ground are basically the same. This kind of model has the best effect; but in this kind of weather But it is rare.
▲ The effect of the cloudy model is uniform
02
Patterns on the building object itself (or reflection)
Modern buildings use a large number of glass structures, and the reflection of the glass will cause mottled surfaces of adjacent buildings. There is basically no solution to this mottled appearance. However, this kind of pattern has an advantage. The reflective patterns are equivalent to adding texture to the surface, which can match more points of the same name and avoid loopholes.
▲ Mottled texture on the building surface (this picture is an aerial photo, not a model)
03
Photometry causes inconsistent brightness of the same texture
During flight, even if the light intensity does not change, the camera's metering function will change as the texture of the ground object changes. Even the same exposure parameters will cause the same texture surface to be lighter or darker. The more dramatic the texture change, the more obvious this change in light and dark. Especially in urban areas, this will cause mottled textures (patterns) to form on the surface of buildings, affecting the look and feel of the model.
The technology that Ruibo is going to share today is a solution to this situation.
▲ In photos taken continuously with the same lens, the brightness of the building facade is inconsistent.
PRINCIPLE OF EXPOSURE
Aerial Photography Exposure Principle
Currently, the more common algorithms include average brightness method, weighted mean method, brightness histogram, etc. The most common one is the average brightness method. The average brightness method is to average the brightness of all pixels in the image, and continuously adjust the exposure parameters to achieve the final target brightness. The weighted average method sets different weights on different areas of the image to calculate the image brightness. For example, the selection of various metering modes in the camera changes the weights of different areas. The brightness histogram method calculates image brightness by assigning different weights to the peaks in the histogram. There are two ways to expose aerial survey operations: automatic exposure and fixed exposure.
Fixed exposure: Fixed exposure means that all parameters are preset and fixed. Its advantage is that there will be no uneven exposure that will cause mottling on the model; but the requirement is that the external light cannot change significantly, otherwise the model will become darker or brighter as a whole, so Fixed exposure models are rarely used in aerial surveys. Fixed exposures are suitable for scenes with stable light, such as indoor scenes;
Automatic exposure: The intensity of the sun's light changes throughout the day. When the texture of objects changes greatly, the automatic exposure mode is needed to adjust the brightness of the photo. As shown before, automatic exposure can solve some problems of changes in external light sources, but it cannot solve the scene of texture changes.
Ruibo has tried both methods
FIXED EXPOSURE PRINCIPLE
Fixed exposure principle
The earliest R&D team of Ruibo wanted to adopt a fixed exposure scheme. First, preset an exposure parameter, and then add a solar radiation intensity sensor or an ultra-wide-angle high frame rate camera above the camera, in order to measure solar radiation at all times. The change in intensity, combined with the camera histogram parameters, compensates for the camera's preset exposure parameters.
In this way, no matter how the sun and ground objects change, the best exposure parameters are guaranteed, and the mottled light and shadow caused by changes in exposure parameters are perfectly solved.
But the idea is very rich, and the reality is very skinny.
The biggest problem is the attitude of the drone. We found that no matter how we arrange the solar radiation intensity sensor or use a cosine corrector, we cannot avoid changes in photometric parameters caused by the attitude of the drone.
▲ Changes in the attitude of the drone lead to changes in the photometry value
The only solution is to install a gimbal, but this will take you farther and farther. In the future, this technical solution may be considered for the DG10 that is not sensitive to weight.
▲ Top-mounted metering and stable gimbal
▲The impact of clouds on light changes
BASED ON KALMAN SMOOTH EXPOSURE
During
the exposure process, the exposure process is a dynamic process. The main changes in the exposure parameters are "light from the sky" and "underground objects". "Light from the sky" is the change of light and shadow when the sun enters the clouds. At this time, the exposure parameters need to be adjusted accordingly. "Underground objects" are changes in the texture of ground objects. At this time, the exposure parameters need to be appropriately stabilized;
The characteristic of the change of "light in the sky" is that the exposure parameters of the five lenses change synchronously, while the change of "underground objects" is characterized by the fact that the exposure parameters of the five lenses are not synchronous.
Therefore, only a simple interpretation is needed before doing exposure smoothing. If the exposure parameters of the five lenses change relatively synchronously, then the exposure adjustment can be made directly at this time. If the exposure parameters of the five lenses are relatively out of sync, then the exposure cannot be adjusted directly at this time. , otherwise it will easily cause spots on the model;
For changes in "underground objects", the camera can treat local light and shadow changes as noise during the exposure process; noise is a kind of interference and belongs to an uncertain dynamic system. Kalman filtering can determine what the system will do next. Make educated guesses. Even if there is interference from noisy information, Kalman filtering can usually figure out what is going on very well and find out the subtle correlations between phenomena.
By observing the change trend of light and shadow, the change amount is used as a noise signal to infer the interference factors that affect the image quality, thereby predicting the state value at the next moment, and then adjusting the predicted value by comparing the error between the actual value and the predicted value, and finally obtains a better Accurate exposure signal to achieve smooth transition.
Kalman filter is an optimization algorithm based on the state space model, which improves the accuracy of state estimation by minimizing the variance of the system state. During the camera exposure process, we can regard the data acquisition process as a linear dynamic system, and its state space model can be expressed as:
x(k) = A x(k-1) + B u(k) + w(k)
y(k) = C*x(k) + v(k)
Among them, x(k) represents the state vector of the system at time k, A is the system state transition matrix, B is the system input matrix, u(k) is the input signal, w(k) is the system process noise, and y(k) represents The observation vector of the system at time k, C is the observation matrix, and v(k) is the system observation noise.
The basic process of Kalman filtering is as follows:
Prediction step: predict the value and variance of the next state based on prior information, that is
x(k|k-1) = A x(k-1|k-1) + B u(k)
P(k|k-1) = A*P(k-1|k-1)*A’ + Q
Among them, x(k|k-1) represents the predicted value of x(k), P(k|k-1) represents the covariance matrix of the predicted value of x(k), and Q is the covariance matrix of the system process noise. .
Update step: update based on observed values and predicted values, and calculate the optimal state value and variance, that is
K(k) = P(k|k-1)C’(C*P(k|k-1)*C’ + R)^(-1)
x(k|k) = x(k|k-1) + K(k)(y(k) - Cx(k|k-1))
P(k|k) = (I - K(k)*C)*P(k|k-1)
Among them, K(k) represents the Kalman gain, x(k|k) represents the optimal estimate of x(k), and P(k|k) represents the covariance matrix of the optimal estimate of x(k). R is the covariance matrix of the system observation noise. In this system, we analyzed and optimized each parameter, and designed an adaptive way to adjust the filter parameters to make the filtering effect more accurate and achieve the results we expected.
In layman's terms, the Kalman filter algorithm is equivalent to a "fortune teller", which uses past exposure values to predict the next exposure value to achieve smooth exposure and reduce speckles as much as possible;
POSESCRIPT
Postscript The ins and
outs of the above global smooth exposure. Its core advantage is to avoid model spots caused by changes in exposure parameters caused by texture changes during the oblique photography process, and improve the effect of the model; this exposure method is different from traditional photography exposure in that , instead of responding to changes in ground texture in a timely manner and pursuing the correct exposure of a single image, it instead analyzes a series of images as a whole, treats local texture changes as noise, and achieves correct exposure of the entire image, while for sunlight coverage The places that should be bright should be lighter, while some dark places or shadow places should be darker where they should be, so as to achieve a consistent overall look and feel of the model.
▲Normal working mode
▲ Smooth working mode
Source: Chengdu Ruibo Technology
- END -