【Low Light Denoising】Rethinking Noise Synthesis and Modeling in Raw Denoising

1. Motivation

• Real original image noise is contributed by a variety of noise sources, and there are differences between different sensors. Existing methods, whether physics-based or leanrning-based, cannot accurately model all noise sources.

On why it’s impossible to accurately model all noise sources:

Every semiconductor and every processing step in a circuit is a potential noise source, and we don’t know what the most appropriate distribution is for most of them, so it’s impossible to accurately isolate and model them all.

2. Contribution

• An effective and versatile noise synthesis method is proposed for RAW domain image denoising.
• Proposed raw pattern-aligned patch and high-bit reconstruction to synthesize more realistic noise

3. Methods

3.1 Establish noise model

For a raw image generated by a typical CMOS sensor, the process from incident photons to output digital signal can be modeled as the following formula:
$D$ is the digital signal stored in the RAW image,$I$ is the number of incident photons,$N_p$is the signal-related photon scattering noise, $K_a$and $K_d$are analog gain and digital gain respectively, $N_1$and $N_2$Represents the sum of other noise generated before analog gain and digital gain.

If the noise is decomposed into signal-related and signal-independent, then: Here’s

the key point: Since the number of incident photons strictly follows the Poisson distribution, the photon scattering noise related to the signal can be accurately synthesized through the "mean-variance" "The linear fitting of " can obtain the overall gain.

For signal-independent noise, the authors proposed a simple and clever method: sample directly from the black frame. Because the black frame is collected under no-light conditions, it naturally only contains all noise that is not related to the signal.

The final synthesized noisy image can be expressed by the following formula:

3.2 Experiments and improvements

The noise synthesis method proposed by the author is relatively simple. First, you need to collect black frames to build a database (default exposure time, collect 10 black frames at each ISO), and then try to use a pixel-wise method to sample noise from the black frames, but the experimental results The display does not perform well in low light conditions.

The author attributes the poor effect to the following two points:

• Pixel-wise sampling destroys the spatial correlation characteristics of noise (such as line noise, fixed pattern noise), which is particularly important under dark light conditions.
• Signal-independent noise is the main noise in dark light environments, and the original noise distribution of black frames is destroyed during the ADC quantization process. (It doesn’t make sense to say this in this place. Isn’t the noise also quantized during actual denoising? Using the original noise distribution for training will lead to inconsistent distributions faced during training inference)

The first point is relatively easy to solve. Just desample based on patch, and then ensure the bayer format is aligned (called PAP in the paper). For example, the noiseless patch during training is 512×512 in size, and the bayer format is RGGB. During sampling, you only need to randomly select 512×512 from the black frame under the restriction that the starting rows and columns are all even numbers (RGGB is guaranteed). patch, and then add it to the clean patch.

For the second point, the author proposes a high-bit reconstruction (HB) method to reconstruct the black frame to a higher bit width and restore the true noise distribution.

In order to illustrate the importance of high-bit reconstruction, the author compared the results of "PG" and "PG + LB", and "ELD" and "ELD + LB". "+LB" means quantizing the simulated noise to 14bit. As can be seen from Table 1, after the noise is quantized to 14bit, the performance of dark light image denoising has obvious decline.

It can be seen from the results of the ablation experiment that PAP and HB can significantly improve the denoising performance of dark-light images.

3.3 High-bit reconstruction

The high-bit reconstruction part seems a bit confusing, but you can probably understand it if you understand Figure 1. Figure 1

(a) shows how a continuous signal is quantized into a discrete signal, within each quantization step range (black line) , consecutive signal values ​​will be replaced by one value. Described in mathematical language: Assume that the quantization step size is $[x-\frac{1}{q},x+\frac{1}{q}]$ , the distribution function of continuous signal values ​​is$f(\cdot )$ , for any$a\in [x-\frac{1}{q},x+\frac{1}{q}]$ , there is$f(a)=c$，$c$ is the ordinate corresponding to the red point.

Since the black frame is a quantized discrete signal, which is the discrete red point in the picture, how to reconstruct the continuous signal before quantization? The explanation in Figure 1 (b) is relatively clear. First, a distribution is used to fit the discrete red points, which is the yellow line in (b); then the red points are mapped to the sampling step [ $[x-\frac{1}{q},x+\frac{1}{q}]$ (this process is indicated by a red arrow), that is,$p_3$Map to $(p_1,p_2)$ within the range.

Then the question comes again, how to map it? You can’t randomly sample a value~ This is what the paper says:

The sample probability of high-bit values ​​is obtained by normalizing the estimated continuous distribution within the quantization step
.

I don’t understand it right now, I’ll add more when I understand.

4. Comments

• The noise modeling method proposed by the author is somewhat original. It abandons the complex physical modeling of analyzing the statistical distribution of different noise components one by one, directly decomposes the noise into signal-related and signal-irrelevant parts, and directly samples the real data from the black frame. The signal-independent noise just makes up for the shortcomings of ELD (cannot model all noise sources)
• For high-bit reconstruction, it is actually a data augmentation method from another perspective. Because the collected black frames are limited, randomness is introduced in the process of reconstruction to high bits, thereby increasing the diversity of noise and making training samples more sufficient. (The diversity and adequacy of training samples are important for neural networks)

Reference

Rethinking Noise Synthesis and Modeling in Raw Denoising