Paper reading "Kindling the Darkness: A Practical Low-light Image Enhancer"

In addition to suboptimal lighting, multiple types of degradation, such as noise and color distortion, are hidden in the dark due to the limited quality of the camera. Simply increasing the brightness of dark areas will inevitably amplify hidden degradation. This work builds a simple yet effective ignited dark network (denoted kinD), which is inspired by retinex theory to decompose the image into two parts. One component (illumination) is responsible for the regulation of light, while the other component (reflectivity) is responsible for degradation removal. The network is trained on pairs of images taken under different exposure conditions. It can resist severe visual defects, and the user can adjust the brightness of the light at will.

introduce

Figure 1 Natural images under different lighting conditions

The first situation is extremely low light. Severe noise and color distortion are hidden in the darkness. By simply zooming in on the intensity of the image, the degradation is shown in the upper right corner. The second photo was taken at sunset (low ambient light) with most objects backlit. It is also difficult to get rid of the problem like the second case when imaging against the light source (the sun) at noon, although the surrounding light is stronger and the scene is more clearly visible. Direct zooming in on those relatively bright areas in the last two photos would result in saturation.

Methods based on deep learning perform well in numerical underlying vision tasks such as denoising and super-resolution, but most of them require training data to contain ground truth. For specific problems, such as low-light image enhancement, although the order of light intensity can be determined, there is no ground-truth real data. From a user perspective, the light levels preferred by different people/needs may vary.

The article summarizes the challenges of low-light image enhancement as follows:

1. How to effectively estimate the illumination components from a single image and flexibly adjust the illumination?

2. How to remove degradation such as noise and color distortion that were previously hidden in the dark after lighting the dark area?

3. How to train a model to enhance low-light images without clear ground truth and only looking at two/several different examples?

An ideal low-light image enhancement algorithm should be able to effectively remove the degradation hidden in the dark and flexibly adjust the exposure conditions. ,

The main contributions of the paper:

1. Inspired by Retinex theory, this network decomposes the image into two parts: reflectance and illumination.

2. The network is trained using pairs of images captured under different light/exposure conditions, rather than using any ground-truth reflectance and lighting information.

3. The model provides a mapping function that can flexibly adjust the light level according to the different needs of the user. Effectively removes visual defects magnified by light and dark areas.

Review of existing methods

Ordinary method

1. For images with weak global lighting, their visibility can be enhanced by directly amplifying the brightness. But noise and color distortion show up along the details. For images containing bright areas, it's easy to cause (partial) saturation/overexposure.

2. Techniques represented by histogram equalization and its subsequent steps improve the contrast of the image.

3. Gamma correction (GC): Performed individually on each pixel in a non-linear manner, it can improve brightness but does not take into account the relationship between adjacent pixels.

(Almost no real lighting factors are considered, making the enhanced results visually unstable and inconsistent with the real scene)

brightness-based methods

1. The SSR and MSR methods based on Retinex theory are limited in generating the final result. The output looks unnatural and is over-enhanced in some places.

2.NPE: Enhance contrast and maintain natural illumination at the same time.

3. SRIE weighted variation model: Simultaneously estimate reflectivity and illumination, and control illumination to form the target image. Can suppress noise but falls short in handling color distortion and larger noise.

Deep learning based methods

1. LLNet (Pattern Recognition 2017): A deep network is constructed as a module for simultaneous contrast enhancement and denoising.

2. MSR-net (arXiv 2017): It is considered that multi-scale retinex is equivalent to a feed-forward convolutional neural network with different Gaussian convolution kernels. A convolutional neural network is constructed to learn the end-to-end mapping between dark images and bright images.

3.RetinexNet (BMVC2018): It integrates image decomposition and light mapping, reflection denoising, and lighting enhancement. The impact of noise on different lighting areas is not considered.

4.SID (CVPR2018): A low-light image processing pipeline (SID) based on end-to-end training of a fully convolutional network, which can handle noise and color distortion simultaneously. However, this work is for data in raw format, which limits its applicable scenarios. If the network is modified to accept data in JPEG format, performance will decrease significantly.

Method based on image denoising

1. BM3D, WNNM based on specific priors of natural images (local similarity, piecewise smoothness, signal sparsity, etc.).

2. DL-based denoising: SSDA-stacked sparse autoencoder, DnCNN-residual learning and batch normalization

(This does not take into account that different areas of a light-enhanced image carry different levels of noise)

proposed method

KinD Network is divided into three parts: (1) Image decomposition network: Layer Decomposition Net (2) Reflection component correction network: Reflectance Restoration Net (3) Illumination component correction network: Illumination Adjustment Net. The entire network architecture is shown in the figure below.

Taking dark light/normal illumination image pairs as training samples, Layer Decomposition Net decomposes them in sequence to obtain illumination components and reflection components. Then get �~�� and �~�� through Reflectance Restoration Net and Illumination Adjustment Net.

2.1 Layer Decomposition Net

Layer Decomposition Net has two branches, one branch is used to predict the reflection component, and the other branch is used to predict the illumination component. The reflection component branch uses the five-layer Unet network as the main network structure, followed by a convolution layer and a Sigmoid layer. The illumination component branch consists of three convolutional layers, which also uses the feature map in the reflection component branch. Please refer to the paper for specific details.

Layer Decomposition Net:

2.2 Reflectance Restoration Net

The reflection component of the low-light image has more degenerate components, so the reflection component of the high-light image is used as the true value constraint training; at the same time, the distribution of the reflection component is also related to the illumination component, so the information of the illumination component is also embedded into the network. inside. The corrected reflection component predicted by the network is; the Loss of Reflectance Restoration Net is:

2.3 Illumination Adjustment Net

The network is relatively lightweight and consists of three convolutional layers. The correction of the illumination component is similar to the correction of the reflection component. Here, the illumination component of the highlight map is also used as the true value constraint training. The network predicts the corrected reflection component; the Loss of Illumination Adjustment Net is:

Experimental results

Summarize

Image dark light enhancement based on deep learning is based on the Retinex theory. It uses a convolutional neural network to decompose the image S to obtain the illumination component I and the reflection component R. This is equivalent to integrating the prior knowledge of image enhancement into deep learning. . Most of the recent newer and better papers focus on how to better decompose images and generate better quality illumination components and reflection components. There are also some people who use GAN to directly perform image enhancement through image generation.