Detailed explanation of the paper - "Deep Color Consistent Network for Low-Light Image Enhancement"

Abstract

Low Light Image Enhancement (LLIE) studies how to refine lighting to obtain natural normal light images. Current LLIE methods mainly focus on improving illumination, but do not reasonably incorporate color information into the LLIE process to consider color consistency. Therefore, there is often a color difference between the augmented image and the ground truth.
To address this problem, we propose a new deep color consistency network, called DCC-Net, to preserve the color consistency of LLIE. A new "divide and conquer" synergistic strategy is proposed, which jointly preserves color information while enhancing illumination.
Specifically, our DCC-Net decoupling strategy decouples each color image into two main components, namely grayscale image and color histogram. Use grayscale images to generate reasonable structure and texture, and color histograms are good for maintaining color consistency. That is to use the two to cooperate to complete the LLIE task.
In order to match the color features and content features of the image and reduce the color consistency gap between the enhanced image and the real picture, we also design a new pyramid color embedding (PCE) module, which can better embed color information into the LLIE process middle.
Extensive experiments on 6 real datasets show that our DCC-Net augmented images are more natural and colorful, which outperforms the current state-of-the-art methods.

3. Proposed Method

In this section, we introduce the framework (see Figure 2) and details of DCC-Net, which aims to maintain color consistency and naturalness when acquiring normal-light images. DCC-Net has three subnets (i.e., G-Net, C-Net, R-Net) and a pyramidal color embedding (PCE) module.

The overall framework of DCC-Net. It can be seen that there are three subnets G-Net, C-Net and R-Net, where the goal of G-Net is to restore grayscale images with rich content information, the focus of C-Net is to learn color distribution, and R-Net will grayscale images Combines the intensity image and color information to restore natural and color-consistent normal-light images.

3.1 Network Structure

• G-Net
  Given an input low-light image, the goal of G-Net is to predict the grayscale image of a normal-light image that contains rich structure and texture information, but does not contain color information. This process is expressed as
  $$G_{\text{pre }}=GNet\left(S_{\text{low }}\right)$$
  where$G_{pre}$For predicting grayscale images, $S_{low}$For the input low-light image, GNet is the transformation of G-Net. Specifically, G-Net uses an encoder-decoder pipeline, which is similar to the classic U-Net.
  For G-Net, we use l1 loss to reconstruct the grayscale image:
  $$l_g=\frac{1}{H\times W}{\Vert G_{pre}-G_{\text{high }}\Vert }_1,$$
  where,$l_g$Reconstruction loss for grayscale images, $G_{high}$is the grayscale image of the normal light image, H and W are the grayscale image $G_{high}$height and width. Therefore, G-Net does not consider color information, but instead works on recovering texture and structure.
• C-Net
  color histogram is a color feature widely used in image retrieval systems [9]. The color histogram mainly describes the proportion of different colors in the whole image, and does not care about the spatial position of the color. In this paper, we compute the color histogram in the RGB color space. Specifically, the color histogram of the image should be a matrix of N × 256, where N = 3 corresponds to three color channels (R, G, B), and 256 is consistent with the range of pixel values.
  C-Net is designed based on color histogram for color feature learning. The goal of C-Net is to obtain color features consistent with normal light images (see Figure 2). We also leverage the encoder-decoder pipeline of C-Net, which converts the input low-light image into a predicted color histogram by the following formula:
  $$C_{\text{pre }}=CNet\left(S_{\text{low }}\right)$$
  where$C_{pre}$For the obtained color histogram, CNet is the calculation process of C-Net. In order to better reconstruct the color histogram, we also use l1 loss to constrain C-Net, which can be described as follows:
  $$l_c=\frac{1}{N\times 256}{\Vert C_{\text{pre }}-C_{\text{high }}\Vert }_1\text{, }$$
  where$l_c$Reconstruction loss for color histogram, $C_{high}$A true color histogram for a normally lit image. Note that the color histogram cannot describe the content and details in the image. That is, C-Net puts all the attention on learning consistent color features, which is good for enhancement.
• R-Net
  R-Net combines the grayscale images and color histograms obtained by G-Net and C-Net to synergistically restore normal-light images. Transform the input low-light image, predicted grayscale image and color histogram into a constant light image through R-Net, as follows:
  $$S_{\text{pre }}=RNet\left(S_{\text{low }},G_{\text{pre }},C_{\text{pre }}\right)$$
  where$S_{pre}$for image enhancement.
  To reconstruct the normal light image at the pixel level, we use the color image reconstruction loss $l_r$，其定义如下:
  $$l_r=\frac{1}{N\times H\times W}{\Vert S_{\text{pre }}-S_{\text{high }}\Vert }_1\text{, }$$
  where,$N,H,W$ respectively represent the normal light image$S_{high}$The number of channels, height, width.
  At the structural level, we use the SSIM loss as a constraint:
  $$l_{}=1-\mathrm{SSIM}\left(S_{\text{pre }},S_{\text{high }}\right)$$
  In addition to the SSIM(·)definition SSIM
  $$\mathrm{SSIM}(x,y)=\frac{2m_x{m}_y+c_1}{m_x^2+m_y^2+c_1}\cdot \frac{2p_{xy}+c_2}{p_x^2+p_y^2+c_2}$$

where $x,y\in R^{H\times W\times 3}$ represents the two images to be tested,$m_x,m_y\in R$ represents the mean of the two images,$p_x,p_y\in R$ is the variance corresponding to the two images,$c_1$and $c_2$Two constant arguments in case the denominator is zero.
In addition, the total variation loss $l_{tv}$as a regularization term to preserve the smoothness of the augmented image.

3.2. Pyramid Color Embedding (PCE)

The PCE module is designed to better embed color information into R-Net, as shown in Figure 3. It can be seen that PCE has six pyramid-structured color embedding (CE) modules. CE achieves dynamic embedding of color features. The main component of CE is the dual affinity matrix (DMA) which solves the problem of information mismatch.

Figure 3: Detailed structure of the pyramidal color embedding (PCE) and color embedding (CE) modules, where $\odot Element$ -wise multiplication, ⊕ means element-wise addition, ⊗ means upsampling operation.

• Dual affinity matrix
  can obtain corresponding grayscale images and color histograms from G-Net and C-Net, which provide rich structure and texture details, as well as color information, respectively. R-Net applies these two methods to achieve better enhancement effects. Since color histograms do not contain spatial information, simply concatenating them together will result in inaccurate lighting in the enhanced image. In addition, simple stitching also results in a mismatch between color information and content, which may produce chromatic aberrations in the enhanced image.
  In order to solve the information mismatch problem and obtain better color information embedding, a new color embedding module is proposed, which can dynamically fuse color features into R-Net according to the affinity between color and content features middle. The purpose of the proposed dual affinity matrix (DAM) is to compute an affinity matrix to match color and content features, further preventing the enhanced image from producing inconsistent colors. Specifically, given the color feature $C$ and content features$F$ , the size is$N\times H\times W$ , DAM first calculates the Manhattan distance of each location and the inner product of C and F, the formula is as follows:
  $$\begin{array}{c}M(x,y)=-\Vert F(x,y)-C(x,y){\Vert }_1,\\ P(x,y)=F(x,y)\cdot C(x,y),\end{array}$$
  其中$F(x,y)，C(x,y)\in R^N$ represents the vector of F, C at (x, y),$M,P\in R^{H\times W}$ is the Manhattan distance matrix and the inner product matrix. Then, the dual affinity matrix A (dual affinity matrix) can be calculated as:
  $$A=2\times \mathrm{sigmoid}(M)\odot \tanh \left(P\right),$$
  where$tyou(\_)$和$sigmoid(\cdot )$ are tanh function and sigmoid function respectively. Note that for each position$(x,y)$， $M(x,y)\le 0$ , so that$sigmod(M)\in [0,0.5]$ . Therefore, we use$2\times sigmoid\left(M\right)$ to guarantee$A\in [0,1]$ range.

• Color embedding
  CE obtains the dynamic embedding of color information, and its structure is shown in Figure 3. After CE obtains the dual affinity matrix A, it multiplies A and the color feature C element-by-element, and sums the weighted color feature and the content feature F to obtain the color information of the embedded feature: E = A ⊙ C + FE=A\
  $$E=A\odot C+F$$
  where E is the output feature used in the R-Net decoder. There is also an upsampling operation of the color feature C to change its resolution, which is then further input into the next CE as the original color feature.

• Pyramid structure
  Given color features, we can leverage them to guide the augmentation process to achieve consistent colors. In order to fully explore the color information, we give a PCE containing 6 pyramid-structured CEs (as shown in Fig. 3). Given the color features $C_i$and content features $F_i$, PCE features from shallow to deep in each layer are described as follows:
  $$E_i,C_{i+1}=CE\left(F_i,C_i\right),i=1,2,\cdots ,$$E i E_i
  in$$6$$$E_i$Represents the output feature, $CE(\cdot )$ represents the transformation of CE. $C_i$number $(i-1)$ CE calculation. In contrast, in the encoder of R-Net,$F_i$is copied from the corresponding layer. The pyramid structure embeds color features into six layers. In other words, progressive design can take full advantage of color information. Therefore, the enhanced image color will be more consistent.

3.3. Objective Function

The target function of our DCC-Net is described as:
$$l_{\text{total }}=l_g{l}_g+l_c{l}_c+l_r{l}_r+l_{ss}{l}_{ss}+l_{tv}{l}_{tv}$$
λ g $l_g,l_c,l_r,l_{ss},l_{tv}$are several trade-off parameters. Among them, $l_g$and $l_c$are used to restore grayscale images and color histograms, respectively. Use $L_r$和$l_{ss}$Pixel-level and structure-level reconstruction of standard light images. In order to prevent overfitting and maintain smoothness, $L_{tv}$as a regularization term.