Single Image Rain Removal Based on Cross-Domain Collaborative Learning
1. Introduction
Deep Convolutional Neural Networks (DCNNs) have achieved excellent performance in image parsing tasks. However, most current methods treat rain streak removal as a one-to-one problem, often ignoring the in-domain shift of different synthetic datasets. Therefore, a derailment model trained on one synthetic dataset cannot effectively remove rainbands on other synthetic datasets.
To address this issue, this paper proposes a cross-domain collaborative learning (CDCL) framework to minimize intra-domain movement and inter-domain gaps.
First, a dual-branch derailment network based on collaborative learning is proposed to eliminate the distribution shift of image rain streaks in the synthetic domain; then, a cross-domain pseudo-label generation (CDPLG) method is designed to obtain more accurate images for real rain domain images. Accurate and more robust pseudo-labels, and use the pseudo-labels generated online to train a dual-branch derailment network to reduce the domain difference between the synthetic domain and the real domain; performed on public benchmark datasets (including synthetic datasets and real datasets) Extensive experiments are performed, and the experimental results show that our proposed framework achieves good performance compared to the state-of-the-art models.
First, there is a certain difference in the distribution of real rain streaks and synthetic rain streaks shown in Figure 1(a). From Fig. 1(a), it can be seen that the real rain screen image has multiple orientations compared with the synthetic rain screen image, which leads to the fact that the synthetic rain screen image cannot contain a large enough rain screen in the real rain screen image Stripe distribution range.
Second, existing rainfall removal models only consider the inter-domain gap between the synthetic domain and the real domain, and ignore the intra-domain shift on different synthetic datasets, as shown in Fig. 1(b). From Fig. 1(b), it can be seen that the density and orientation of the two rain images from different synthetic datasets are quite different.
In addition, we perform simple experiments on three different synthetic datasets, verifying the effect of rainfall removal due to intra-domain movement within the synthetic domain. The experimental results are shown in Table 1.
As can be seen from Table 1, when a rain removal model trained on one synthetic dataset is tested on another synthetic dataset, satisfactory results cannot be obtained due to the in-domain shift of the synthetic dataset. It should be noted that the rainwater removal model has the same structure as the single-branch derailment network in this paper.
Extensive experiments on public benchmark datasets (both synthetic and real) demonstrate that our proposed framework is effective compared to state-of-the-art fully-supervised and semi-supervised single-image disentanglement models.
3. Proposed method
As shown in Figure 2, to reduce the intra-domain offset between different synthetic domains and the inter-domain gap between the synthetic domain and the real domain. The model mainly includes network structure, labeled synthetic domain collaborative learning, unlabeled real domain collaborative learning and integrated loss function.
3.1 Network Architecture
The rain image can be modeled as a linear summation of the background image and the rain pattern information, expressed as:
o = b + r （1）
In the formula, o is the original rain pattern image, b is the background image, and r is the rain pattern information.
How to design a robust rain pattern learning network to better learn synthetic rain patterns and real rain patterns is crucial to improving the performance of single image analysis models. To solve this problem, this paper proposes the Rain Streak Learning Network (RSL).
Formula (1) can also be transformed into:
b = o− RSL(o) （2）
To improve the expressive power of raingrains, we construct a Multi-Scale Attention Residual Block (MSARB).
The two single-branch derailment networks contain the same structural parameters, as shown in Table 3
4. Experiments and results
4.1 Datasets and metrics
We choose three synthetic datasets including Rain200L (Yang et al., 2017), Rain1200 (Zhang & Patel, 2018) and Rain1400 (Fu et al., 2017), and a real image disassembly dataset for evaluation Our CDCL framework is used for the performance of image de-aliasing.
The Rain200L dataset (Yang et al., 2017) contains 1800 synthetic image pairs in the training set and 200 image pairs in the test set.
The Rain1200 dataset (Zhang & Patel, 2018) includes 12,000 image pairs for training and 1200 image pairs for testing.
The Rain1400 dataset (Fu et al., 2017) consists of 12,600 pairs of synthetic images for training and 1400 pairs of synthetic images for testing.
The Real Image Parsing dataset (Wang et al., 2019) contains 1000 real rainy images. The details of the four public datasets are shown in Table 4.
In addition, peak signal-to-noise ratio (PSNR) (Huynh-Thu & Ghanbari, 2008) and structural similarity (SSIM) (Wang et al., 2004) were used to evaluate model performance.
4.3 Results of synthetic images
The qualitative results of Rain200L, Rain1200 and Rain1400 are shown in Figure 4 and Figure 5, respectively. From Fig. 4 and Fig. 5, it can be seen that the proposed supervised and semi-supervised CDCL has higher qualitative performance, especially in some local regions marked with red wireframes in Fig. 4 and Fig. 5. Therefore, it can be asserted that our proposed framework outperforms state-of-the-art supervised and semi-supervised image deraining models.
As shown in Table 5, our proposed model achieves better performance on Rain200L, Rain1200 and Rain1400, compared with other supervised image dekerning models. Moreover, our proposed model also achieves higher results in SSIM and PSNR values obtained with unlabeled synthetic data compared to other semi-supervised image solving methods.
4.8 Modeling Complexity and Running Time
In this section, we compare the parameters and time complexity of the proposed model with state-of-the-art supervised and semi-supervised image solving inference models. The experimental results of each model are shown in Table 12.
The model achieves acceptable performance in terms of parameters and time complexity, which shows that the method can achieve satisfactory solution results at an acceptable computational cost.
5. Conclusion
In this paper, we propose a cross-domain collaborative learning approach to the single image parsing task. Firstly, a MSARB-based dual-branch derailment network is proposed, which eliminates the distribution shift of image rainprint in the synthetic domain through collaborative learning; then, the dual-branch derailment network is trained using online generated pseudo-labels to reduce the synthetic domain and Domain Difference Between Real Domains; Extensive experiments on multiple datasets demonstrate that a collaborative learning approach based on labeled synthetic domains and unlabeled real domains effectively improves the generalization ability of network models for rain removal. Compared with existing semi-supervised rainstripe removal methods based on single synthetic domain rainstripe information, our proposed method fully utilizes the diversity and complementarity of different synthetic rainstripes, and better realizes the transition from synthetic domain to real domain. derailed transfer of knowledge. However, the adaptability of the model to real rain pattern information is still insufficient. In the future, we will design a rainprint learning network to better simulate the direction and density of real rainprint information.