A general large model in the field of remote sensing Published in CVPR on November 13, 2023
SpectralGPT: Spectral Foundation Model (arxiv.org)
E. Ablation Studies
During the pre-training phase, we conduct a comprehensive study on various factors that may affect downstream task performance. These factors include masking ratio, ViT patch size, data size, reconstruction target, decoder depth, and model size. To perform a more rigorous evaluation of the pre-trained models, we fine-tune all ablation models on the BigEarthNet multi-label classification dataset, using only a 10% subset of the training set, which is a more difficult challenge, and evaluate using mAP measurements. . We chose ViT-B as the backbone model to ensure consistency between experiments. In addition to the reduction involving data size and training plan length, all models were pre-trained on the fMoW-S2 dataset for 200 epochs. This comprehensive evaluation framework allows us to gain a deeper understanding of the impact of these factors on model performance.
1)Token size: Table V(a) Figure 8(a) provides important insights into the impact of token size on model performance, It is consistently shown thatlarger patch sizes lead to reduced model performance, which is consistent with previous research results [30]. This phenomenon can be attributed to the inherent characteristics of the ViT architecture. For larger token sizes, such as 16 x 16, each image contains fewer tokens, resulting in a reduction in fine-grained spatial information as the model progresses through its deeper layers. Therefore, the reduction in spatial detail can have a negative impact on the overall performance of the model. However, it is worth noting that regardless of the token size setting, the pre-trained model always enhances mAP, emphasizing its ability to improve performance in various configurations. It is worth noting that Although the size of the input image is 96 × 96 or 128 × 128, the recognition performance when the marker size is 8 × 8 is significantly better than 16 × 16< a i=6>, emphasizing the versatility and effectiveness of the pre-trained model.
2)Data scale: Table V(b) and Figure 8(b) are conducted on the impact of pre-training data in our study comprehensive analysis. We use two datasets (i.e., fMoW-S2, BigEarthNet) for pre-training while keeping the standard input image size of 96 × 96. To delve deeper into this comparison, we initially pretrained the model exclusively on fMoW-S2 and then seamlessly continued pretraining on BigEarthNet without any intermediate fine-tuning steps. Our pre-training datasets include fMoW-S2’s extensive training set, which includes an impressive 712,874 images from around the world, and BigEarthNet’s training set, which includes 351,496 images from European regions, excluding snow-covered regions. , image with cloud or cloud shadow effects.
The analysis in Table V(b) highlights the substantial impact of data size and distribution on model pre-training. Models pretrained onthe same dataset as the downstream task consistently show superior performance, highlighting the role of dataset consistency in effective transfer learning Key role. Furthermore, fMoW-S2 outperforms BigEarthNet in terms of pre-training, mainly due to its larger dataset and wider geographical coverage. Interestingly, the concept of continuous pre-training, combining both datasets, resulted in models with higher mAP scores. This improvement can be partially attributed to the transition from 96 × 96 images during fMoWS2 pre-training to 128 × 128 images during BigEarthNet pre-training, highlighting the beneficial impact of increasing image size on overall model efficiency. Models pretrained on the same dataset as the downstream task consistently show excellent performance, highlighting the critical role of dataset consistency in effective transfer learning. Furthermore, fMoW-S2 outperforms BigEarthNet in terms of pre-training, mainly due to its larger dataset and wider geographical coverage. Interestingly, the concept of continuous pre-training, combining both datasets, resulted in models with higher mAP scores. This improvement can be partially attributed to the transition from 96 × 96 images during fMoWS2 pre-training to 128 × 128 images during BigEarthNet pre-training, highlighting the benefit of increasing image size to overall model efficiency Impact.
3)Masking ratio: Table V© and Figure 8© reveal the impact of masking ratio, revealing a noteworthy trend, That is, the higher the masking ratio, the better the model performance. Unlike the traditional masking rate of 75%, we found that the optimal masking rate for multispectral images is 90%. This observation is consistent with the hypothesis proposed in [29] that the masking ratio in MIM methods is complexly related to the redundancy of information in the data. Multispectral images themselves have greater information redundancy, and there is a strong correlation between their spectral bands. Therefore, a higher masking ratio is essential for the model to effectively learn meaningful representations from these images. In addition, the 90% masking rate significantly improves the efficiency of the pre-training stage, reduces memory complexity and speeds up training time, providing practical advantages for model development.
4)Reconstruction target: Table V(d) and Figure 8(d) normalize the reconstruction target pair under the background of multispectral image The impact of normalized, standardized data and unnormalized, standardized raw data was analyzed in depth. Normalization (scale all data to the [0,1] range) and Normalization (convert the data to mean 0 and standard deviation 1) are the goals of both studies. Notably, the results show that the difference in model performance betweennormalized and normalized reconstruction objectives is small mainly because both objectives belong to Pixel-level data conversion. However, models pretrained on raw data perform much worse than models with normalized reconstruction objectives. We attribute this phenomenon to the properties of multispectral images. Spectral values are usually large and vary between frequency bands, so a model pretrained on raw data may require a longer hold time to converge and show the same performance as normalization and normalization Same performance as the pretrained model on the data. Our argument suggests that using more semantically meaningful targets in a specific representation space may improve model performance.
5)Decoder depth: Table V(e) and Figure 8(e) examine the impact of decoder depth on model performance , following the principles of the MIM approach, where the pretrained encoder serves as the backbone for downstream tasks while discarding the decoder component. Notably, the results show that the shallow decoder configuration is not suitable for spectral model pre-training. This observation is consistent with the hypothesis that spectral images are characterized by high dimensionality and complexity, requiring decoders with enhanced capacity which is consistent with the field consistent with previous findings [29].
6)Model size: Table VI and Figure 8(f) quantify the fine-tuning results of ViT-B and Vit-L and qualitative comparative analysis, revealing compelling insights. Macro-mAP and micro-mAP are listed to comprehensively evaluate the performance of the model. ViT-B with 12 transformer layers and 86 million parameters showed promising performance gains when adopting this approach, achieving a mAP (micro) of 85.41, which is 5.26 higher than ViT-B trained from scratch. On the other hand, ViT-L with 24 layers and 307 million parameters significantly outperforms ViT-B, with a mAP(micro) of 86.92, significantly exceeding the model trained from scratch by 4.44. In addition, ViT-B has a total of 32 layers and 632 million parameters, which greatly improves the performance of the neural network on BigEarthNet, with mAP(micro) of 89.23. It is worth noting that although our model is fine-tuned on only 10% of the downstream training data, the ViT-H model using SpectralGPT + pre-trained weights beats all models trained using the entire training set Model, SOTA mAP(micro) is 91.39. These results highlight the critical roleof an appropriate pre-training strategy and show thatlarger ViT models are able to learn Complex image representations, making them ideal for tasks requiring higher accuracy.
7)Pre-training plan:In Figure 8(g), we show the fine-tuning results of the model trained in different pre-training periods , evaluated using macro-mAP and micro-mAP indicators respectively. Notably, the model pretrained only 50 times showed significant performance improvement compared to the model trained from scratch. The trend observed in the figure shows that the model continues to benefit from longer pre-training epochs, suggesting that extended training can further improve performance. Furthermore, the results in Table VI reinforce this finding, as ViT-L and ViT-H consistently achieve higher mAP compared to ViT-B, highlighting the extended pre-training and Validity of larger model architecture.
F. Visual Comparison and Geographical Feature Restorability
Take different masking ratios (i.e. 50%, 75%, 90% and 95%) as input, Figure 9 shows visually Image reconstruction results obtained using SatMAE and our SpectralGPT. As expected, as the masking ratio increases, the reconstructed image deviates more from the original image. However, it is worth emphasizing that the proposed SpectralGPT significantly outperforms SatMAE in spectral image reconstruction performance, especially in preserving visual structure and texture details . Specifically, when using 50% of visible patches, SatMAE's reconstruction results are comparable to those using SpectralGPT, although some details in the SatMAE results are slightly blurred. As the mask ratio increases (e.g., from 75% mask to 90% to 95%), SatMAE's reconstruction performance drops significantly. In comparison, our SpectralGPT exhibits superior reconstruction capabilities (see SatMAE). Even if the masking rate exceeds 90%, key structural and shape components are still retained in the vision, This shows that our model has strong learning, reasoning and generalization capabilities.
In addition to an in-depth discussion and sensitivity analysis on masking ratios, we also conduct a more extensive investigation of spectral reconstruction capabilities by using only 10% of the visible patches, with the remainder masked. These studies utilize various spectral band combinations, prioritizing the representation of geographical features. As shown inFigure 10, we show the visualization of eight different band combinations. These visualizations clearly highlight the significant advantages of our proposed SpectralGPT (closer to the generated original image) especially in terms of band spectrum reconstruction capabilities and its application value in the context of EO missions. In our study, we identified eight geological features that correspond to the observation targets in practical applications, as detailed in Table 7. Furthermore, there are clear visual differences between the geological features obtained using SatMAE and SpectralGPT. These apparent visual differences can be attributed to spectral degradation caused by SatMAE's relatively limited reconstruction and inference capabilities compared to our more powerful SpectralGPT.
in conclusion
The explosive development of basic models represents a major technological revolution after the emergence of deep learning. Currently, various industries are witnessing major leaps in technological and application advancements, driven in large part by the emergence of foundational models. The RS field is no exception, with many EO applications reaping significant benefits. Spectral imaging has been recognized by EO for its ability to provide rich insights into the composition of observed objects and materials, making it a transformative technology with huge potential to address global challenges and reshape various industries. However, the expanding availability of spectral data from various RS platforms undoubtedly poses significant challenges. There is an urgent need to develop basic models specifically designed for spectral remote sensing data. In order to fully unleash and exploit the potential of spectral remote sensing data, several challenging obstacles must be overcome and addressed. This includeseffectively processing and utilizing various RS spectral big datafrom different sources,from complex spatial-spectral Extract meaningful knowledge representation from mixed information,and solve the spectral degradation problem of adjacent spectral correlation modeling.
To address these challenges, we propose SpectralGPT, a customized spectral RS base model with a novel 3D GPT architecture. With its innovative 3D GPT architecture, training on more than 1 million spectral images and more than 600 million parameters, SpectralGPT gives Spectral RS the intelligent processing capabilities of big data. SpectralGPT can flexibly handle a variety of inputs in terms of size, resolution, time rate of change, and geographic coverage. This 3D masking strategy can effectively extract information from spatial-spectral coupling tokens. Additionally, innovative multi-objective reconstruction is able to capture sequence-preserving spectral signatures. characteristics while reducing spectrum degradation. Notably, our progressive training mode enhances the capabilities of the base model beyond a transition point in performance. These breakthroughs achieved by SpectralGPT democratize access to Spectral RS big data, making it more accessible and cost-effective for large-scale EO applications.
Our study also includes a comprehensive evaluation of MAE-based pre-trained base models, focusing on spectral reconstruction capabilities. We systematically evaluate the model's performance with inputs ranging from 50% to as low as 5% of visible markers. This extensive analysis allows us to measure their proficiency in spectral reconstruction and inference, particularly in geographical areas such as agriculture, nature, oceanography, geology and vegetation. The band combinations of the reconstructed spectral images were visualized using SatMAE and SpectralGPT, demonstrating the potential of the latter in practical EO tasks and Geo-field applications.
Looking forward, our research will pursue several goals. We plan to expand the amount and diversity of RS data used for training to includevarious modalities, resolutions, time series, and image sizes. This enrichment will enhance the robustness of the RS base model. Additionally, we aim to expand SpectralGPT’s capabilities by integrating a wider range of downstream tasks. This will make SpectralGPT a general artificial intelligence model with better generalization capabilities, ideal for various EO and earth science applications.