Python-GEE Remote Sensing Cloud Big Data Analysis, Management and Visualization
In recent years, remote sensing technology has developed by leaps and bounds. Multiple remote sensing platforms such as aerospace, aviation, and near space have continued to increase. The spatial, temporal, and spectral resolution of data has continued to increase. The amount of data has soared. Remote sensing data has become more and more characteristic of big data. . The emergence of remote sensing big data provides unprecedented opportunities for related research, but how to deal with these data also poses great challenges. Traditional workstations and servers have been unable to meet the needs of large-scale, multi-scale massive remote sensing data processing.
Google Earth Engine is an online visualization computing and analysis cloud platform for global-scale earth science data (especially satellite remote sensing data) provided by Google. The platform can access and synchronize satellite images such as MODIS, Landsat and Sentinel, and meteorological reanalysis data sets such as NCEP, which are commonly used in the field of remote sensing. At the same time, relying on millions of super servers around the world, it provides sufficient computing power to process these data. Up to now, there are more than 200 public datasets on GEE, and more than 4,000 new images are added every day, with a capacity of more than 50PB. Compared with traditional image processing tools such as ENVI, Google Earth Engine has incomparable advantages in processing massive remote sensing data. On the one hand, the GEE platform provides abundant computing resources, and on the other hand, its huge cloud storage saves a lot of research personnel time for data download and preprocessing. It can be said that GEE represents the most advanced level in this field in the world in terms of calculation and analysis and visualization of remote sensing data, and it is a revolution in the field of remote sensing.
At present, GEE has been valued and applied by more and more foreign scientific and technological workers for its powerful functions, but its domestic application is still very limited. GEE provides APIs based on Javascript and Python languages. The former is the programming language of the official main platform. It is easy to use but has shortcomings in input and output and drawing visualization. Python, as the most popular programming language at present, can make up for Javascript in this area. There are deficiencies in the aspect, and it is more convenient for batch processing and machine learning.
Chapter 1, Theoretical Basis
Introduction of Earth Engine platform and application, main data resources
Important concepts, data types and objects of Earth Engine remote sensing cloud
Comparison and selection of JavaScript and Python remote sensing cloud programming
Python basics (syntax, data types and program control structures, functions and classes and objects, etc.)
Commonly used Python software packages ((pandas, numpy, os, etc.) introduction and basic function demonstration (Excel/csv data file reading and data processing, directory operation, etc.)
6. Differences between JavaScript and Python remote sensing cloud APIs, learning methods and resource recommendations
Chapter 2, Development Environment Construction
Introduction to local and cloud Python remote sensing cloud development environments
Build the local development environment
Anaconda installation, pip/conda package installation method and virtual environment creation, etc.;
Installation of necessary software packages such as earthengine-api and geemap;
Remote sensing cloud local authorization management;
Jupyter Notebook/Visual Studio Code installation and running debugging.
3. Cloud Colab development environment construction
4. Geemap introduction and common function demonstration
Chapter 3, Basics of Remote Sensing Big Data Processing
Introduction to the image data analysis and processing process of the remote sensing cloud platform: Introduce the basic framework of the image data analysis and processing process of the remote sensing cloud platform, including data acquisition, data preprocessing, algorithm development, visualization, etc.
Object display and attribute field exploration of elements and images: Introduces how to display and explore attribute fields of objects such as elements and images on the remote sensing cloud platform, including how to select elements and image objects, view attribute information, filter data, etc.
Time, space and attribute filtering method of image/feature set: Introduce how to filter image/feature set in time, space and attribute, including how to select time period, geographical area and attribute conditions to achieve more accurate data analysis.
Band operations, conditional operations, vegetation index calculations, clipping and mosaic, etc .: Introduce how to perform band operations, conditional operations, vegetation index calculations, clipping and mosaic operations on the remote sensing cloud platform to achieve more in-depth data analysis.
Declouding common optical images such as Landsat/Sentinel-2 : Introduces how to use different methods to remove clouds in commonly used optical images such as Landsat/Sentinel-2 on the remote sensing cloud platform to improve the quality of image data.
Iterative cycle of images and feature sets: Introduce how to use the iterative cycle function of the remote sensing cloud platform to batch process images and feature sets to improve data analysis efficiency.
Image data integration (Reducer): Introduce how to use the Reducer function of the remote sensing cloud platform to integrate multiple image data into one data set to facilitate subsequent data analysis.
Neighborhood Analysis and Spatial Statistics: Introduce how to perform neighborhood analysis and spatial statistics on the remote sensing cloud platform to obtain more in-depth spatial information.
Common errors and code optimization: Introduce common errors in the data analysis process of the remote sensing cloud platform and how to optimize the code to improve the efficiency and accuracy of data analysis.
10. Python remote sensing cloud data analysis exclusive package construction: Introduce how to use Python to build a data analysis exclusive package on the remote sensing cloud platform to facilitate multiple use and share analysis codes.
Chapter Four, Typical Case Operation Practice
11. Machine learning classification algorithm case: This case combines Landsat and other long-term image series and machine learning algorithms to demonstrate the basic remote sensing classification process at the national scale. The specific content includes image statistics in the study area, spatially stratified random sampling, random sample segmentation, time series image preprocessing and synthesis, machine learning algorithm application, classification postprocessing and accuracy evaluation, etc.
12. Decision tree forest classification algorithm case: In this case, combined with L-band radar and Landsat optical time series images, the decision tree classification algorithm is used to extract the forest distribution map of the designated area from 2007 to 2020, and the spatial comparison is made with JAXA annual forest products. The case involves joint use of multi-source data, construction of decision tree classification algorithm, dynamic optimization of threshold, spatial analysis of classification results, etc.
13. Flood disaster monitoring case: This case is based on Sentinel-1 C-band radar and other images to monitor provincial-level torrential rain disasters. The case content includes Sentinel-1 C image processing, construction of various water body recognition algorithms, image difference analysis, and result visualization. .
14. Drought remote sensing monitoring case : This case uses 40-year-old satellite remote sensing rainfall data products such as CHIRPS to monitor the extreme drought at the provincial scale. The case content includes basic processing of meteorological data, integration of annual and monthly scale data, calculation of long-term average LPA/deviation, and visualization of data results.
15. Case analysis of phenological characteristics : This case is based on time series images such as Landsat and MODIS, and analyzes the phenological differences (sampling point scale) and large-scale (such as China) phenological spatial variation characteristics of typical surface vegetation for many years through the change of vegetation index. The case content includes time series image synthesis, image smoothing (Smoothing) and gap filling (Gap-filling), result visualization, etc.
16. Case of forest vegetation health monitoring: This case uses 20 years of MODIS vegetation index to monitor the forest in a selected area for a long time, and analyze the greening or browning of forest vegetation. Methods for joining and compositing imagery, trend analysis, spatial statistics, and visualization are involved.
17. Case of dynamic monitoring of ecological environment quality : This case uses RSEI remote sensing ecological index and Landsat series images to quickly monitor the ecological status of selected cities. The main technologies involved include calculation of vegetation index, extraction of surface temperature, normalization of data, PCA analysis of principal components, construction of RSEI ecological index and visualization of results, etc.
Chapter 5, Input and Output and Efficient Management of Data Assets
Interaction between local data and cloud: Introduce how to convert local csv, kml, vector and raster data to cloud data, and explain the method of data export.
Batch download of server-side data: including direct local download, batch download of image sets, and how to quickly download large-scale and long-term data products, such as global forest products and 20-year MODIS data products. .
Local data upload and attribute setting: including the use of earthengine commands, how to upload a small amount of local vector and raster data and set attributes (small files), and how to upload data in batches and automatically set attributes, and how to use fast upload techniques Upload very large image files, such as domestic high-resolution images.
4. Personal data asset management: Introduce how to use Python and earthengine command line to manage personal data assets, including creating, deleting, moving, renaming and other operations, and also explain how to cancel upload/download tasks in batches.
Chapter 6, Publication-level Visualization of Cloud Data Papers
Introduction to Python visualization and main software packages: Introduce matplotlib and seaborn visualization packages, explain basic graphics concepts, graphics composition, and quick drawing of commonly used graphics.
Drawing of topography and sample plot distribution map of the research area: Combine local or cloud vector files, cloud topographic data, etc. to draw a schematic diagram of the research area. It involves the drawing process, Chinese display, color matching and beautification, etc. It will also introduce the online download and local application of cpt-city exquisite palette palette.
Image coverage statistics and drawing in the research area: make statistics on the coverage quantity and cloud-free image coverage of Landsat and Sentinel and other series of images in the designated area, and draw regional image statistical maps or pixel-level cloud-free image coverage thematic maps.
Analysis and drawing of sample spectral characteristics and phenological characteristics: Quickly draw spectral and phenological characteristics of different types of plots, dynamically download and integrate thumbnails (thumbnails) and vegetation index time series of sampling points in the past 30 years, etc.
Classification result thematic map drawing and space-time dynamic time-lapse photography Timelapse production: single or multiple classified thematic map drawing and color matching beautification, making Timelapse with clear land use changes, and introducing animation text addition and other content.
6. Area statistics and drawing of classification results: Based on the classification results and vector boundary files in the cloud, count the areas of different land types in different regions, extract the statistical results, and display the statistical areas in different graphics; make statistical drawings of land use changes, etc.
