Google Earth Engine (GEE) - Landsat 8TI/TOA/SR image comparison analysis difference and cloud removal is NDVI calculation

• 1 Introduction
• 2 Background
• 3 Preprocessing with Landsat 8 in Google Earth Engine
  • 3.1 Preprocessing level example.
  • 3.2 Top of Atmosphere (TOA)
  • 3.3 Surface Reflectivity (SR)
  • 3.4 Complete code for Landsat image comparison
  • 3.5 Cloud Shading
    • 3.5.1 Single Image Masking: Part 1
    • 3.5.2 Single Image Masking: Part 2
    • 3.5.3 Masking across multiple dates
    • 3.5.4 Visualizing image counts
    • 3.5.5 Generating and exporting histograms
• 4 Conclusion
• 5 Complete Cloud Shield Codes
• 6 References

1 Introduction

In this module, we will discuss the following concepts:

1. Learn about the types of data corrections commonly applied to remote sensing imagery.
2. How to visually compare spatial data at different preprocessing levels in the same dataset.
3. How to perform cloud occlusion and cloud occlusion evaluation for Landsat 8 surface reflectance images in Google Earth Engine.

2 Background

What is preprocessing?
Most of the data you'll find in Google Earth Engine (GEE) is preprocessed to some degree. This involves a number of different quality control methods to ensure the highest level of accuracy and consistency in the raster collection. Depending on the collection, there may be multiple levels of preprocessing available, and it is important to understand the differences to successfully integrate remote sensing data into ecological studies. Before making data available in GEE, publishers consistently address three common sources of error in imagery products: atmosphere (ie, air chemistry), topography (ie, elevation), and geometry (ie, pixel consistency).

Atmospheric Correction
The atmosphere acts as a great hindrance when solar energy bounces off the Earth's surface and returns to our sensors in space. This happens in the form of scattering and absorption (see Module 3 for more information ). Identifying and correcting for these effects is important for accurately representing and interpreting real surface conditions, such as tree species leaf pigments or differences between urban and agricultural pixels.

Terrain and Terrain Correction
The lighting effects of slope, aspect, and elevation present additional challenges for collecting and processing remote sensing data. Various correction methods have been developed, including the use of digital elevation models, to develop predictions for problematic terrain. If your research was conducted at high altitudes or in flat terrain, you'll be relieved to know that preprocessing for terrain effects has been handled by experts (although with caution, manual methods do exist).

Geometric Correction
This process ensures that the alignment of raster images is systematic and consistent over time and relative to each other. For Landsat, the georeferencing and orthorectification process was done with independent ground control points and a previously created digital elevation model. For archival datasets like Landsat, making sure the pixels are aligned over and over again, year after year, is paramount. Otherwise, remote sensing scientists and ecologists will have little ability to perform multitemporal analysis.

It's important to remember that none of these quality assurance methods are 100% foolproof! Follow the "know your data" motto and scrutinize your images from both a qualitative and quantitative perspective. We'll show a few examples of this later in this module.
 

3 Preprocessing with Landsat 8 in Google Earth Engine

Having (free!) dedicated support and behind-the-scenes work before data is available in Google Earth Engine is an incredible advantage. However, you may still find it necessary to manipulate your dataset of interest to facilitate specific research applications. In this module, we will work with Landsat 8 data, the diagram below details several use cases for different processing levels.
 

Decision Workflow from Young et al. , 2017, showing suggested use cases for different levels of Landsat data preprocessing.

3.1 Preprocessing level example.

To qualitatively understand the differences between different preprocessing levels, we can look at several truecolor images of southern Oregon, USA, in late summer 2018. During this time frame, the Carr Fire from Northern California . To evaluate our initial image, we will load the "raw" Landsat 8 collection. Raw data (also referred to as "radiation at the sensor") have not been corrected for any form of potential impact and are generally not used for ecological studies. However, it is helpful to establish a baseline for upcoming levels. Run the script below to generate an image similar to the one below.

NOTE : In addition to the following preprocessing levels, Landsat data has been divided into two quality levels, Tier 1 and Tier 2. Tier 1 is the higher quality option. Click here to learn more about the differences between the tiers.

// 设置影像中心和缩放倍数
Map.setCenter(-122.3158, 42.4494, 12);

//导入影像
var raw = ee.Image('LANDSAT/LC08/C01/T1/LC08_045031_20180811');

// 可视化参数
var rawvis = {bands: ['B4','B3','B2'], min: 0.0, max: 30000.0, gamma: 1};

// 加载图层
Map.addLayer(raw, rawvis, 'raw');

Original Landsat 8 image from southern Oregon. Our image is centered on Mt McLoughlin (2,893 m) with Upper Klamath Lake to the east. If you zoom out, you can see the cities of Medford (west) and Klamath Falls (east).

3.2 Top of Atmosphere (TOA)

The next level of preprocessing takes our "raw" data and applies corrections for the effects of solar activity, including solar irradiance, Earth-Sun distance, and solar elevation. For researchers, the Top of Atmosphere (TOA) is often useful for evaluating single-date, single-scenario imagery (i.e., classification of land cover in a relatively small study area). This is due to varying degrees of solar effects depending on the date, time and latitude of the collection. Append the following code to your script and the resulting image will look similar to the one below.

// 对比toa数据
var toa = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_045031_20180811');

// 定义真彩色影像
var toavis = {bands: ['B4','B3','B2'], min: 0.0, max: 0.4};

// 加载toa图层
Map.addLayer(toa, toavis, 'toa');

Loading the TOA collection, some images look sharper due to some corrections for solar effects. However, it seems there is still work to be done!

3.3 Surface Reflectivity (SR)

This data is preprocessed at the highest level in an attempt to best represent what is actually happening on the ground, where a certain amount of solar energy is reflected back (reflected) to airborne and spaceborne sensors. However, even surface reflectance products are adversely affected by low sun angles, excessive cloud cover, and coverage beyond 65 degrees north-south latitude (Young et al., 2017). Nonetheless, it is recommended to use Landsat surface reflectance data for analysis over multiple dates (eg, change detection) or large geographic ranges (eg, algorithmic forecasting). Add this last piece of code to your script to see the image below in your map viewer pane.

// 加载一张sr影像
var sr = ee.Image('LANDSAT/LC08/C01/T1_SR/LC08_045031_20180811');

// 定义真彩色参数
var srvis = {bands: ['B4','B3','B2'], min: 0, max: 3000, gamma: 1.4};

// 加载图层
Map.addLayer(sr, srvis, 'sr');

Applying atmospheric correction appears to have greatly improved the clarity of our images, especially in the agricultural areas west of Upper Klamath Lake and Mount McGlorin.

3.4 Complete code for Landsat image comparison

The code for these examples is from a specific Landsat image, but you now have the framework to survey any region of interest (ie your study area) to compare different levels of preprocessing.

// 设置图形展示中心
Map.setCenter(-122.3158, 42.4494, 12);

// 导入一景影像
var raw = ee.Image('LANDSAT/LC08/C01/T1/LC08_045031_20180811');

//定义真彩色影像
var rawvis = {bands: ['B4','B3','B2'], min: 0.0, max: 30000.0, gamma: 1};

// 加载此图层
Map.addLayer(raw, rawvis, 'raw');

//比较，从第1级陆地卫星8号大气层顶部数据加载一张图像。
var toa = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_045031_20180811');

//定义真彩色影像
var toavis = {bands: ['B4','B3','B2'], min: 0.0, max: 0.4};

// 加载此图层
Map.addLayer(toa, toavis, 'toa');

// 加载地表反射率数据
var sr = ee.Image('LANDSAT/LC08/C01/T1_SR/LC08_045031_20180811');

// 定义真彩色影像
var srvis = {bands: ['B4','B3','B2'], min: 0, max: 3000, gamma: 1.4};

// 加载sr影像
Map.addLayer(sr, srvis, 'sr');

3.5 Cloud Shading

As we found out, by the time we get to the surface reflection product, the preprocessing for atmospheric, topographic and geometric corrections has already been done. Before making Landsat data available in Google Earth Engine, a very important step in processing Landsat data not included is the removal of near-Earth weather phenomena. This usually comes in the form of a cloud. Clouds are especially prevalent over the tropics (i.e. dense rainforests) and bodies of water. In this section, we'll look at the latter in the Quetico-Superior region of northeastern Minnesota and southwestern Ontario, where terrain is elusive in summer due to extensive cloud cover, partly composed of hundreds of small to Medium sized lake.

Summer clouds gather over Lake Jean in Quitico Provincial Park. Image credit: Tey Shulk .

3.5.1 Single Image Masking: Part 1

Let's start by loading an image that we know is cloudy. There are plenty of options to choose from in this area, but this late August image shows us clouds in many forms. Another thing to consider is that clouds have the added effect of casting shadows on the land below, which further expands the geographic range we eventually have to remove. Start a new script and run the code below to generate the image shown below.

// 加载一个初始图像来测试查看多云的图像。
var sr = ee.Image('LANDSAT/LC08/C01/T1_SR/LC08_027026_20180829');

// 为真彩色图像定义visparams。
var srvis = {bands: ['B4', 'B3', 'B2'], min: 0, max: 3000, gamma: 1.4};

// 把你的多云图像添加到地图上。
Map.addLayer(sr, srvis, 'single_scene');
Map.setCenter(-91.8859, 48.8936, 8.81);

Cloudy image over the country of Quetico-Superior.

3.5.2 Single Image Masking: Part 2

Now for our cloud problem. Landsat provides a pixel_qa band, in short, it assigns different values ​​based on previously quantized features such as the likelihood of clouds and fog. You'll find more complex code for building cloud masks, but this is an easy, conservative way to remove those pesky white specks from your image. Append the following code to your existing script and rerun to see something similar to the image below. Remember to uncheck "single_scene" in the layer controls!

// 定义简单的云层掩码，基于 "pixel_qa "波段的数值。
//本质上，322=土地，324=水的值。
var qa = sr.select('pixel_qa');
var mask = qa.eq(322).or(qa.eq(324));
var sr_cm = sr.updateMask(mask);

// 加载图层

Map.addLayer(sr_cm, srvis, 'single_scene_masked');

Clouds and cloud shadows are successfully removed, but the resulting image does not leave many usable pixels.

3.5.3 Masking across multiple dates

In a single Landsat scene with a cloud mask, we lose quite a bit of geographic coverage. But there is another way! We can also apply a mask over a date range. To do this, we need to create a function, which we'll cover in detail in Module 9 . Now, use the function below (and the rest of the code) to continue your script. Append the code to your existing script.

While Google Earth Engine does have pre-made NDVI image sets from Landsat, these datasets are only available in 2017. Therefore, we will also calculate the NDVI and add it to our image set. This will allow us to generate a median value for each pixel for the entire 2018 growing season, measuring the vegetation health of the study area. Rerun your code and the resulting image should look like this.

// 定义去云函数，这里用QA波段
var cloudMask = function(image) {
  // var mask = image.select('pixel_qa').eq(322).or(image.select('pixel_qa').eq(324));
  var mask = image.select('pixel_qa').eq(322);
  return image.mask(mask);
};

// 使用NDVI
var collection = ee.ImageCollection("LANDSAT/LC08/C01/T1_SR")
.filterDate('2018-06-01','2018-09-30')
.filter(ee.Filter.calendarRange(140, 270)) // roughly, our potential growing season
.filterMetadata('WRS_PATH', 'equals', 27)
.filterMetadata('WRS_ROW', 'equals', 26)
.map(cloudMask)
.map(function NDVI(i){
  return i.addBands(i.normalizedDifference(['B5','B4']).rename('NDVI'))  // generates NDVI band
});

var median = collection.median();

// 红色=低值，深绿色=高值
var collectionVis = {bands: 'NDVI', min: 0.5, max: 0.95, palette: ['red','yellow','green','003300']};

Map.addLayer(median, collectionVis, 'NDVI');

Visualize the median NDVI for the entire 2018 growing season. Feel free to switch between different background layers. Your image may not be exactly the same as the one shown here.

3.5.4 Visualizing image counts

Our NDVI image looks good. But how confident are we in these values? Specifically, it is important to assess how many images actually make up our median. We can quickly check whether the median NDVI we get is representative of the entire study area by visualizing the sum of the number of images used at each pixel location. Append the following code to your script and click Run. You should see an image similar to the image below.

For the count layer, if we find spatial anomalies in the NDVI values, there are several options. We can expand our seasonal date range or choose to include multiple years of data. Eventually, we may accept failure and decide that the weather is too cloudy to be usable and decide to explore a different dataset - which is perfectly acceptable!

// 统计ndvi波段数量
var counts = collection.select('NDVI').count();

//获取影像数量
print(collection.size());

// 在这个调色板中，红色=低值，蓝色=高值。
var countVis = {min: 0, max: 6, palette: ['b2182b','ef8a62','fddbc7', 
                                          'd1e5f0','67a9cf','2166ac']};

Map.addLayer(counts, countVis, 'counts');

Visualize the number of images used to calculate the median per pixel. Darker reds have lower values, and darker blues have higher values.

3.5.5 Generating and exporting histograms

To understand the quantitative distribution of count values, we can construct a histogram of the count data. If our study area is cloudy so that the average represents only one or two values, we may need to reconsider our source data or collection year. Add this final code block to your script and you will be able to see the histogram in the Console tab. You will make your own geometric shapes (see Module 1 if you need a refresher on how to do this ). Your histogram may vary slightly depending on your shape, but it is large enough to contain the image and it should resemble the distribution in the image below.

//画出你自己的矩形，并尝试覆盖大部分的Landsat场景 
// 然后，我们将制作一个分布的直方图。
//由于场景面积大，GEE要求我们进行一些聚合。 
// 让我们对我们的分布有一个大致的了解。
var histogram = ui.Chart.image.histogram(counts, geometry, 120);

// 展示直方图
print(histogram);

Use the histogram function to quantify the number of distributions of values ​​from the count layer.

4 Conclusion

In this unit, we review some common corrections applied to remote sensing imagery that aid in the production of the high-quality products you'll find in Google Earth Engine. We also introduce a simple framework for visualizing these differences and see how changes in treatment levels affect the resulting images during a smoky summer in southern Oregon. Finally, we built a workflow script to use Google Earth Engine to remove clouds from growing season images, generate mean vegetation index values, and evaluate the distribution of the images used.

5 Complete Cloud Shield Codes

// Load an initial image to test view a cloudy image.
var sr = ee.Image('LANDSAT/LC08/C01/T1_SR/LC08_027026_20180829');

// Define visparams for true-color image.
var srvis = {bands: ['B4', 'B3', 'B2'], min: 0, max: 3000, gamma: 1.4};

// Add your cloudy image to the map.
Map.addLayer(sr, srvis, 'single_scene')
Map.setCenter(-91.8859, 48.8936, 8.81);

// Define simple cloud mask, based in values from the 'pixel_qa' band.
// Essentially, values of 322 = land and 324 = water.
var qa = sr.select('pixel_qa');
var mask = qa.eq(322).or(qa.eq(324));
var sr_cm = sr.updateMask(mask);

// Add your new map for comparison to the cloudy image.
Map.addLayer(sr_cm, srvis, 'single_scene_masked');

// Define how your cloudMask function should work.
var cloudMask = function(image) {
  // var mask = image.select('pixel_qa').eq(322).or(image.select('pixel_qa').eq(324));
  var mask = image.select('pixel_qa').eq(322);
  return image.mask(mask);
};

// This time we'll look at a more ecologically-focused example, using NDVI.
var collection = ee.ImageCollection("LANDSAT/LC08/C01/T1_SR")
.filterDate('2018-06-01','2018-09-30')
.filter(ee.Filter.calendarRange(140, 270)) // roughly, our potential growing season
.filterMetadata('WRS_PATH', 'equals', 27)
.filterMetadata('WRS_ROW', 'equals', 26)
.map(cloudMask)
.map(function NDVI(i){
  return i.addBands(i.normalizedDifference(['B5','B4']).rename('NDVI'))  // generates NDVI band
});

var median = collection.median();

// Red = lower values and darker green = higher values.
var collectionVis = {bands: 'NDVI', min: 0.5, max: 0.95, palette: ['red','yellow','green','003300']};

Map.addLayer(median, collectionVis, 'NDVI');

// Create counts band.
var counts = collection.select('NDVI').count();

// Find our potential maximum count by getting the size of the initial image collection.
print(collection.size());

// In this palette, red = lower values and blue = higher values.
var countVis = {min: 0, max: 6, palette: ['b2182b','ef8a62','fddbc7', 
                                          'd1e5f0','67a9cf','2166ac']};

Map.addLayer(counts, countVis, 'counts');

// Draw your own rectangle and try to cover most of the Landsat scene and 
// then we'll make a histogram of the distribution. Because the scene area 
// is so large  GEE requires us to perform some aggregating. But this will 
// give us a general sense of our distribution.
var histogram = ui.Chart.image.histogram(counts, geometry, 120);

// Display the histogram.
print(histogram);

 

6 References

Young, NE, Anderson, RS, Chignell, SM, Vorster, AG, Lawrence, R., & Evangelista, PH (2017). Survival guide for Landsat preprocessing. Ecology, 98(4), 920-932.