GEE: batch download daily datasets

The biggest difficulty encountered was the data processing problem of the client and server of GEE, but luckily it was resolved.

My question is how to download the daily ozone images of Sentinel-5P on the GEE platform?

Code link (hang): https://code.earthengine.google.com/86171ff00fd095da3f0da02e5872ecd6

01 code description

First, custom parameters.

// 自定义参数
var start_time = '2022-07-01'
var end_time = '2022-08-31'
var province_name = '四川省'

Next, the study area is selected.

//加载自己的研究区
var roi = ee.FeatureCollection("projects/ee-chaoqiezione/assets/china_admin_province");
roi = roi.filter(ee.Filter.eq('省', province_name)).first();

It should be noted that you need to replace "projects/ee-chaoqiezione/assets/china_admin_province" with the vector file path in your own cloud disk Assets or the vector file on the GEE platform, and modify the province_name accordingly.

Next, generate a list of dates for each day between start_time and end_time.

// 生成日期列表
var dates = ee.List.sequence(
  ee.Date(start_time).millis(),   // 将开始日期转化为毫秒级的unix时间戳(1970年至start_time毫秒数)
  ee.Date(end_time).millis(),   // 如此亦是
  24*60*60*1000  // 一天的毫秒数
).map(function(dateMillis){
  return ee.Date(dateMillis).format('YYYY-MM-dd')  // 将unix时间戳转化为年月日的字符串
})

Here is very important content, and the code cannot be changed at will, as follows:
1. The conversion of string dates into Date objects is for the convenience of subsequent list generation, because strings do not have these methods or functions;

2. Convert the start and end dates to unix timestamps. I feel that this is very similar to the Julian day of ENVI IDL. You can refer to it;

3. The step size is set to the number of milliseconds in a day;

4. Then convert the unix timestamp of each day from the generated start date to the end date into a normal date object, and then convert the date object into a string.

The biggest doubt here is why it is necessary to convert the processed date object into a string?

In fact, it is related to some things on the client and server later. We will explain later that the main reason is that the loop is not executed in parallel on the server, but on the client.

Next, iteration time ==> processing and exporting images;

// 迭代时间 ==> 处理和导出影像
dates.evaluate(function(dates) {
  dates.map(function(date){
    var start = ee.Date(date);
    var stop = start.advance(1, 'day');

    var scol= ee.ImageCollection("COPERNICUS/S5P/NRTI/L3_O3")
              .filterBounds(roi)
              .filterDate(start,stop)
              .select("O3_column_number_density");
    
    var before = scol.qualityMosaic("O3_column_number_density").clip(roi);
    before = before.set({name:ee.String(start.format('YYYY-MM-dd'))});
    
    Export.image.toDrive({
      image: before.select("O3_column_number_density"),
      region: roi.geometry(),
      scale:1000,
      description: "O3_column_number_density_1000m_" + province_name + "_" + date,
      folder: 'O3_column_number_density',
    });
  })
})

We used the evaluate function, evaluate()which allows you to execute code on the client side, unlike most Earth Engine methods, which are executed on the server side. The key is to understand that the calculation model of Earth Engine is delayed, parallel, and executed on the server side.

If you try to use map to solve the above content, that is, do not use the evaluate function, then there is a problem with the export, because GEE uses the map function to perform parallel operations on the server side, but GEE can only send a maximum of three export tasks at a time, which is why in the map When trying to export multiple jobs in , only three images are actually exported.

Evaluate is actually an asynchronous operation. When you want to use a function to execute some variables in dates, but these variables are still processed in parallel on the server side. If we use evaluate, then once the variables are processed , then the function will execute these variables. Asynchrony means that when the variables have not been processed, the program will not wait forever, but continue to execute other codes, and then wait for the variables to be processed.

 As for other details, I won’t elaborate on them one by one due to time, just keep one point in mind.

When you want to use methods such as print and toDrive in the map function, it is a good choice to embed an evaluate, especially when you need to complete complex calculations and obtain results on the server side.

Since using map alone is fast, this is because it executes tasks in parallel, so when you use the evaluate method, the program will slow down slightly.

The complete code is as follows:
(Slightly modified, check the difference by yourself)

// 自定义参数
var start_time = '2022-07-01'
var end_time = '2022-08-31'
var province_name = '四川省'


//加载自己的研究区
var roi = ee.FeatureCollection("projects/ee-chaoqiezione/assets/china_admin_province");
roi = roi.filter(ee.Filter.eq('省', province_name));
print('行政区划', roi)
// 生成日期列表
var dates = ee.List.sequence(
  ee.Date(start_time).millis(),   // 将开始日期转化为毫秒级的unix时间戳(1970年至start_time毫秒数)
  ee.Date(end_time).millis(),   // 如此亦是
  24*60*60*1000  // 一天的毫秒数
).map(function(dateMillis){
  return ee.Date(dateMillis).format('YYYY-MM-dd')  // 将unix时间戳转化为年月日的字符串
})
Map.addLayer(roi.style({fillColor: "00000000"}), {width: 2}, province_name)
Map.centerObject(roi, 6)
// 迭代时间 ==> 处理和导出影像
dates.evaluate(function(dates) {
  dates.map(function(date){
    var start = ee.Date(date);
    var stop = start.advance(1, 'day');

    var scol= ee.ImageCollection("COPERNICUS/S5P/NRTI/L3_O3")
              .filterBounds(roi)
              .filterDate(start,stop)
              .select("O3_column_number_density");
    
    var before = scol.qualityMosaic("O3_column_number_density").clip(roi).unmask(-9999);
    before = before.set({name:ee.String(start.format('YYYY-MM-dd'))});
    
    print(before)
    Export.image.toDrive({
      image: before.select("O3_column_number_density"),
      region: roi.geometry(),
      scale:1000,
      description: "O3_column_number_density_1000m_" + province_name + "_" + date,
      folder: 'O3_column_number_density',
    });
  })
})