I am trying to calculate the correlation along the time dimension between two datasets in xarray. My datasets are all lat x lon x time. One of the datasets has so much missing data that it doesn't make sense to interpolate and remove missing values, instead I want missing values to be ignored. I have some simple codes, but none of them fit my specific situation. For example:
def covariance(x,y,dims=None):
return xr.dot(x-x.mean(dims), y-y.mean(dims), dims=dims) / x.count(dims)
def correlation(x,y,dims=None):
return covariance(x,y,dims) / (x.std(dims) * y.std(dims))Works great without missing data, but of course doesn't work with nans. While there is a nice example written for xarray here, even with this code, I'm having trouble computing pearson correlations instead of spearman correlations.
import numpy as np
import xarray as xr
import bottleneck
def covariance_gufunc(x, y):
return ((x - x.mean(axis=-1, keepdims=True))
* (y - y.mean(axis=-1, keepdims=True))).mean(axis=-1)
def pearson_correlation_gufunc(x, y):
return covariance_gufunc(x, y) / (x.std(axis=-1) * y.std(axis=-1))
def spearman_correlation_gufunc(x, y):
x_ranks = bottleneck.rankd