NLDAS3 Download with fsspec, zarr, and xarray is sloooooooooooow#
TLDR; Added NLDAS3 download to tsgettoolbox, but it is sloooooooooooow. Based on information github.com/NASAWaterInsight/NLDAS-3 and github.com/NASAWaterInsight/NLDAS-3/blob/develop/scripts/read_nldas3_kerchunk_from_refs.py. (neither of which actually worked for me). Was necessary to include the fix described here: github.com/NASAWaterInsight/NLDAS-3/pull/43/changes
The following code snippet is a working minimal example of how to use fsspec, zarr, and xarray to download NLDAS3 data from AWS S3. It takes almost 6 minutes to download a year’s worth of hourly data for two variables at one (lat, lon) point.:
import fsspec
import xarray as xr
import pandas as pd
time_interval = "hourly" # or "daily"
variables = ["Rainf", "Tair"]
startDate = pd.to_datetime("2020-01-01")
endDate = pd.to_datetime("2020-12-31")
lon = -82.5
lat = 29.5
## Set up a reference file system based on the chunk refs stored in
## the parquet file. This only loads the metadata needed to create
## the impression of a zarr store on the local machine.
## This essentially abstracts away the difference between individual
## netCDF files on the s3 bucket by mapping zarr chunks to a
## combination of URLs and byte offsets/lengths of netCDF chunks
## under each URL
ref_fs = fsspec.filesystem(
"reference",
fo=f"s3://nasa-waterinsight/virtual/nldas3_{time_interval}.parq",
remote_protocol="s3",
asynchronous=True,
remote_options={"asynchronous": True, "anon": True},
target_options={"anon": True},
lazy=True,
)
with xr.open_dataset(
ref_fs.get_mapper(""),
engine="zarr",
decode_times=True,
backend_kwargs={"consolidated": False},
) as ds:
## use data coordinates to identify a subset of the data to retrieve.
sub = ds.sel(lon=lon, lat=lat, method="nearest").sel(
time=slice(startDate, endDate)
)
# Do this here instead of in the loop below to avoid repeated calls to
# load() for each variable.
index = sub.time.load()
ndf = pd.DataFrame()
for get_var in variables:
df = pd.DataFrame(
sub[get_var].load(),
index=index,
columns=[f"{get_var}"],
)
df.index.name = "Datetime:UTC"
df = df.tz_localize("UTC")
ndf = ndf.combine_first(df)
print(ndf)
Would print out:
Rainf Tair
Datetime:UTC
2020-01-01 00:00:00+00:00 7.691921e-09 283.439056
2020-01-01 01:00:00+00:00 2.010366e-08 282.842987
2020-01-01 02:00:00+00:00 3.069147e-08 282.317627
2020-01-01 03:00:00+00:00 2.899162e-09 281.904449
2020-01-01 04:00:00+00:00 3.427521e-11 281.598297
... ... ...
2020-12-30 20:00:00+00:00 1.134291e-03 297.592377
2020-12-30 21:00:00+00:00 4.624545e-04 296.677246
2020-12-30 22:00:00+00:00 2.785645e-04 294.661896
2020-12-30 23:00:00+00:00 1.941294e-04 292.758759
2020-12-31 00:00:00+00:00 2.498008e-04 291.901154
[8761 rows x 2 columns]
I am only lazy downloading one variable at a time from one (lat, lon) point. I guess that the expense is having to open up 1000s of hourly files on AWS S3 to extract the variable at the point.
NLDAS2 reformatted the data to be more efficient for downloading, calling them “rods”. I don’t know the particulars of what they did for NLDAS2, but it is much faster than NLDAS3 by orders of magnitude.
Here is Something to Chew On - There is More to Gum#
Supposed to have available the NOAH-MP land surface model data, but it is not available yet. Should be easy to add to “tsgettoolbox” once they are available.
See the 26.4.0 or greater version of “tsgettoolbox” for the NLDAS3 download
functionality. Install with pip install tsgettoolbox or conda install -c
conda-forge tsgettoolbox as appropriate for your virtual environment.
The tsgettoolbox command line interface is described here:
https://timcera.bitbucket.io/linked_pkg_dirs/tsgettoolbox/docs/command_line.html#ldas-nldas3-forcing
The tsgettoolbox Python interface is described here:
Comments
comments powered by Disqus