.. post:: 2026-10-07
   :tags: zarr, xarray, nldas3, fsspec
   :category: tsgettoolbox
   :author: Tim Cera
   :location: Gainesville, FL
   :language: en

NLDAS3 Download with fsspec, zarr, and xarray is sloooooooooooow
----------------------------------------------------------------
TLDR; Added NLDAS3 download to tsgettoolbox, but it is sloooooooooooow.  Based
on information `github.com/NASAWaterInsight/NLDAS-3 <https://github.com/NASAWaterInsight/NLDAS-3>`__ and
`github.com/NASAWaterInsight/NLDAS-3/blob/develop/scripts/read_nldas3_kerchunk_from_refs.py <https://github.com/NASAWaterInsight/NLDAS-3/blob/develop/scripts/read_nldas3_kerchunk_from_refs.py>`__.
(neither of which actually worked for me).  Was necessary to include the fix
described here: `github.com/NASAWaterInsight/NLDAS-3/pull/43/changes <https://github.com/NASAWaterInsight/NLDAS-3/pull/43/changes>`__

The following code snippet is a working minimal example of how to use fsspec,
zarr, and xarray to download NLDAS3 data from AWS S3.  It takes almost
6 minutes to download a year's worth of hourly data for two variables at one
(lat, lon) point.::

    import fsspec
    import xarray as xr
    import pandas as pd

    time_interval = "hourly"  # or "daily"
    variables = ["Rainf", "Tair"]
    startDate = pd.to_datetime("2020-01-01")
    endDate = pd.to_datetime("2020-12-31")
    lon = -82.5
    lat = 29.5

    ## Set up a reference file system based on the chunk refs stored in
    ## the parquet file. This only loads the metadata needed to create
    ## the impression of a zarr store on the local machine.
    ## This essentially abstracts away the difference between individual
    ## netCDF files on the s3 bucket by mapping zarr chunks to a
    ## combination of URLs and byte offsets/lengths of netCDF chunks
    ## under each URL
    ref_fs = fsspec.filesystem(
        "reference",
        fo=f"s3://nasa-waterinsight/virtual/nldas3_{time_interval}.parq",
        remote_protocol="s3",
        asynchronous=True,
        remote_options={"asynchronous": True, "anon": True},
        target_options={"anon": True},
        lazy=True,
    )

    with xr.open_dataset(
        ref_fs.get_mapper(""),
        engine="zarr",
        decode_times=True,
        backend_kwargs={"consolidated": False},
    ) as ds:
        ## use data coordinates to identify a subset of the data to retrieve.
        sub = ds.sel(lon=lon, lat=lat, method="nearest").sel(
            time=slice(startDate, endDate)
        )

        # Do this here instead of in the loop below to avoid repeated calls to
        # load() for each variable.
        index = sub.time.load()

        ndf = pd.DataFrame()
        for get_var in variables:
            df = pd.DataFrame(
                sub[get_var].load(),
                index=index,
                columns=[f"{get_var}"],
            )
            df.index.name = "Datetime:UTC"
            df = df.tz_localize("UTC")

            ndf = ndf.combine_first(df)

    print(ndf)

Would print out::

                                      Rainf        Tair
    Datetime:UTC                                       
    2020-01-01 00:00:00+00:00  7.691921e-09  283.439056
    2020-01-01 01:00:00+00:00  2.010366e-08  282.842987
    2020-01-01 02:00:00+00:00  3.069147e-08  282.317627
    2020-01-01 03:00:00+00:00  2.899162e-09  281.904449
    2020-01-01 04:00:00+00:00  3.427521e-11  281.598297
    ...                                 ...         ...
    2020-12-30 20:00:00+00:00  1.134291e-03  297.592377
    2020-12-30 21:00:00+00:00  4.624545e-04  296.677246
    2020-12-30 22:00:00+00:00  2.785645e-04  294.661896
    2020-12-30 23:00:00+00:00  1.941294e-04  292.758759
    2020-12-31 00:00:00+00:00  2.498008e-04  291.901154
    
    [8761 rows x 2 columns]

I am only lazy downloading one variable at a time from one (lat, lon) point.
I guess that the expense is having to open up 1000s of hourly files on AWS S3
to extract the variable at the point.

NLDAS2 reformatted the data to be more efficient for downloading, calling them
"rods".  I don't know the particulars of what they did for NLDAS2, but it is
much faster than NLDAS3 by orders of magnitude.

Here is Something to Chew On - There is More to Gum
---------------------------------------------------

Supposed to have available the NOAH-MP land surface model data, but it is not
available yet.  Should be easy to add to "tsgettoolbox" once they are
available.

See the 26.4.0 or greater version of "tsgettoolbox" for the NLDAS3 download
functionality.  Install with ``pip install tsgettoolbox`` or ``conda install -c
conda-forge tsgettoolbox`` as appropriate for your virtual environment.

:ref:`tsgettoolbox_documentation`

The tsgettoolbox command line interface is described here:

https://timcera.bitbucket.io/linked_pkg_dirs/tsgettoolbox/docs/command_line.html#ldas-nldas3-forcing

The tsgettoolbox Python interface is described here:

https://timcera.bitbucket.io/linked_pkg_dirs/tsgettoolbox/docs/_function_autosummary/tsgettoolbox.ldas_nldas3_forcing.html