subsetting with xc.open_mfdataset

[gleckler1]

I am dealing with a large TXY dataset that requires memory management (10000,3600,1800). I am imagining a possibility such as:
f = xc.open_mfdataset(path+'/*.nc', decode_times=False, combine='nested', concat_dim='time', preprocess=extract_date, date_range('1980-1-1','1980-2-1'))

but don't see via xr or xc how to accomplish something akin to the date_range part. I'm sure I'm missing something. Any suggestions?

[pochedls]

xarray doesn't load data on open, so you can subset that data after open with something like ds_sub = ds.sel(time=slice('1980-01-01', '1980-02-01')). If you want to load it into memory, you can call ds_sub.load(). Depending on what you're doing, you could let xarray / dask try to manage the chunking / memory consumption.

[tomvothecoder]

Just an FYI about when to use .load() per xarray docs:

Normally, it should not be necessary to call this method in user code, because all xarray functions should either work on deferred data or load data automatically. However, this method can be necessary when working with many file objects on disk.

• Source: https://docs.xarray.dev/en/stable/generated/xarray.Dataset.load.html

[gleckler1]

Thanks SP. I was thinking along these lines but in doing so the time and time_bnds in "f" (=xc.mfopen) would span the entire time series whereas the ds_sub would have the desired chunk. This would mean I could not use f for subsequent operations, e.g., f = f.bounds.add_missing_bounds() ... I guess the answer involves adding bounds to ds_sub

[tomvothecoder]

You should be able to maintain the existing, subsetted version oftime_bnds in the dataset after subsetting on time. Let me know if this is your intention.

Example:

import xarray as xr
import xcdat as xc

ds = xc.open_dataset(
    "http://psl.noaa.gov/thredds/dodsC/Datasets/noaa.ersst.v5/sst.mnmean.nc"
)

print(ds.time_bnds)
<xarray.DataArray 'time_bnds' (time: 2030, bnds: 2)>
array([[cftime.DatetimeGregorian(1853, 12, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(1854, 1, 16, 12, 0, 0, 0, has_year_zero=False)],
       [cftime.DatetimeGregorian(1854, 1, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(1854, 2, 15, 0, 0, 0, 0, has_year_zero=False)],
       [cftime.DatetimeGregorian(1854, 2, 15, 0, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(1854, 3, 16, 12, 0, 0, 0, has_year_zero=False)],
       ...,
       [cftime.DatetimeGregorian(2022, 11, 16, 0, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(2022, 12, 16, 12, 0, 0, 0, has_year_zero=False)],
       [cftime.DatetimeGregorian(2022, 12, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(2023, 1, 16, 12, 0, 0, 0, has_year_zero=False)],
       [cftime.DatetimeGregorian(2023, 1, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(2023, 2, 16, 12, 0, 0, 0, has_year_zero=False)]],
      dtype=object)
Coordinates:
  * time     (time) object 1854-01-01 00:00:00 ... 2023-02-01 00:00:00
Dimensions without coordinates: bnds
Attributes:
    xcdat_bounds:  True


ds_sub = ds.sel(time=slice('1980-01-01', '1980-02-01'))

print(ds_sub.time_bnds)
<xarray.DataArray 'time_bnds' (time: 2, bnds: 2)>
array([[cftime.DatetimeGregorian(1979, 12, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(1980, 1, 16, 12, 0, 0, 0, has_year_zero=False)],
       [cftime.DatetimeGregorian(1980, 1, 16, 12, 0, 0, 0, has_year_zero=False),
        cftime.DatetimeGregorian(1980, 2, 15, 12, 0, 0, 0, has_year_zero=False)]],
      dtype=object)
Coordinates:
  * time     (time) object 1980-01-01 00:00:00 1980-02-01 00:00:00
Dimensions without coordinates: bnds
Attributes:
    xcdat_bounds:  True