Datacube v1.8.15

The previous release of the Open Data Cube (1.8.14) replaced a deprecated import API for driver loading. Unfortunately the new API used is only supported by Python>=3.10. This means that v1.8.14 was not compatible with Python 3.8 and 3.9.

The Open Data Cube aims to support Python >= 3.8 and we apologise to the community for this oversight.

This v1.8.15 release replaces the problematic API with one that is supported by Python 3.8 and 3.9 (#1469), and updates whats_new.rst (#1470). There are no other changes in this release.

Thanks to @emmaai for identifying the issue and @Ariana-B for fixing it.

Datacube v1.8.14

Major changes in this release:

1. The attempted fix to the projection bug in the previous release turned out to be inadequate, this release contains a fix for the bug fix.
2. The behaviour of the archive-less-mature option on dataset-add has been enhanced (default time leniency increased to +/-500ms, with the option for the user to set their own leniency threshold.
3. Several old and poorly maintained APIs have been formally deprecated in this release, and will be removed from datacube-core in the 1.9.0 release: ingestion (which the documentation has recommended against using for many years), executors and grid workflow. If you still use any of these APIs, you can continue to use them in this release (and will still be able to use them in all future 1.8.x releases) by simply ignoring the deprecation warnings. If you have workflows that still depend on these APIs, please raise your requirements on the ODC Slack community so we can help you identify alternative approaches. There is no planned release date for v1.9.0 at this stage.

Full list of changes in this release:

• Second attempt to address unexpected handling of image aspect ratios in rasterio and GDAL. (#1457)
• Fix broken PyPI publishing Github action (#1454)
• Documentation improvements (#1455)
• Increase default maturity leniency to +-500ms (#1458)
• Add option to specify maturity timedelta when using --archive-less-mature option (#1460)
• Mark executors as deprecated (#1461)
• Mark ingestion as deprecated (#1463)
• Replace deprecated pkg_resources with importlib.resources and importlib.metadata (#1466)
• Update whats_new.rst for release (#1467)

Includes code contributions from @SpacemanPaul and @Ariana-B.

With thanks to @Kirill888, @robbibt and @snowman2 for their help in identifying and finding a viable fix for the projection bug and to Geoscience Australia for their continuing support of ODC development and maintenance.

Datacube v1.8.13

Key improvements this release:

1. Fix for the upstream reprojection bug identified by @robbibt
2. Implementation of "archive less mature" functionality in datacube dataset add.

Full list of changes:

• Fix broken Github action workflows (#1425, #1427, #1433)
• Setup Dependabot, and Dependabot-generated updates (#1416, #1420, #1423, #1428, #1436, #1447)
• Documentation fixes (#1417, #1418, #1430)
• datacube dataset cli commands print error message if missing argument (#1437)
• Add pre-commit hook to verify license headers (#1438)
• Support open-ended date ranges in datacube dataset search, dc.load, and dc.find_datasets (#1439, #1443)
• Pass Y and Y Scale factors through to rasterio.warp.reproject, to eliminate projection bug affecting
  non-square Areas Of Interest (See Issue #1448) (#1450)
• Add archive_less_mature option to datacube dataset add and datacube dataset update (#1451)
• Allow for +-1ms leniency in finding other maturity versions of a dataset (#1452)
• Update whats_new.rst for release (#1453)

With contributions from @SpacemanPaul, @Ariana-B, @omad, and @MartinPontius

Special thanks to @Kirill888 for his assistance with resolving #1448 and thanks to all supporting organisations, especially Geoscience Australia.

Datacube v1.8.12

The previous two releases had a serious bug in the new Transaction API that resulted in parallel writes to the database being highly likely to fail. This release is mostly to provide a fix for this issue.

Full list of changes:

• Rename Geometry type attribute to geom_type, to align with Shapely 2.0 (#1402)
• Remove some deprecated SQLAlchemy usages (#1403, #1407)
• Fix RTD docs build (#1399)
• Minor Documentation fixes (#1409, #1413)
• Bug-fix and code cleanup in virtual products (#1410)
• Reduce transaction isolation level to improve database write concurrency (#1414)
• Update dependency versions and whats_new.rst for release (#1415)

Includes contributions from @SpacemanPaul, @emmaai, @Ariana-B, @omad, and first-time contributor @charalamm.

Special thanks to @alexgleith for reporting the Transaction API bug and working with us to test the fix.

Datacube v1.8.11

Administrative release to address two technical issues in release 1.8.10:

1. 1.8.10 attempted to improve compatibility with recent versions of xarray, however further work was needed to fully achieve that compatibility.
2. 1.8.10 was released with dependency discrepancies between setup.py (pip), conda-environment.yml (conda) and constrains.in (Docker).

Full list of changes:

• Simplify Github actions (#1393)
• Update conda create environment README (#1394)
• Update conda environment file and add notes to release process to ensure pip and conda
  dependencies are in sync and up-to-date. (#1395)
• Update docker constraints (#1396)
• Compatible with the changes w.r.t. MultiIndex and coord/dims introduced since xarray>2022.3.0 (#1397)
• Final sync of conda/pip dependencies and update release notes. (#1398)

Includes contributions from @SpacemanPaul, @pindge, @Ariana-B, @emmaai and @omad.

Datacube v1.8.10

Bug fixes, enhancements, and updates to work with newer versions of upstream libraries.

Notes

1. The new APIs for bulk-reads, bulk-writes and index cloning should be considered unstable and may change in subsequent releases.
2. Recent refactoring in the XArray library has lead to changes in behaviour that affect some ODC operations and are unlikely to be addressed by the XArray team. This release includes changes in the way the ODC works with XArray to circumvent these issues. If you experience Xarray issues with this ODC release, please raise an issue on Github and we will try to address them before the next release.

Full list of changes:

• Add grid_spec to list_products (#1357)
• Add database relationship diagram to doc (#1350)
• Maintain search field index tables, and use them for dataset queries (#1360)
• Change Github lint action to use conda and remove flake8 from action (#1361)
• Fix database relationship diagram instruction for docker (#1362)
• Document group_by for dataset.load (#1364)
• Add search_by_metadata facility for products (#1366)
• Postgis driver cleanup - remove faux support for lineage (#1368)
• Add support for nested database transactions (#1369)
• Fix Github doc lint action (#1370)
• Tighten EO3 enforcement in postgis driver, refactor tests, and rename Dataset.type to Dataset.product
  (with type alias for compatibility) (#1372)
• Fix deprecation message due to distutils Version classes (#1375)
• Postgresql drivers cleanup - consolidate split_uri into utils and removed unused constants (#1378)
• Postgresql drivers cleanup - Handle NaNs in search fields and allow caching in sanitise_extent (#1379)
• Fix example product yaml documentation (#1384)
• Bulk read/write API methods and fast whole-index cloning. Cloning does NOT include lineage information yet,
  and new API methods may be subject to change. (#1381)
• Documentation update. (#1385)
• Clean up datetime functions (#1387)
• Dependency updates (#1388, #1391)
• Upgrades for compatibility with newer versions of Shapely and Xarray. (#1389)
• Finalise release notes for 1.8.10 release (#1392)

Includes contributions from: @pindge, @omad, @Ariana-B, @sanjeev0210 and @SpacemanPaul

Datacube v1.8.9

Most notable changes:

1. patch_url argument to dc.load() and dc.load_data() (introduced in v1.8.8) is now also supported for Dask loading.
2. Fixed a day-zero bug affecting search over range-type search fields where the target and search field value is of zero-width.
3. Performance improvements to CRS geometry class.
4. Numerous improvements to documentation and github actions.

Full list of changes:

• Performance improvements to CRS geometry class (#1322)
• Extend patch_url argument to dc.load() and dc.load_data() to Dask loading. (#1323)
• Add sphinx.ext.autoselectionlabel extension to readthedoc conf to support :ref: command (#1325)
• Add pyspellcheck for .rst documentation files and fix typos (#1327)
• Add rst documentation lint github action and apply best practices (#1328)
• Follow PEP561_ to make type hints available to other packages (#1331)
• Updated GitHub actions config to remove deprecated set-output (#1333)
• Add what's new page link to menu and general doc fixes (#1335)
• Add search_fields to required for metadata type schema and update doc (#1339)
• Fix typo and update metadata documentation (#1340)
• Add readthedoc preview github action (#1344)
• Update nodata in readthedoc for products page (#1347)
• Add eo-datasets to extensions & related software doc page (#1349)
• Fix bug affecting searches against range types of zero width (#1352)
• Add 1.8.9 release date and missing PR to whats_news.rst (#1353)

Includes contributions from @SpacemanPaul, @omad, @pindge, @snowman2.

With thanks and appreciation to all contributors, users and supporting organisations, expecially Geoscience Australia.

Datacube v1.8.8

Ofiicial release (Same as 1.8.8-rc1)

Most notable new features are:

1. the new database transaction API, as discussed in ODC-EP07 Database Transaction API. Simple example:

with dc.index.transaction() as trans:
   # Archive old datasets and add new ones in single transaction
   dc.index.datasets.archive([old_ds1.id, old_ds2.id])
   dc.index.datasets.add(ds1)
   dc.index.datasets.add(ds2)

   # If execution gets to here, the transaction is committed.
   # If an exception was raised by any of the above methods, the transaction is rolled back.

2. Add patch_url argument to dc.load and dc.load_data allowing signing of URIs as required by some commercial data providers (e.g. Microsoft Planetary Computer).

Also includes an update of the main test docker build to Ubuntu 22.04 and Python 3.10, and significant progress on the new postgis index driver.

Please note that the postgis index driver is still flagged as "experimental" and is missing several key features.

The release is almost identical to 1.8.8rc1. Full list of changes since 1.8.7:

• Migrate main test docker build to Ubuntu 22.04 and Python 3.10. (#1283)
• Dynamically create tables to serve as spatial indexes in postgis driver. (#1312)
• Populate spatial index tables, automatically and manually. (#1314)
• Perform spatial queries against spatial index tables in postgis driver. (#1316)
• EO3 data fixtures and tests. Fix SQLAlchemy bugs in postgis driver. (#1309)
• Dependency updates. (#1308, #1313)
• Remove several features that had been deprecated in previous releases. (#1275)
• Fix broken paths in api docs. (#1277)
• Fix readthedocs build. (#1269)
• Add support for Jupyter Notebooks pages in documentation (#1279)
• Add doc change comparison for tuple and list types with identical values (#1281)
• Add flake8 to Github action workflow and correct code base per flake8 rules (#1285)
• Add dataset id check to dataset doc resolve to prevent uuid returning error when id used in None (#1287)
• Add how to run targeted single test case in docker guide to README (#1288)
• Add help message for all dataset, product and metadata subcommands when required arg is not passed in (#1292)
• Add error code 1 to all incomplete dataset, product and metadata subcommands (#1293)
• Add exit_on_empty_file message to product and dataset subcommands instead of returning no output when file is empty (#1294)
• Add flags to index drivers advertising what format datasets they support (eo/eo3/non-geo (e.g. telemetry only))
  and validate in the high-level API. General refactor and cleanup of eo3.py and hl.py. (#1296)
• Replace references to 'agdc' and 'dataset_type' in postgis driver with 'odc' and 'product'. (#1298)
• Add warning message for product and metadata add when product and metadata is already in the database. (#1299)
• Ensure SimpleDocNav.id is of type UUID, to improve lineage resolution (#1304)
• Replace SQLAlchemy schema and query definitions in experimental postgis driver with newer "declarative" style ORM.
  Portions of API dealing with lineage handling, locations, and dynamic indexes are currently broken in the postgis
  driver. As per the warning message, the postgis driver is currently flagged as "experimental" and is not considered
  stable. (#1305)
• Implement patch_url argument to dc.load() and dc.load_data() to provide a way to sign dataset URIs, as
  is required to access some commercial archives (e.g. Microsoft Planetary Computer). API is based on the odc-stac
  implementation. Only works for direct loading. More work required for deferred (i.e. Dask) loading. (#1317)
• Implement public-facing index-driver-independent API for managing database transactions, as per Enhancement Proposal
  EP07 (#1318)
• Update Conda environment to match dependencies in setup.py (#1319)
• Final updates to whats_new.rst for release (#1320)

Includes contributions from @SpacemanPaul @tijmenr @pindge and @omad

Thanks to the ODC Steering Council and Geoscience Australia for their ongoing support of ODC development.

1.8.8rc1

RC release to facilitate development in downstream packages using the new transaction API.

Most notable new feature is the new database transaction API, as discussed in ODC-EP07 Database Transaction API. API Example:

with dc.index.transaction() as trans:
   # Archive old datasets and add new ones in single transaction
   dc.index.datasets.archive([old_ds1.id, old_ds2.id])
   dc.index.datasets.add(ds1)
   dc.index.datasets.add(ds2)

   # If execution gets to here, the transaction is committed.
   # If an exception was raised by any of the above methods, the transaction is rolled back.

Also includes an update of the main test docker build to Ubuntu 22.04 and Python 3.10, and significant progress on the new postgis index driver.

Please note that the postgis index driver is still flagged as "experimental" and is missing several key features.

Full list of changes since 1.8.7:

• Migrate main test docker build to Ubuntu 22.04 and Python 3.10. (#1283)
• Dynamically create tables to serve as spatial indexes in postgis driver. (#1312)
• Populate spatial index tables, automatically and manually. (#1314)
• Perform spatial queries against spatial index tables in postgis driver. (#1316)
• EO3 data fixtures and tests. Fix SQLAlchemy bugs in postgis driver. (#1309)
• Dependency updates. (#1308, #1313)
• Remove several features that had been deprecated in previous releases. (#1275)
• Fix broken paths in api docs. (#1277)
• Fix readthedocs build. (#1269)
• Add doc change comparison for tuple and list types with identical values (#1281)
• Add flake8 to Github action workflow and correct code base per flake8 rules (#1285)
• Add dataset id check to dataset doc resolve to prevent uuid returning error when id used in None (#1287)
• Add how to run targeted single test case in docker guide to README (#1288)
• Add help message for all dataset, product and metadata subcommands when required arg is not passed in (#1292)
• Add error code 1 to all incomplete dataset, product and metadata subcommands (#1293)
• Add exit_on_empty_file message to product and dataset subcommands instead of returning no output when file is empty (#1294)
• Add flags to index drivers advertising what format datasets they support (eo/eo3/non-geo (e.g. telemetry only))
  and validate in the high-level API. General refactor and cleanup of eo3.py and hl.py. (#1296)
• Replace references to 'agdc' and 'dataset_type' in postgis driver with 'odc' and 'product'. (#1298)
• Add warning message for product and metadata add when product and metadata is already in the database. (#1299)
• Ensure SimpleDocNav.id is of type UUID, to improve lineage resolution (#1304)
• Replace SQLAlchemy schema and query definitions in experimental postgis driver with newer "declarative" style ORM.
  Portions of API dealing with lineage handling, locations, and dynamic indexes are currently broken in the postgis
  driver. As per the warning message, the postgis driver is currently flagged as "experimental" and is not considered
  stable. (#1305)
• Implement patch_url argument to dc.load() and dc.load_data() to provide a way to sign dataset URIs, as
  is required to access some commercial archives (e.g. Microsoft Planetary Computer). API is based on the odc-stac
  implementation. Only works for direct loading. More work required for deferred (i.e. Dask) loading. (#1317)
• Implement public-facing index-driver-independent API for managing database transactions, as per Enhancement Proposal
  EP07 (#1318)
• Update Conda environment to match dependencies in setup.py (#1319)

1.8.7

• Cleanup mypy typechecking compliance. (#1266)
• When dataset add operations fail due to lineage issues, the produced error message now clearly indicates that
  the problem was due to lineage issues. (#1260)
• Added support for group-by financial years to virtual products. (#1257, #1261)
• Remove reference to rasterio.path. (#1255)
• Cleaner separation of (experimental) postgis and (stable) postgres drivers, and suppress SQLAlchemy cache warnings. (#1254)
• Prevent Shapely deprecation warning. (#1253)
• Fix DATACUBE_DB_URL parsing to understand syntax like: postgresql:///datacube?host=/var/run/postgresql (#1256)
• Clearer error message when local metadata file does not exist. (#1252)
• Address upstream security alerts and update upstream library versions. (#1250)
• Clone postgres index driver as postgis, and flag as experimental. (#1248)
• Implement a local non-persistent in-memory index driver, with maximal backwards-compatibility
  with default postgres index driver. Doesn't work with CLI interface, as every invocation
  will receive a new, empty index, but useful for testing and small scale proof-of-concept
  work. (#1247)
• Performance and correctness fixes backported from odc-geo. (#1242)
• Deprecate use of the celery executor. Update numpy pin in rtd-requirements.txt to suppress
  Dependabot warnings. (#1239)
• Implement a minimal "null" index driver that provides an always-empty index. Mainly intended
  to validate the recent abstraction work around the index driver layer, but may be useful
  for some testing scenarios, and ODC use cases that do not require an index. (#1236)
• Regularise some minor API inconsistencies and restore redis-server to Docker image. (#1234)
• Move (default) postgres driver-specific files from datacube.index to datacube.index.postgres.
  datacube.index.Index is now an alias for the abstract base class index interface definition
  rather than postgres driver-specific implementation of that interface. (#1231)
• Update numpy and netcdf4 version in docker build (#1229)
  rather than postgres driver-specific implementation of that interface. (#1227)
• Migrate test docker image from datacube/geobase to osgeo/gdal. (#1233)
• Separate index driver interface definition from default index driver implementation. (#1226)
• Prefer WKT over EPSG when guessing CRS strings. (#1223, #1262)
• Updates to documentation. (#1208, #1212, #1215, #1218, #1240, #1244)
• Tweak to segmented in geometry to suppress Shapely warning. (#1207)
• Fix to ensure skip_broken_datasets is correctly propagated in virtual products (#1259)
• Deprecate Rename, Select and ToFloat built-in transforms in virtual products (#1263)

Includes contributions from @whatnick, @alexgleith, @maawoo, @jeremyh, @iamtekson, @alfredoahds, @SpacemanPaul, @Kirill888, @robbitbt, @tebadi, @uchchwhash, and @mpaget.

Acknowledgements to the Open Datacube Steering Council and all supporting organisations, including Geoscience Australia, Digital Earth Africa, CSIRO, Frontier SI and Aerometrex.