Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove product bundles json from repo, and fetch it dynamically, add helper function to get valid item types and bundles in cli #1094

Merged
merged 20 commits into from
Jan 31, 2025

Conversation

charcey
Copy link
Contributor

@charcey charcey commented Jan 16, 2025

Description
Currently we have a copy of the orders product bundle spec in this repository. When there is an update to the bundles spec someone ends up having to update the spec here. This can cause headaches for users who might expect to be able to order a new item type but can't until the spec is updated.

Recently, a new endpoint was added to the Orders API that serves the spec. This presents a great opportunity to dynamically grab the spec instead of managing a version of it statically.

(editors note: removed old text here, and just laying out the implemented path after internal discussions)

The bundles spec information is lazily loaded into an envvar PRODUCT_BUNDLES. The first time the variable is accessed, the spec is fetched from the previously mentioned public endpoint, and every subsequent access (within the same runtime) will use the previously loaded information. It tries twice to get the spec and since we expect the spec endpoint to be highly available there is no work around if the spec cannot be accessed.

Side effects:
Previously the spec was loaded in (from a static file) on any data, orders, or subscriptions cli help command (e.g. planet data --help). This was to list the valid item types which is pulled from the bundles spec (there are other issues around this, but not solving that here). In addition, for orders help commands the bundle names were loaded in. Because having to reach out to the internet to run --help is not a good practice, this information is now available through new commands : planet data show-item-types, planet orders show-item-types, planet orders show-bundles, planet subscriptions show-item-types. These changes plus the lazy loading ensures we are only reaching out to the internet for the spec when absolutely necessary. 🥳

Examples:

charcey-C02DP4WKML85:planet-client-python charcey$ planet orders --help
Usage: planet orders [OPTIONS] COMMAND [ARGS]...

  Commands for interacting with the Orders API

Options:
  -u, --base-url TEXT  Assign custom base Orders API URL.
  --help               Show this message and exit.

Commands:
  cancel           Cancel order by order ID.
  create           Create an order.
  download         Download order by order ID.
  get              Get order
  list             List orders
  request          Generate an order request.
  show-bundles     Show valid bundle names for ordering.
  show-item-types  Show valid item types for ordering.
  wait             Wait until order reaches desired state.

charcey-C02DP4WKML85:planet-client-python charcey$ planet orders show-item-types
Valid item types:
- SkySatScene
- TanagerMethane
- REOrthoTile
- PSScene
- TanagerScene
- REScene
- PelicanScene
- SkySatCollect

charcey-C02DP4WKML85:planet-client-python charcey$ planet orders show-bundles
Valid bundles:
- analytic
- radiance_hdf5
- analytic_udm2
...
- methane
- integrated_methane_enhancement

Related Issue(s):

Closes #734

Proposed Changes:

For inclusion in changelog (if applicable):

  1. Orders product bundles spec is fetched dynamically
  2. planet data show-item-types to show valid item types for data requests
  3. planet orders show-bundles to show valid bundle names and planet orders show-item-types to show valid item types
  4. planet subscriptions show-item-types to show valid bundle names for scene subscriptions

Not intended for changelog:

  1. Updated lots of tests. Removing all PSOrthoTile references as that has been deprecated, using PSScene instead. New tests for new cli commands. Use mocked spec for all tests. Fixed various typos.
  2. Refactored access patterns of spec in specs.py to reduce traversing json so much. Preloading bundle names and item types into new keys in the dictionary.

Diff of User Interface

Old behavior: N/A

New behavior: N/A

User behavior is the exact same.

However, if the spec cannot be reached, an error will surface to the user.

Example:

>>> from planet.order_request import product
>>> product(
...                     item_ids=[
...                         "20220304_093300_37_2430",
...                         "20220305_093440_25_2429",
...                     ],
...                     product_bundle="analytic_udm2",
...                     item_type="PSScene",)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/charcey/workspace/planet-client-python/planet/order_request.py", line 125, in product
    item_type = specs.validate_item_type(item_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charcey/workspace/planet-client-python/planet/specs.py", line 112, in validate_item_type
    PRODUCT_BUNDLES["item_types"],
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/charcey/workspace/planet-client-python/planet/specs.py", line 62, in __getitem__
    raise FetchBundlesSpecError(
planet.specs.FetchBundlesSpecError: Unable to fetch product bundles spec from API to perform client side validation on item types and product bundles. Please retry!

PR Checklist:

  • This PR is as small and focused as possible
  • If this PR includes proposed changes for inclusion in the changelog, the title of this PR summarizes those changes and is ready for inclusion in the Changelog.
  • I have updated docstrings for function changes and docs in the 'docs' folder for user interface / behavior changes
  • This PR does not break any examples or I have updated them

(Optional) @mentions for Notifications:

@ischneider
Copy link
Member

Here's a way to replace the module import side-effect:

class _LazyLoader:

    def __getitem__(self, key):
        cache = getattr(self, "cache", None)
        if cache is None:
            response = httpx.get("https://api.planet.com/compute/ops/bundles/spec")
            response.raise_for_status()
            cache = response.json()
            setattr(self, "cache", cache)
        return cache[key]

PRODUCT_BUNDLES_SPEC_JSON = _LazyLoader()

I feel like we might want to introduce minimal retry (mostly to avoid a potential network hiccup) and clearer handling of unexpected conditions as this could be a source of confusion ...

planet/specs.py Outdated Show resolved Hide resolved
@charcey charcey changed the title Remove product bundles json from repo, fetch it dynamically remove product bundles json from repo, and fetch it dynamically, add helper function to get valid item types and bundles in cli Jan 27, 2025
planet/cli/data.py Outdated Show resolved Hide resolved
planet/cli/orders.py Outdated Show resolved Hide resolved
planet/cli/orders.py Outdated Show resolved Hide resolved
planet/specs.py Outdated Show resolved Hide resolved
planet/specs.py Outdated Show resolved Hide resolved
@charcey charcey requested a review from ischneider January 30, 2025 15:51
@charcey charcey merged commit dc51fcd into main Jan 31, 2025
9 checks passed
@charcey charcey deleted the charcey/product-bundles branch January 31, 2025 21:25
@asonnenschein asonnenschein mentioned this pull request Feb 6, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update Orders Product Bundles Reference JSON file and link to SSoT JSON file
3 participants