Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOGE-106] Instance aggregate/search limit=None means maximum #2121

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

doctrino
Copy link
Contributor

@doctrino doctrino commented Feb 25, 2025

Description

Fixed

  • The client.data_modeling.instances.aggregate() method now correctly returns maximum, 1000, results when setting
    the limit parameter to None, -1, or math.inf.

Checklist:

  • Tests added/updated.
  • Documentation updated. Documentation is generated from docstrings - these must be updated according to your change.
    If a new method has been added it should be referenced in cognite.rst in order to generate docs based on its docstring.
  • Changelog updated in CHANGELOG.md.
  • Version bumped. If triggering a new release is desired, bump the version number in _version.py and pyproject.toml per semantic versioning.

Copy link

codecov bot commented Feb 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.38%. Comparing base (11cf2b4) to head (75e51af).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2121   +/-   ##
=======================================
  Coverage   90.38%   90.38%           
=======================================
  Files         152      152           
  Lines       23127    23131    +4     
=======================================
+ Hits        20903    20908    +5     
+ Misses       2224     2223    -1     
Files with missing lines Coverage Δ
cognite/client/_api/data_modeling/instances.py 88.29% <100.00%> (+0.11%) ⬆️
cognite/client/_version.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@doctrino doctrino marked this pull request as ready for review February 25, 2025 18:17
@doctrino doctrino requested review from a team as code owners February 25, 2025 18:17
erlendvollset
erlendvollset previously approved these changes Feb 25, 2025
@doctrino doctrino requested review from a team and finnag and removed request for a team February 25, 2025 18:19
@doctrino doctrino enabled auto-merge (squash) February 25, 2025 18:19
erlendvollset
erlendvollset previously approved these changes Feb 25, 2025
body: dict[str, Any] = {
"view": view.dump(camel_case=True),
"instanceType": instance_type,
"limit": self._AGGREGATE_LIMIT if is_unlimited(limit) or limit is None else limit,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"limit": self._AGGREGATE_LIMIT if is_unlimited(limit) or limit is None else limit,
"limit": self._AGGREGATE_LIMIT if is_unlimited(limit) else limit,

ref its implementation:

def is_unlimited(limit: float | int | None) -> bool:
    return limit in {None, -1, math.inf}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The limit is None is just for mypy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My mypy is happy without it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok then it is me that did not trust mypy

@haakonvt
Copy link
Contributor

Perhaps change instances/search at the same time?

haakonvt
haakonvt previously approved these changes Feb 26, 2025
Copy link
Contributor

@haakonvt haakonvt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

haakonvt
haakonvt previously approved these changes Feb 28, 2025
Copy link

@finnag finnag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous comment was sent prematurely. Trying again:

How do you know the behavior is as you write? There should be tests.
What happens if you specify -2, 0, or 1040 as limit?

Have we publicly documented 3 different ways of getting the max limit (-1, None, inf) ? If we have, we'll have to support that more or less forever. If we haven't, do we want to support it forever?

Do we really support floats as limit (math.inf is a float..)

@doctrino
Copy link
Contributor Author

doctrino commented Mar 3, 2025

Previous comment was sent prematurely. Trying again:

How do you know the behavior is as you write? There should be tests. What happens if you specify -2, 0, or 1040 as limit?

The the API will give a 400 and tell you that it is an invalid limit. The philosophy of the PySDK is that we do as little validation of the input to the API as possible, this is such that we do not have to keep up with the validation of the API side.

Have we publicly documented 3 different ways of getting the max limit (-1, None, inf) ? If we have, we'll have to support that more or less forever. If we haven't, do we want to support it forever?

We support that for all other limit parameters in the SDK, so we have already committed to supporting it. I think we should continue to do so here, so we get consistency.

Do we really support floats as limit (math.inf is a float..)

Yes, again for consistency with all other limit parameters.

@doctrino doctrino requested a review from finnag March 3, 2025 13:38
@doctrino doctrino changed the title [DOGE-106] Change behavior of aggregate [DOGE-106] Instance aggregate/search limit=None means maximum Mar 5, 2025
@@ -164,6 +164,11 @@ def _load(data: dict, cognite_client: CogniteClient | None = None) -> NodeApply
class InstancesAPI(APIClient):
_RESOURCE_PATH = "/models/instances"

def __init__(self, config: ClientConfig, api_version: str | None, cognite_client: CogniteClient) -> None:
super().__init__(config, api_version, cognite_client)
self._AGGREGATE_LIMIT = 1000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these be class variables instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that could be. Consistently in the SDK we have limits as instance
attributes. Suggest that if we move these two class variables, we do it everywhere, which should in that case be a separate PR.

@dbrattli
Copy link
Contributor

dbrattli commented Mar 5, 2025

Even if you don't do validation in the SDK, you still have to test that feature you make here actually works, that it's possible to return maximum by using e.g None.

Copy link

@finnag finnag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR still lacks a test.

There should be a test that that the values None, -1, or math.inf actually set the limit to 1000 instead, and that other values are passed on unmodified.

Copy link
Contributor

@haakonvt haakonvt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Ideally the test would also include some random numbers/objects to ensure they are passed along untouched, but that is probably already covered by existing tests.

@doctrino doctrino requested review from finnag, dbrattli, a team and larshagencognite and removed request for a team March 6, 2025 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants