Adding missing fields to FlyteTask remote entity #3093

UmerAhmad · 2025-01-25T08:46:42Z

Tracking issue

Why are the changes needed?

Missing fields from FlyteTask entity from remote, for example, we need k8s_pod for template data analysis, but it only includes container as of now.

What changes were proposed in this pull request?

Adding the following fields to FlyteTask init, and promotion from model (also config):

security_context
config
k8s_pod
sql
extended_resources

How was this patch tested?

unit tests

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

Summary by Bito

Enhanced FlyteTask remote entity with security_context, k8s_pod, sql, and extended_resources support, including default values and validation. Improved type system with strict type hint matching and added webhook task functionality with rate limiting for worker queues. Implemented shared memory support and enhanced caching capabilities across components. Added comprehensive test coverage and validation improvements for Ray tasks, remote execution, and type handling.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

welcome · 2025-01-25T08:46:44Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

flyte-bot · 2025-01-25T08:46:56Z

Code Review Agent Run #42cb06

Actionable Suggestions - 1

flytekit/remote/entities.py - 1
- Consider adding parameter validation checks · Line 52-56

Review Details

Files reviewed - 1 · Commit Range: 70784e4..70784e4
- flytekit/remote/entities.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-01-25T08:49:33Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Enhanced FlyteTask Entity with Additional Fields	- `entities.py` - Added support for security_context, k8s_pod, sql, and extended_resources fields in FlyteTask
Testing - Test Suite Refactoring	- `test_remote_register.py` - Added test cases for sql and k8s_pod template fields - `test_remote.py` - Refactored tests to use pytest fixtures and updated mock client references

flytekit/remote/entities.py

flyte-bot · 2025-01-25T09:57:23Z

Code Review Agent Run #c16628

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: ef01024..ef01024
- flytekit/remote/entities.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-01-26T01:17:23Z

Code Review Agent Run #4cb82d

Actionable Suggestions - 0

Additional Suggestions - 1

tests/flytekit/unit/remote/test_remote.py - 1
- Consider combining similar None assignments · Line 832-833

Review Details

Files reviewed - 1 · Commit Range: ef01024..8f696e6
- tests/flytekit/unit/remote/test_remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Umer Ahmad <[email protected]> Signed-off-by: Umer Ahmad <[email protected]>

…ntity

Signed-off-by: Umer Ahmad <[email protected]>

tests/flytekit/unit/remote/test_remote.py

flyte-bot · 2025-02-08T00:31:52Z

Code Review Agent Run #17dd82

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: b42edfa..950d4c5
- flytekit/remote/entities.py
- tests/flytekit/unit/remote/test_remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

eapolinario

the spark unit test failure is real. Can you take a look?

…ntity

Signed-off-by: Umer Ahmad <[email protected]>

flyte-bot · 2025-02-21T11:31:09Z

Code Review Agent Run #b0f738

Actionable Suggestions - 0

Additional Suggestions - 10

flytekit/extras/webhook/agent.py - 1
- Consider expanding HTTP success criteria · Line 87-92
flytekit/image_spec/image_spec.py - 1
- Consider adding pip secret mount validation · Line 89-89
tests/flytekit/integration/remote/workflows/basic/dataclass_wf.py - 1
- Consider handling empty strings in list · Line 15-17
tests/flytekit/unit/cli/pyflyte/test_run.py - 1
- Consider using pathlib for file operations · Line 324-332
tests/flytekit/integration/remote/test_remote.py - 3
- Consider more descriptive variable naming · Line 561-570
- Consider standardizing remote wait timeout values · Line 1042-1051
- Consider extracting timeout value to constant · Line 259-259
tests/flytekit/unit/types/structured_dataset/test_structured_dataset.py - 1
- Consider adding more test data types · Line 717-753
flytekit/models/execution.py - 1
- Consider adding interruptible parameter validation · Line 219-219
tests/flytekit/unit/core/test_type_engine.py - 1
- Consider expanding dictionary type test coverage · Line 2884-2886

Review Details

Files reviewed - 78 · Commit Range: 950d4c5..93fdb0d
- dev-requirements.in
- flytekit/__init__.py
- flytekit/clients/grpc_utils/auth_interceptor.py
- flytekit/clis/sdk_in_container/init.py
- flytekit/clis/sdk_in_container/run.py
- flytekit/clis/sdk_in_container/serve.py
- flytekit/configuration/plugin.py
- flytekit/core/cache.py
- flytekit/core/constants.py
- flytekit/core/node.py
- flytekit/core/promise.py
- flytekit/core/python_auto_container.py
- flytekit/core/python_function_task.py
- flytekit/core/resources.py
- flytekit/core/task.py
- flytekit/core/type_engine.py
- flytekit/core/type_match_checking.py
- flytekit/core/worker_queue.py
- flytekit/core/workflow.py
- flytekit/extend/backend/base_agent.py
- flytekit/extras/webhook/__init__.py
- flytekit/extras/webhook/agent.py
- flytekit/extras/webhook/constants.py
- flytekit/extras/webhook/task.py
- flytekit/image_spec/default_builder.py
- flytekit/image_spec/image_spec.py
- flytekit/interaction/click_types.py
- flytekit/interaction/string_literals.py
- flytekit/loggers.py
- flytekit/models/core/workflow.py
- flytekit/models/execution.py
- flytekit/models/task.py
- flytekit/remote/data.py
- flytekit/remote/remote.py
- flytekit/utils/dict_formatter.py
- flytekit/utils/rate_limiter.py
- plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_agent.py
- plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_mixin.py
- plugins/flytekit-aws-sagemaker/setup.py
- plugins/flytekit-aws-sagemaker/tests/test_boto3_agent.py
- plugins/flytekit-greatexpectations/tests/test_schema.py
- plugins/flytekit-onnx-pytorch/dev-requirements.txt
- plugins/flytekit-openai/flytekitplugins/openai/batch/agent.py
- plugins/flytekit-openai/tests/openai_batch/test_agent.py
- plugins/flytekit-pandera/flytekitplugins/pandera/pandas_transformer.py
- plugins/flytekit-pandera/setup.py
- plugins/flytekit-ray/flytekitplugins/ray/task.py
- plugins/flytekit-ray/tests/test_ray.py
- plugins/flytekit-spark/tests/test_remote_register.py
- pydoclint-errors-baseline.txt
- pyproject.toml
- tests/flytekit/integration/remote/test_remote.py
- tests/flytekit/integration/remote/workflows/basic/dataclass_wf.py
- tests/flytekit/integration/remote/workflows/basic/deep_child_workflow.py
- tests/flytekit/unit/bin/test_python_entrypoint.py
- tests/flytekit/unit/cli/pyflyte/test_run.py
- tests/flytekit/unit/cli/pyflyte/test_run_lps.py
- tests/flytekit/unit/core/image_spec/test_default_builder.py
- tests/flytekit/unit/core/image_spec/test_image_spec.py
- tests/flytekit/unit/core/test_annotated_bindings.py
- tests/flytekit/unit/core/test_array_node_map_task.py
- tests/flytekit/unit/core/test_cache.py
- tests/flytekit/unit/core/test_generice_idl_type_engine.py
- tests/flytekit/unit/core/test_node_creation.py
- tests/flytekit/unit/core/test_resources.py
- tests/flytekit/unit/core/test_type_engine.py
- tests/flytekit/unit/core/test_type_match_checking.py
- tests/flytekit/unit/core/test_worker_queue.py
- tests/flytekit/unit/extras/webhook/test_agent.py
- tests/flytekit/unit/extras/webhook/test_end_to_end.py
- tests/flytekit/unit/extras/webhook/test_task.py
- tests/flytekit/unit/interaction/test_click_types.py
- tests/flytekit/unit/interaction/test_string_literals.py
- tests/flytekit/unit/models/core/test_workflow.py
- tests/flytekit/unit/models/test_execution.py
- tests/flytekit/unit/models/test_tasks.py
- tests/flytekit/unit/types/structured_dataset/test_structured_dataset.py
- tests/flytekit/unit/utils/test_rate_limiter.py
Files skipped - 4
- .github/workflows/build_image.yml - Reason: Filter setting
- .github/workflows/pythonbuild.yml - Reason: Filter setting
- .github/workflows/pythonpublish.yml - Reason: Filter setting
- .gitignore - Reason: Filter setting
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

UmerAhmad · 2025-02-21T18:39:49Z

Patched! - thanks for pointing it out, saw that plugins had a separate suite locally. And, it looks like the sagemaker test is globally failing.

eapolinario · 2025-02-24T14:19:46Z

Patched! - thanks for pointing it out, saw that plugins had a separate suite locally. And, it looks like the sagemaker test is globally failing.

Correct. The sagemaker failures are not related to your PR.

welcome · 2025-02-26T21:21:05Z

Congrats on merging your first pull request! 🎉

UmerAhmad requested review from wild-endeavor, kumare3, eapolinario, pingsutw, cosmicBboy and samhita-alla as code owners January 25, 2025 08:46

UmerAhmad requested review from thomasjpfan and Future-Outlier as code owners January 25, 2025 08:46

UmerAhmad force-pushed the add-missing-fields-to-flytetask-entity branch from 70784e4 to ef01024 Compare January 25, 2025 08:48

flyte-bot reviewed Jan 25, 2025

View reviewed changes

flytekit/remote/entities.py Show resolved Hide resolved

UmerAhmad changed the title ~~Adding missing fields to flytetask remote entity~~ Adding missing fields to FlyteTask remote entity Jan 25, 2025

UmerAhmad added 2 commits February 7, 2025 16:09

adding missing fields to flytetask remote entity

b42edfa

Signed-off-by: Umer Ahmad <[email protected]> Signed-off-by: Umer Ahmad <[email protected]>

Patch fetch task remote unit test

9a58c5a

Signed-off-by: Umer Ahmad <[email protected]> Signed-off-by: Umer Ahmad <[email protected]>

UmerAhmad force-pushed the add-missing-fields-to-flytetask-entity branch 2 times, most recently from d41336a to 9a58c5a Compare February 8, 2025 00:13

UmerAhmad and others added 2 commits February 7, 2025 19:15

Merge branch 'flyteorg:master' into add-missing-fields-to-flytetask-e…

9dffb0a

…ntity

Change patch to be global using fixture

950d4c5

Signed-off-by: Umer Ahmad <[email protected]>

UmerAhmad commented Feb 8, 2025

View reviewed changes

tests/flytekit/unit/remote/test_remote.py Show resolved Hide resolved

UmerAhmad mentioned this pull request Feb 8, 2025

[Core feature] Pyflyte - Workflow Version Comparisons flyteorg/flyte#6196

Open

2 tasks

eapolinario approved these changes Feb 19, 2025

View reviewed changes

eapolinario enabled auto-merge (squash) February 19, 2025 18:50

eapolinario requested changes Feb 19, 2025

View reviewed changes

UmerAhmad and others added 2 commits February 21, 2025 02:20

Merge branch 'flyteorg:master' into add-missing-fields-to-flytetask-e…

6a9f585

…ntity

patch spark plugin remote register

93fdb0d

Signed-off-by: Umer Ahmad <[email protected]>

auto-merge was automatically disabled February 21, 2025 10:27
Head branch was pushed to by a user without write access

UmerAhmad requested a review from eapolinario February 21, 2025 18:17

eapolinario approved these changes Feb 26, 2025

View reviewed changes

eapolinario merged commit 2e12f43 into flyteorg:master Feb 26, 2025
109 of 110 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding missing fields to FlyteTask remote entity #3093

Adding missing fields to FlyteTask remote entity #3093

UmerAhmad commented Jan 25, 2025 •

edited by flyte-bot

Loading

welcome bot commented Jan 25, 2025

flyte-bot commented Jan 25, 2025 •

edited

Loading

Code Review Agent Run #42cb06

flyte-bot commented Jan 25, 2025 •

edited

Loading

Changelist by Bito

flyte-bot commented Jan 25, 2025 •

edited

Loading

Code Review Agent Run #c16628

flyte-bot commented Jan 26, 2025 •

edited

Loading

Code Review Agent Run #4cb82d

flyte-bot commented Feb 8, 2025 •

edited

Loading

Code Review Agent Run #17dd82

eapolinario left a comment

flyte-bot commented Feb 21, 2025 •

edited

Loading

Code Review Agent Run #b0f738

UmerAhmad commented Feb 21, 2025

eapolinario commented Feb 24, 2025

welcome bot commented Feb 26, 2025

Adding missing fields to FlyteTask remote entity #3093

Adding missing fields to FlyteTask remote entity #3093

Conversation

UmerAhmad commented Jan 25, 2025 • edited by flyte-bot Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Check all the applicable boxes

Related PRs

Docs link

Summary by Bito

welcome bot commented Jan 25, 2025

flyte-bot commented Jan 25, 2025 • edited Loading

Code Review Agent Run #42cb06

flyte-bot commented Jan 25, 2025 • edited Loading

Changelist by Bito

flyte-bot commented Jan 25, 2025 • edited Loading

Code Review Agent Run #c16628

flyte-bot commented Jan 26, 2025 • edited Loading

Code Review Agent Run #4cb82d

flyte-bot commented Feb 8, 2025 • edited Loading

Code Review Agent Run #17dd82

eapolinario left a comment

Choose a reason for hiding this comment

flyte-bot commented Feb 21, 2025 • edited Loading

Code Review Agent Run #b0f738

UmerAhmad commented Feb 21, 2025

eapolinario commented Feb 24, 2025

welcome bot commented Feb 26, 2025

UmerAhmad commented Jan 25, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Jan 25, 2025 •

edited

Loading

flyte-bot commented Jan 25, 2025 •

edited

Loading

flyte-bot commented Jan 25, 2025 •

edited

Loading

flyte-bot commented Jan 26, 2025 •

edited

Loading

flyte-bot commented Feb 8, 2025 •

edited

Loading

flyte-bot commented Feb 21, 2025 •

edited

Loading