From af6f25385ba693b0f415a9c54353d9664abbd679 Mon Sep 17 00:00:00 2001 From: Nikki Everett Date: Fri, 23 Feb 2024 11:48:55 -0600 Subject: [PATCH] Updates for agents integrations docs (#1498) * split bq docs into agents and plugins docs Signed-off-by: nikki everett * update docs for plugins with a corresponding agent Signed-off-by: nikki everett * update top-level toctree, redirects, and integrations docs landing page for new agents Signed-off-by: nikki everett * more updates for bq integration Signed-off-by: nikki everett * more updates for databricks integration Signed-off-by: nikki everett * copyedits for databricks plugin page Signed-off-by: nikki everett * lots of cleanup Signed-off-by: nikki everett * add airflow agent to integrations Signed-off-by: nikki everett * update integrations landing page Signed-off-by: nikki everett * more updates for bq and mmcloud agents Signed-off-by: nikki everett * formatting fixes Signed-off-by: nikki everett * update toctrees and airflow agents example Signed-off-by: nikki everett * formatting Signed-off-by: nikki everett * updates for mmcloud agent Signed-off-by: nikki everett * lots of cleanup Signed-off-by: nikki everett * lots of cleanup Signed-off-by: nikki everett * update note text Signed-off-by: nikki everett * add note Signed-off-by: nikki everett * add anchor Signed-off-by: nikki everett * update links Signed-off-by: nikki everett * add README Signed-off-by: nikki everett * add link to agents guide Signed-off-by: nikki everett * add general info on local testing Signed-off-by: nikki everett * update sandbox ephemeral storage to fix failing test Signed-off-by: nikki everett * update correct storage param Signed-off-by: nikki everett * small edit Signed-off-by: nikki everett * try bumping flytekit version Signed-off-by: nikki everett * try removing all but flytekit version requirement Signed-off-by: nikki everett * remove flytekit version Signed-off-by: nikki everett * reset mnist classifier imagespec and requirements so others can debug Signed-off-by: nikki everett * typo Signed-off-by: nikki everett --------- Signed-off-by: nikki everett --- docs/conf.py | 16 +- docs/index.md | 9 +- docs/integrations.md | 46 +- examples/airflow_agent/README.md | 48 + .../airflow_agent}/__init__.py | 0 .../airflow_agent/airflow_agent_example.py | 23 + .../airflow_agent/requirements.in | 2 + .../airflow_agent/requirements.txt | 986 ++++++++++++++++++ .../Dockerfile | 0 examples/bigquery_integration/README.md | 21 + .../bigquery_integration}/__init__.py | 0 .../bigquery_integration}/bigquery.py | 2 +- .../bigquery_integration/bigquery_agent.md | 37 + .../bigquery_integration/bigquery_plugin.md | 32 + .../requirements.in | 0 .../requirements.txt | 0 examples/bigquery_plugin/README.md | 30 - .../Dockerfile | 0 examples/databricks_integration/README.md | 21 + .../databricks_integration}/__init__.py | 0 .../databricks_agent.md | 34 + .../databricks_integration}/databricks_job.py | 0 .../databricks_plugin.md | 49 + .../requirements.in | 0 .../requirements.txt | 0 examples/databricks_plugin/README.md | 51 - .../Dockerfile | 0 examples/mmcloud_integration/README.md | 22 + .../mmcloud_integration}/__init__.py | 0 .../mmcloud_integration/mmcloud_agent.md | 37 + .../mmcloud_integration/mmcloud_example.py} | 1 + .../mmcloud_integration/mmcloud_plugin.md} | 28 +- .../requirements.in | 0 .../requirements.txt | 0 examples/sensor/README.md | 20 +- .../Dockerfile | 0 examples/snowflake_integration/README.md | 22 + .../requirements.in | 0 .../requirements.txt | 0 .../snowflake_integration/__init__.py | 0 .../snowflake_integration}/snowflake.py | 2 +- .../snowflake_integration/snowflake_agent.md | 33 + .../snowflake_plugin.md} | 21 +- 43 files changed, 1457 insertions(+), 136 deletions(-) create mode 100644 examples/airflow_agent/README.md rename examples/{bigquery_plugin/bigquery_plugin => airflow_agent/airflow_agent}/__init__.py (100%) create mode 100644 examples/airflow_agent/airflow_agent/airflow_agent_example.py create mode 100644 examples/airflow_agent/airflow_agent/requirements.in create mode 100644 examples/airflow_agent/airflow_agent/requirements.txt rename examples/{bigquery_plugin => bigquery_integration}/Dockerfile (100%) create mode 100644 examples/bigquery_integration/README.md rename examples/{databricks_plugin/databricks_plugin => bigquery_integration/bigquery_integration}/__init__.py (100%) rename examples/{bigquery_plugin/bigquery_plugin => bigquery_integration/bigquery_integration}/bigquery.py (98%) create mode 100644 examples/bigquery_integration/bigquery_integration/bigquery_agent.md create mode 100644 examples/bigquery_integration/bigquery_integration/bigquery_plugin.md rename examples/{bigquery_plugin => bigquery_integration}/requirements.in (100%) rename examples/{bigquery_plugin => bigquery_integration}/requirements.txt (100%) delete mode 100644 examples/bigquery_plugin/README.md rename examples/{databricks_plugin => databricks_integration}/Dockerfile (100%) create mode 100644 examples/databricks_integration/README.md rename examples/{mmcloud_plugin/mmcloud_plugin => databricks_integration/databricks_integration}/__init__.py (100%) create mode 100644 examples/databricks_integration/databricks_integration/databricks_agent.md rename examples/{databricks_plugin/databricks_plugin => databricks_integration/databricks_integration}/databricks_job.py (100%) create mode 100644 examples/databricks_integration/databricks_integration/databricks_plugin.md rename examples/{databricks_plugin => databricks_integration}/requirements.in (100%) rename examples/{databricks_plugin => databricks_integration}/requirements.txt (100%) delete mode 100644 examples/databricks_plugin/README.md rename examples/{mmcloud_plugin => mmcloud_integration}/Dockerfile (100%) create mode 100644 examples/mmcloud_integration/README.md rename examples/{snowflake_plugin/snowflake_plugin => mmcloud_integration/mmcloud_integration}/__init__.py (100%) create mode 100644 examples/mmcloud_integration/mmcloud_integration/mmcloud_agent.md rename examples/{mmcloud_plugin/mmcloud_plugin/example.py => mmcloud_integration/mmcloud_integration/mmcloud_example.py} (98%) rename examples/{mmcloud_plugin/README.md => mmcloud_integration/mmcloud_integration/mmcloud_plugin.md} (52%) rename examples/{mmcloud_plugin => mmcloud_integration}/requirements.in (100%) rename examples/{mmcloud_plugin => mmcloud_integration}/requirements.txt (100%) rename examples/{snowflake_plugin => snowflake_integration}/Dockerfile (100%) create mode 100644 examples/snowflake_integration/README.md rename examples/{snowflake_plugin => snowflake_integration}/requirements.in (100%) rename examples/{snowflake_plugin => snowflake_integration}/requirements.txt (100%) create mode 100644 examples/snowflake_integration/snowflake_integration/__init__.py rename examples/{snowflake_plugin/snowflake_plugin => snowflake_integration/snowflake_integration}/snowflake.py (99%) create mode 100644 examples/snowflake_integration/snowflake_integration/snowflake_agent.md rename examples/{snowflake_plugin/README.md => snowflake_integration/snowflake_integration/snowflake_plugin.md} (67%) diff --git a/docs/conf.py b/docs/conf.py index e08fc3850..4cbd740b2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -231,8 +231,8 @@ "auto_examples/basics/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/index.html", "auto_examples/basics/task.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/task.html", "auto_examples/basics/shell_task.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/shell_task.html", - "auto_examples/bigquery_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/index.html", - "auto_examples/bigquery_plugin/bigquery.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/bigquery.html", + "auto_examples/bigquery_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_integration/index.html", + "auto_examples/bigquery_plugin/bigquery.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_integration/bigquery.html", "auto_examples/blast/blastx_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/blast/blastx_example.html", "auto_examples/blast/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/blast/index.html", "auto_examples/customizing_dependencies/image_spec.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/customizing_dependencies/image_spec.html", @@ -248,8 +248,8 @@ "auto_examples/data_types_and_io/dataclass.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/dataclass.html", "auto_examples/data_types_and_io/folder.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/folder.html", "auto_examples/data_types_and_io/pickle_type.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/pickle_type.html", - "auto_examples/databricks_plugin/databricks_job.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/databricks_job.html", - "auto_examples/databricks_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/index.html", + "auto_examples/databricks_plugin/databricks_job.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_integration/databricks_job.html", + "auto_examples/databricks_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_integration/index.html", "auto_examples/dbt_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/dbt_plugin/index.html", "auto_examples/dbt_plugin/dbt_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/dbt_plugin/dbt_example.html", "auto_examples/development_lifecycle/inspecting_executions.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/inspecting_executions.html", @@ -317,8 +317,8 @@ "auto_examples/kftensorflow_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/kftensorflow_plugin/index.html", "auto_examples/mlflow_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mlflow_plugin/index.html", "auto_examples/mlflow_plugin/mlflow_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mlflow_plugin/mlflow_example.html", - "auto_examples/mmcloud_plugin/example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/example.html", - "auto_examples/mmcloud_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/index.html", + "auto_examples/mmcloud_plugin/example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/example.html", + "auto_examples/mmcloud_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/index.html", "auto_examples/mnist_classifier/pytorch_single_node_and_gpu.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_and_gpu.html", "auto_examples/mnist_classifier/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/index.html", "auto_examples/mnist_classifier/pytorch_single_node_multi_gpu.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_multi_gpu.html", @@ -358,8 +358,8 @@ "auto_examples/sagemaker_training_plugin/sagemaker_custom_training.htmlhttps://docs.flyte.org/en/latest/": "flytesnacks/examples/sagemaker_training_plugin/sagemaker_custom_training.html", "auto_examples/sensor/file_sensor_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sensor/file_sensor_example.html", "auto_examples/sensor/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sensor/index.html", - "auto_examples/snowflake_plugin/snowflake.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/snowflake.html", - "auto_examples/snowflake_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/index.html", + "auto_examples/snowflake_plugin/snowflake.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_integration/snowflake.html", + "auto_examples/snowflake_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_integration/index.html", "auto_examples/sql_plugin/sql_alchemy.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/sql_alchemy.html", "auto_examples/sql_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/index.html", "auto_examples/sql_plugin/sqlite3_integration.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/sqlite3_integration.html", diff --git a/docs/index.md b/docs/index.md index d2ad70e78..4902d5734 100644 --- a/docs/index.md +++ b/docs/index.md @@ -98,14 +98,15 @@ flyte_lab :hidden: Integrations +auto_examples/airflow_agent/index auto_examples/airflow_plugin/index auto_examples/athena_plugin/index auto_examples/aws_batch_plugin/index auto_examples/sagemaker_pytorch_plugin/index auto_examples/sagemaker_training_plugin/index -auto_examples/bigquery_plugin/index +auto_examples/bigquery_integration/index auto_examples/k8s_dask_plugin/index -auto_examples/databricks_plugin/index +auto_examples/databricks_integration/index auto_examples/dbt_plugin/index auto_examples/dolt_plugin/index auto_examples/duckdb_plugin/index @@ -114,7 +115,7 @@ auto_examples/greatexpectations_plugin/index auto_examples/hive_plugin/index auto_examples/k8s_pod_plugin/index auto_examples/mlflow_plugin/index -auto_examples/mmcloud_plugin/index +auto_examples/mmcloud_integration/index auto_examples/modin_plugin/index auto_examples/kfmpi_plugin/index auto_examples/onnx_plugin/index @@ -123,7 +124,7 @@ auto_examples/pandera_plugin/index auto_examples/kfpytorch_plugin/index auto_examples/ray_plugin/index auto_examples/sensor/index -auto_examples/snowflake_plugin/index +auto_examples/snowflake_integration/index auto_examples/k8s_spark_plugin/index auto_examples/sql_plugin/index auto_examples/kftensorflow_plugin/index diff --git a/docs/integrations.md b/docs/integrations.md index 89016afca..0d7d2b0bf 100644 --- a/docs/integrations.md +++ b/docs/integrations.md @@ -80,11 +80,35 @@ orchestrated by Flyte itself, within its provisioned Kubernetes clusters. - Run Ray jobs on a K8s Cluster. ``` +(flyte_agents)= + +## Flyte agents + +[Flyte agents](https://docs.flyte.org/en/latest/flyte_agents/index.html) are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. Each agent service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task. (For example, the BigQuery agent handles BigQuery tasks.) The agent service then initiates a job with the appropriate service. If you don't see the agent you need below, see "[Developing agents](https://docs.flyte.org/en/latest/flyte_agents/developing_agents.html)" to learn how to develop a new agent. + +```{list-table} +:header-rows: 0 +:widths: 20 30 + +* - {ref}`Airflow agent ` + - Run Airflow jobs in your workflows with the Airflow agent. +* - {ref}`BigQuery agent ` + - Run BigQuery jobs in your workflows with the BigQuery agent. +* - {ref}`Databricks ` + - Run Databricks jobs in your workflows with the Databricks agent. +* - {ref}`Memory Machine Cloud ` + - Execute tasks using the MemVerge Memory Machine Cloud agent. +* - {doc}`Sensor ` + - Run sensor jobs in your workflows with the sensor agent. +* - {ref}`Snowflake ` + - Run Snowflake jobs in your workflows with the Snowflake agent. +``` + (external_service_backend_plugins)= ## External Service Backend Plugins -As the term suggests, external service backend plugins relies on external services like +As the term suggests, external service backend plugins rely on external services like [AWS Sagemaker](https://aws.amazon.com/sagemaker), [Hive](https://docs.qubole.com/en/latest/user-guide/engines/hive/index.html) or [Snowflake](https://www.snowflake.com/) for handling the workload defined in @@ -94,27 +118,25 @@ the Flyte task that use the respective plugin. :header-rows: 0 :widths: 20 30 -* - {doc}`AWS Sagemaker: Model Training ` +* - {doc}`AWS Sagemaker: Model Training plugin ` - Train models with built-in or define your own custom algorithms. -* - {doc}`AWS Sagemaker: Pytorch Training ` +* - {doc}`AWS Sagemaker: Pytorch Training plugin ` - Train Pytorch models using Sagemaker, with support for distributed training. -* - {doc}`AWS Athena ` +* - {doc}`AWS Athena plugin ` - Execute queries using AWS Athena -* - {doc}`AWS Batch ` +* - {doc}`AWS Batch plugin ` - Running tasks and workflows on AWS batch service * - {doc}`Flyte Interactive ` - Execute tasks using Flyte Interactive to debug. -* - {doc}`Hive ` +* - {doc}`Hive plugin ` - Run Hive jobs in your workflows. -* - {doc}`MMCloud ` +* - {ref}`MMCloud plugin ` - Execute tasks using MemVerge Memory Machine Cloud -* - {doc}`Sensor ` - - Run Sensor jobs in your workflows. -* - {doc}`Snowflake ` +* - {ref}`Snowflake ` - Run Snowflake jobs in your workflows. -* - {doc}`Databricks ` +* - {ref}`Databricks ` - Run Databricks jobs in your workflows. -* - {doc}`BigQuery ` +* - {ref}`BigQuery ` - Run BigQuery jobs in your workflows. ``` diff --git a/examples/airflow_agent/README.md b/examples/airflow_agent/README.md new file mode 100644 index 000000000..80264ad56 --- /dev/null +++ b/examples/airflow_agent/README.md @@ -0,0 +1,48 @@ +(airflow_agent)= + +# Airflow agent + +```{note} +The Airflow agent does not support all [Airflow operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html). We have tested many, but if you run into issues, please [file a bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+). +``` + +## Installation + +To install the plugin, run the following command: + +`pip install flytekitplugins-airflow` + +## Example usage + +```{note} + +You don't need an Airflow cluster to run Airflow tasks, since Flytekit will +automatically compile Airflow tasks to Flyte tasks and execute them on the Flyte cluster. + +``` + +For a usage example, see the {doc}`Airflow agent example ` page. + +## Local testing + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. + +``` + +## Flyte deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the Airflow agent in your Flyte deployment, see the {ref}`Airflow agent deployment guide`. + +```{toctree} +:maxdepth: -1 +:hidden: +airflow_agent_example +``` diff --git a/examples/bigquery_plugin/bigquery_plugin/__init__.py b/examples/airflow_agent/airflow_agent/__init__.py similarity index 100% rename from examples/bigquery_plugin/bigquery_plugin/__init__.py rename to examples/airflow_agent/airflow_agent/__init__.py diff --git a/examples/airflow_agent/airflow_agent/airflow_agent_example.py b/examples/airflow_agent/airflow_agent/airflow_agent_example.py new file mode 100644 index 000000000..8d05c14bf --- /dev/null +++ b/examples/airflow_agent/airflow_agent/airflow_agent_example.py @@ -0,0 +1,23 @@ +# %% [markdown] +# (airflow_agent_example)= +# # Airflow agent example +# +# %% + +from airflow.sensors.filesystem import FileSensor +from flytekit import task, workflow + + +@task() +def t1(): + print("flyte") + + +@workflow +def wf(): + sensor = FileSensor(task_id="id", filepath="/tmp/1234") + sensor >> t1() + + +if __name__ == "__main__": + wf() diff --git a/examples/airflow_agent/airflow_agent/requirements.in b/examples/airflow_agent/airflow_agent/requirements.in new file mode 100644 index 000000000..b3913ef81 --- /dev/null +++ b/examples/airflow_agent/airflow_agent/requirements.in @@ -0,0 +1,2 @@ +apache-airflow-providers-apache-beam[google] +apache-airflow[google] diff --git a/examples/airflow_agent/airflow_agent/requirements.txt b/examples/airflow_agent/airflow_agent/requirements.txt new file mode 100644 index 000000000..114279f52 --- /dev/null +++ b/examples/airflow_agent/airflow_agent/requirements.txt @@ -0,0 +1,986 @@ +# +# This file is autogenerated by pip-compile with Python 3.10 +# by the following command: +# +# pip-compile dev-requirements.in +# +aiofiles==23.2.1 + # via gcloud-aio-storage +aiohttp==3.9.2 + # via + # apache-airflow-providers-http + # gcloud-aio-auth + # gcsfs +aiosignal==1.3.1 + # via aiohttp +alembic==1.12.1 + # via + # apache-airflow + # sqlalchemy-spanner +annotated-types==0.6.0 + # via pydantic +anyio==4.0.0 + # via httpx +apache-airflow[google]==2.8.0 + # via + # -r dev-requirements.in + # apache-airflow-providers-apache-beam + # apache-airflow-providers-common-sql + # apache-airflow-providers-ftp + # apache-airflow-providers-google + # apache-airflow-providers-http + # apache-airflow-providers-imap + # apache-airflow-providers-sqlite +apache-airflow-providers-apache-beam[google]==5.3.0 + # via -r dev-requirements.in +apache-airflow-providers-common-sql==1.8.0 + # via + # apache-airflow + # apache-airflow-providers-google + # apache-airflow-providers-sqlite +apache-airflow-providers-ftp==3.6.0 + # via apache-airflow +apache-airflow-providers-google==10.11.0 + # via + # apache-airflow + # apache-airflow-providers-apache-beam +apache-airflow-providers-http==4.6.0 + # via apache-airflow +apache-airflow-providers-imap==3.4.0 + # via apache-airflow +apache-airflow-providers-sqlite==3.5.0 + # via apache-airflow +apache-beam[gcp]==2.51.0 + # via apache-airflow-providers-apache-beam +apispec[yaml]==6.3.0 + # via + # apispec + # flask-appbuilder +argcomplete==3.1.4 + # via apache-airflow +asgiref==3.7.2 + # via + # apache-airflow + # apache-airflow-providers-google + # apache-airflow-providers-http +async-timeout==4.0.3 + # via aiohttp +attrs==23.1.0 + # via + # aiohttp + # apache-airflow + # cattrs + # jsonschema + # looker-sdk + # referencing +babel==2.13.1 + # via flask-babel +backoff==2.2.1 + # via + # gcloud-aio-auth + # opentelemetry-exporter-otlp-proto-common + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +blinker==1.7.0 + # via apache-airflow +cachelib==0.9.0 + # via + # flask-caching + # flask-session +cachetools==5.3.2 + # via + # apache-beam + # google-auth +cattrs==23.1.2 + # via looker-sdk +certifi==2023.7.22 + # via + # httpcore + # httpx + # requests +cffi==1.16.0 + # via cryptography +chardet==5.2.0 + # via gcloud-aio-auth +charset-normalizer==3.3.2 + # via requests +click==8.1.7 + # via + # clickclick + # flask + # flask-appbuilder +clickclick==20.10.2 + # via connexion +cloudpickle==2.2.1 + # via apache-beam +colorama==0.4.6 + # via flask-appbuilder +colorlog==4.8.0 + # via apache-airflow +configupdater==3.1.1 + # via apache-airflow +connexion[flask]==2.14.2 + # via + # apache-airflow + # connexion +crcmod==1.7 + # via apache-beam +cron-descriptor==1.4.0 + # via apache-airflow +croniter==2.0.1 + # via apache-airflow +cryptography==41.0.6 + # via + # apache-airflow + # gcloud-aio-auth + # pyopenssl +db-dtypes==1.1.1 + # via pandas-gbq +decorator==5.1.1 + # via gcsfs +deprecated==1.2.14 + # via + # apache-airflow + # limits + # opentelemetry-api + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +dill==0.3.1.1 + # via + # apache-airflow + # apache-beam +dnspython==2.4.2 + # via + # email-validator + # pymongo +docopt==0.6.2 + # via hdfs +docutils==0.20.1 + # via python-daemon +email-validator==1.3.1 + # via flask-appbuilder +exceptiongroup==1.1.3 + # via + # anyio + # cattrs +fastavro==1.9.0 + # via apache-beam +fasteners==0.19 + # via + # apache-beam + # google-apitools +flask==2.2.5 + # via + # apache-airflow + # connexion + # flask-appbuilder + # flask-babel + # flask-caching + # flask-jwt-extended + # flask-limiter + # flask-login + # flask-session + # flask-sqlalchemy + # flask-wtf +flask-appbuilder==4.3.10 + # via apache-airflow +flask-babel==2.0.0 + # via flask-appbuilder +flask-caching==2.1.0 + # via apache-airflow +flask-jwt-extended==4.5.3 + # via flask-appbuilder +flask-limiter==3.5.0 + # via flask-appbuilder +flask-login==0.6.3 + # via + # apache-airflow + # flask-appbuilder +flask-session==0.5.0 + # via apache-airflow +flask-sqlalchemy==2.5.1 + # via flask-appbuilder +flask-wtf==1.2.1 + # via + # apache-airflow + # flask-appbuilder +frozenlist==1.4.0 + # via + # aiohttp + # aiosignal +fsspec==2023.10.0 + # via + # apache-airflow + # gcsfs + # universal-pathlib +gcloud-aio-auth==4.2.3 + # via + # apache-airflow-providers-google + # gcloud-aio-bigquery + # gcloud-aio-storage +gcloud-aio-bigquery==7.0.0 + # via apache-airflow-providers-google +gcloud-aio-storage==9.0.0 + # via apache-airflow-providers-google +gcsfs==2023.10.0 + # via apache-airflow-providers-google +google-ads==22.1.0 + # via apache-airflow-providers-google +google-api-core[grpc]==2.13.0 + # via + # apache-airflow-providers-google + # apache-beam + # google-ads + # google-api-python-client + # google-cloud-aiplatform + # google-cloud-appengine-logging + # google-cloud-automl + # google-cloud-batch + # google-cloud-bigquery + # google-cloud-bigquery-datatransfer + # google-cloud-bigquery-storage + # google-cloud-bigtable + # google-cloud-build + # google-cloud-compute + # google-cloud-container + # google-cloud-core + # google-cloud-datacatalog + # google-cloud-dataflow-client + # google-cloud-dataform + # google-cloud-dataplex + # google-cloud-dataproc + # google-cloud-dataproc-metastore + # google-cloud-datastore + # google-cloud-dlp + # google-cloud-kms + # google-cloud-language + # google-cloud-logging + # google-cloud-memcache + # google-cloud-monitoring + # google-cloud-orchestration-airflow + # google-cloud-os-login + # google-cloud-pubsub + # google-cloud-pubsublite + # google-cloud-recommendations-ai + # google-cloud-redis + # google-cloud-resource-manager + # google-cloud-run + # google-cloud-secret-manager + # google-cloud-spanner + # google-cloud-speech + # google-cloud-storage + # google-cloud-storage-transfer + # google-cloud-tasks + # google-cloud-texttospeech + # google-cloud-translate + # google-cloud-videointelligence + # google-cloud-vision + # google-cloud-workflows + # pandas-gbq + # sqlalchemy-bigquery +google-api-python-client==2.107.0 + # via apache-airflow-providers-google +google-apitools==0.5.31 + # via apache-beam +google-auth==2.23.4 + # via + # apache-airflow-providers-google + # apache-beam + # gcsfs + # google-api-core + # google-api-python-client + # google-auth-httplib2 + # google-auth-oauthlib + # google-cloud-core + # google-cloud-storage + # pandas-gbq + # pydata-google-auth + # sqlalchemy-bigquery +google-auth-httplib2==0.1.1 + # via + # apache-airflow-providers-google + # apache-beam + # google-api-python-client +google-auth-oauthlib==1.1.0 + # via + # gcsfs + # google-ads + # pandas-gbq + # pydata-google-auth +google-cloud-aiplatform==1.36.1 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-appengine-logging==1.3.2 + # via google-cloud-logging +google-cloud-audit-log==0.2.5 + # via google-cloud-logging +google-cloud-automl==2.11.3 + # via apache-airflow-providers-google +google-cloud-batch==0.17.3 + # via apache-airflow-providers-google +google-cloud-bigquery==3.13.0 + # via + # apache-beam + # google-cloud-aiplatform + # pandas-gbq + # sqlalchemy-bigquery +google-cloud-bigquery-datatransfer==3.12.1 + # via apache-airflow-providers-google +google-cloud-bigquery-storage==2.22.0 + # via + # apache-beam + # pandas-gbq +google-cloud-bigtable==2.21.0 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-build==3.21.0 + # via apache-airflow-providers-google +google-cloud-compute==1.14.1 + # via apache-airflow-providers-google +google-cloud-container==2.33.0 + # via apache-airflow-providers-google +google-cloud-core==2.3.3 + # via + # apache-beam + # google-cloud-bigquery + # google-cloud-bigtable + # google-cloud-datastore + # google-cloud-logging + # google-cloud-spanner + # google-cloud-storage + # google-cloud-translate +google-cloud-datacatalog==3.16.0 + # via apache-airflow-providers-google +google-cloud-dataflow-client==0.8.5 + # via apache-airflow-providers-google +google-cloud-dataform==0.5.4 + # via apache-airflow-providers-google +google-cloud-dataplex==1.8.1 + # via apache-airflow-providers-google +google-cloud-dataproc==5.7.0 + # via apache-airflow-providers-google +google-cloud-dataproc-metastore==1.13.0 + # via apache-airflow-providers-google +google-cloud-datastore==2.18.0 + # via apache-beam +google-cloud-dlp==3.13.0 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-kms==2.19.2 + # via apache-airflow-providers-google +google-cloud-language==2.11.1 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-logging==3.8.0 + # via apache-airflow-providers-google +google-cloud-memcache==1.7.3 + # via apache-airflow-providers-google +google-cloud-monitoring==2.16.0 + # via apache-airflow-providers-google +google-cloud-orchestration-airflow==1.9.2 + # via apache-airflow-providers-google +google-cloud-os-login==2.11.0 + # via apache-airflow-providers-google +google-cloud-pubsub==2.18.4 + # via + # apache-airflow-providers-google + # apache-beam + # google-cloud-pubsublite +google-cloud-pubsublite==1.8.3 + # via apache-beam +google-cloud-recommendations-ai==0.10.5 + # via apache-beam +google-cloud-redis==2.13.2 + # via apache-airflow-providers-google +google-cloud-resource-manager==1.10.4 + # via google-cloud-aiplatform +google-cloud-run==0.10.0 + # via apache-airflow-providers-google +google-cloud-secret-manager==2.16.4 + # via apache-airflow-providers-google +google-cloud-spanner==3.40.1 + # via + # apache-airflow-providers-google + # apache-beam + # sqlalchemy-spanner +google-cloud-speech==2.22.0 + # via apache-airflow-providers-google +google-cloud-storage==2.13.0 + # via + # apache-airflow-providers-google + # gcsfs + # google-cloud-aiplatform +google-cloud-storage-transfer==1.9.2 + # via apache-airflow-providers-google +google-cloud-tasks==2.14.2 + # via apache-airflow-providers-google +google-cloud-texttospeech==2.14.2 + # via apache-airflow-providers-google +google-cloud-translate==3.12.1 + # via apache-airflow-providers-google +google-cloud-videointelligence==2.11.4 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-vision==3.4.5 + # via + # apache-airflow-providers-google + # apache-beam +google-cloud-workflows==1.12.1 + # via apache-airflow-providers-google +google-crc32c==1.5.0 + # via + # google-cloud-storage + # google-resumable-media +google-re2==1.1 + # via apache-airflow +google-resumable-media==2.6.0 + # via + # google-cloud-bigquery + # google-cloud-storage +googleapis-common-protos[grpc]==1.61.0 + # via + # google-ads + # google-api-core + # google-cloud-audit-log + # grpc-google-iam-v1 + # grpcio-status + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +graphviz==0.20.1 + # via apache-airflow +greenlet==3.0.1 + # via sqlalchemy +grpc-google-iam-v1==0.12.6 + # via + # google-cloud-bigtable + # google-cloud-build + # google-cloud-datacatalog + # google-cloud-dataform + # google-cloud-dataplex + # google-cloud-dataproc + # google-cloud-dataproc-metastore + # google-cloud-kms + # google-cloud-logging + # google-cloud-pubsub + # google-cloud-resource-manager + # google-cloud-run + # google-cloud-secret-manager + # google-cloud-spanner + # google-cloud-tasks +grpcio==1.59.2 + # via + # apache-beam + # google-ads + # google-api-core + # google-cloud-bigquery + # google-cloud-pubsub + # google-cloud-pubsublite + # googleapis-common-protos + # grpc-google-iam-v1 + # grpcio-gcp + # grpcio-status + # opentelemetry-exporter-otlp-proto-grpc +grpcio-gcp==0.2.2 + # via apache-airflow-providers-google +grpcio-status==1.59.2 + # via + # google-ads + # google-api-core + # google-cloud-pubsub + # google-cloud-pubsublite +gunicorn==21.2.0 + # via apache-airflow +h11==0.14.0 + # via httpcore +hdfs==2.7.3 + # via apache-beam +httpcore==1.0.1 + # via httpx +httplib2==0.22.0 + # via + # apache-beam + # google-api-python-client + # google-apitools + # google-auth-httplib2 + # oauth2client +httpx==0.25.1 + # via + # apache-airflow + # apache-airflow-providers-google +idna==3.4 + # via + # anyio + # email-validator + # httpx + # requests + # yarl +importlib-metadata==6.8.0 + # via opentelemetry-api +importlib-resources==6.1.1 + # via limits +inflection==0.5.1 + # via connexion +itsdangerous==2.1.2 + # via + # apache-airflow + # connexion + # flask + # flask-wtf +jinja2==3.1.3 + # via + # apache-airflow + # flask + # flask-babel + # python-nvd3 +js2py==0.74 + # via apache-beam +json-merge-patch==0.2 + # via apache-airflow-providers-google +jsonschema==4.19.2 + # via + # apache-airflow + # connexion + # flask-appbuilder +jsonschema-specifications==2023.7.1 + # via jsonschema +lazy-object-proxy==1.9.0 + # via apache-airflow +limits==3.6.0 + # via flask-limiter +linkify-it-py==2.0.2 + # via apache-airflow +lockfile==0.12.2 + # via + # apache-airflow + # python-daemon +looker-sdk==23.16.0 + # via apache-airflow-providers-google +mako==1.3.0 + # via alembic +markdown==3.5.1 + # via apache-airflow +markdown-it-py==3.0.0 + # via + # apache-airflow + # mdit-py-plugins + # rich +markupsafe==2.1.3 + # via + # apache-airflow + # jinja2 + # mako + # werkzeug + # wtforms +marshmallow==3.20.1 + # via + # flask-appbuilder + # marshmallow-oneofschema + # marshmallow-sqlalchemy +marshmallow-oneofschema==3.0.1 + # via apache-airflow +marshmallow-sqlalchemy==0.26.1 + # via flask-appbuilder +mdit-py-plugins==0.4.0 + # via apache-airflow +mdurl==0.1.2 + # via markdown-it-py +multidict==6.0.4 + # via + # aiohttp + # yarl +numpy==1.24.4 + # via + # apache-beam + # db-dtypes + # pandas + # pandas-gbq + # pyarrow + # shapely +oauth2client==4.1.3 + # via google-apitools +oauthlib==3.2.2 + # via requests-oauthlib +objsize==0.6.1 + # via apache-beam +opentelemetry-api==1.21.0 + # via + # apache-airflow + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http + # opentelemetry-sdk +opentelemetry-exporter-otlp==1.21.0 + # via apache-airflow +opentelemetry-exporter-otlp-proto-common==1.21.0 + # via + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +opentelemetry-exporter-otlp-proto-grpc==1.21.0 + # via opentelemetry-exporter-otlp +opentelemetry-exporter-otlp-proto-http==1.21.0 + # via opentelemetry-exporter-otlp +opentelemetry-proto==1.21.0 + # via + # opentelemetry-exporter-otlp-proto-common + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +opentelemetry-sdk==1.21.0 + # via + # opentelemetry-exporter-otlp-proto-grpc + # opentelemetry-exporter-otlp-proto-http +opentelemetry-semantic-conventions==0.42b0 + # via opentelemetry-sdk +ordered-set==4.1.0 + # via flask-limiter +orjson==3.9.10 + # via apache-beam +overrides==6.5.0 + # via google-cloud-pubsublite +packaging==23.2 + # via + # apache-airflow + # apache-beam + # apispec + # connexion + # db-dtypes + # google-cloud-aiplatform + # google-cloud-bigquery + # gunicorn + # limits + # marshmallow + # sqlalchemy-bigquery +pandas==2.0.3 + # via + # apache-airflow-providers-google + # db-dtypes + # pandas-gbq +pandas-gbq==0.19.2 + # via apache-airflow-providers-google +pathspec==0.11.2 + # via apache-airflow +pendulum==2.1.2 + # via apache-airflow +pluggy==1.3.0 + # via apache-airflow +prison==0.2.1 + # via flask-appbuilder +proto-plus==1.22.3 + # via + # apache-airflow-providers-google + # apache-beam + # google-ads + # google-cloud-aiplatform + # google-cloud-appengine-logging + # google-cloud-automl + # google-cloud-batch + # google-cloud-bigquery + # google-cloud-bigquery-datatransfer + # google-cloud-bigquery-storage + # google-cloud-bigtable + # google-cloud-build + # google-cloud-compute + # google-cloud-container + # google-cloud-datacatalog + # google-cloud-dataflow-client + # google-cloud-dataform + # google-cloud-dataplex + # google-cloud-dataproc + # google-cloud-dataproc-metastore + # google-cloud-datastore + # google-cloud-dlp + # google-cloud-kms + # google-cloud-language + # google-cloud-logging + # google-cloud-memcache + # google-cloud-monitoring + # google-cloud-orchestration-airflow + # google-cloud-os-login + # google-cloud-pubsub + # google-cloud-recommendations-ai + # google-cloud-redis + # google-cloud-resource-manager + # google-cloud-run + # google-cloud-secret-manager + # google-cloud-spanner + # google-cloud-speech + # google-cloud-storage-transfer + # google-cloud-tasks + # google-cloud-texttospeech + # google-cloud-translate + # google-cloud-videointelligence + # google-cloud-vision + # google-cloud-workflows +protobuf==4.24.4 + # via + # apache-beam + # google-ads + # google-api-core + # google-cloud-aiplatform + # google-cloud-appengine-logging + # google-cloud-audit-log + # google-cloud-automl + # google-cloud-batch + # google-cloud-bigquery + # google-cloud-bigquery-datatransfer + # google-cloud-bigquery-storage + # google-cloud-bigtable + # google-cloud-build + # google-cloud-compute + # google-cloud-container + # google-cloud-datacatalog + # google-cloud-dataflow-client + # google-cloud-dataform + # google-cloud-dataplex + # google-cloud-dataproc + # google-cloud-dataproc-metastore + # google-cloud-datastore + # google-cloud-dlp + # google-cloud-kms + # google-cloud-language + # google-cloud-logging + # google-cloud-memcache + # google-cloud-monitoring + # google-cloud-orchestration-airflow + # google-cloud-os-login + # google-cloud-pubsub + # google-cloud-recommendations-ai + # google-cloud-redis + # google-cloud-resource-manager + # google-cloud-run + # google-cloud-secret-manager + # google-cloud-spanner + # google-cloud-speech + # google-cloud-storage-transfer + # google-cloud-tasks + # google-cloud-texttospeech + # google-cloud-translate + # google-cloud-videointelligence + # google-cloud-vision + # google-cloud-workflows + # googleapis-common-protos + # grpc-google-iam-v1 + # grpcio-status + # opentelemetry-proto + # proto-plus +psutil==5.9.6 + # via apache-airflow +pyarrow==11.0.0 + # via + # apache-beam + # db-dtypes + # pandas-gbq +pyasn1==0.5.0 + # via + # oauth2client + # pyasn1-modules + # rsa +pyasn1-modules==0.3.0 + # via + # gcloud-aio-storage + # google-auth + # oauth2client +pycparser==2.21 + # via cffi +pydantic==2.4.2 + # via apache-airflow +pydantic-core==2.10.1 + # via pydantic +pydata-google-auth==1.8.2 + # via pandas-gbq +pydot==1.4.2 + # via apache-beam +pygments==2.16.1 + # via + # apache-airflow + # rich +pyjsparser==2.7.1 + # via js2py +pyjwt==2.8.0 + # via + # apache-airflow + # flask-appbuilder + # flask-jwt-extended + # gcloud-aio-auth +pymongo==4.6.0 + # via apache-beam +pyopenssl==23.3.0 + # via apache-airflow-providers-google +pyparsing==3.1.1 + # via + # httplib2 + # pydot +python-daemon==3.0.1 + # via apache-airflow +python-dateutil==2.8.2 + # via + # apache-airflow + # apache-beam + # croniter + # flask-appbuilder + # google-cloud-bigquery + # pandas + # pendulum +python-nvd3==0.15.0 + # via apache-airflow +python-slugify==8.0.1 + # via + # apache-airflow + # python-nvd3 +pytz==2023.3.post1 + # via + # apache-beam + # croniter + # flask-babel + # pandas +pytzdata==2020.1 + # via pendulum +pyyaml==6.0.1 + # via + # apispec + # clickclick + # connexion + # google-ads +referencing==0.30.2 + # via + # jsonschema + # jsonschema-specifications +regex==2023.10.3 + # via apache-beam +requests==2.31.0 + # via + # apache-airflow-providers-http + # apache-beam + # connexion + # gcsfs + # google-api-core + # google-cloud-bigquery + # google-cloud-storage + # hdfs + # looker-sdk + # opentelemetry-exporter-otlp-proto-http + # requests-oauthlib + # requests-toolbelt +requests-oauthlib==1.3.1 + # via google-auth-oauthlib +requests-toolbelt==1.0.0 + # via apache-airflow-providers-http +rfc3339-validator==0.1.4 + # via apache-airflow +rich==13.6.0 + # via + # apache-airflow + # flask-limiter + # rich-argparse +rich-argparse==1.4.0 + # via apache-airflow +rpds-py==0.12.0 + # via + # jsonschema + # referencing +rsa==4.9 + # via + # gcloud-aio-storage + # google-auth + # oauth2client +setproctitle==1.3.3 + # via apache-airflow +shapely==2.0.2 + # via google-cloud-aiplatform +six==1.16.0 + # via + # google-apitools + # hdfs + # js2py + # oauth2client + # prison + # python-dateutil + # rfc3339-validator +sniffio==1.3.0 + # via + # anyio + # httpx +sqlalchemy==1.4.50 + # via + # alembic + # apache-airflow + # flask-appbuilder + # flask-sqlalchemy + # marshmallow-sqlalchemy + # sqlalchemy-bigquery + # sqlalchemy-jsonfield + # sqlalchemy-spanner + # sqlalchemy-utils +sqlalchemy-bigquery==1.8.0 + # via apache-airflow-providers-google +sqlalchemy-jsonfield==1.0.1.post0 + # via apache-airflow +sqlalchemy-spanner==1.6.2 + # via apache-airflow-providers-google +sqlalchemy-utils==0.41.1 + # via flask-appbuilder +sqlparse==0.4.4 + # via + # apache-airflow-providers-common-sql + # google-cloud-spanner +tabulate==0.9.0 + # via apache-airflow +tenacity==8.2.3 + # via apache-airflow +termcolor==2.3.0 + # via apache-airflow +text-unidecode==1.3 + # via python-slugify +typing-extensions==4.8.0 + # via + # alembic + # apache-airflow + # apache-beam + # asgiref + # cattrs + # flask-limiter + # limits + # looker-sdk + # opentelemetry-sdk + # pydantic + # pydantic-core +tzdata==2023.3 + # via pandas +tzlocal==5.2 + # via js2py +uc-micro-py==1.0.2 + # via linkify-it-py +unicodecsv==0.14.1 + # via apache-airflow +universal-pathlib==0.1.4 + # via apache-airflow +uritemplate==4.1.1 + # via google-api-python-client +urllib3==2.0.7 + # via requests +werkzeug==2.2.3 + # via + # apache-airflow + # connexion + # flask + # flask-appbuilder + # flask-jwt-extended + # flask-login +wrapt==1.15.0 + # via deprecated +wtforms==3.0.1 + # via + # flask-appbuilder + # flask-wtf +yarl==1.9.2 + # via aiohttp +zipp==3.17.0 + # via importlib-metadata +zstandard==0.22.0 + # via apache-beam + +# The following packages are considered to be unsafe in a requirements file: +# setuptools diff --git a/examples/bigquery_plugin/Dockerfile b/examples/bigquery_integration/Dockerfile similarity index 100% rename from examples/bigquery_plugin/Dockerfile rename to examples/bigquery_integration/Dockerfile diff --git a/examples/bigquery_integration/README.md b/examples/bigquery_integration/README.md new file mode 100644 index 000000000..a66d849eb --- /dev/null +++ b/examples/bigquery_integration/README.md @@ -0,0 +1,21 @@ +```{eval-rst} +.. tags:: GCP, Data, Integration, Advanced +``` + +# BigQuery + +You can configure your Flyte deployment to connect to the Google BigQuery service to query BigQuery tables. + +There are two implementations of the BigQuery integration: + +* **{ref}`BigQuery agent `:** This implementation uses the [agents framework](https://docs.flyte.org/en/latest/flyte_agents/index.html). We strongly recommend using the BigQuery agent rather than the plugin, as agents can be tested locally without running the Flyte backend, are designed to be scalable, can handle large workloads efficiently, and decrease load on FlytePropeller. +* **{ref}`BigQuery plugin `:** This implementation uses the legacy plugin framework. Only use this documentation if you have already configured your Flyte deployment to use the plugin. If you are using the plugin, we highly recommend migrating to the agent framework version instead, if possible. + + +```{toctree} +:maxdepth: -1 +:hidden: +bigquery_agent +bigquery_plugin +bigquery +``` diff --git a/examples/databricks_plugin/databricks_plugin/__init__.py b/examples/bigquery_integration/bigquery_integration/__init__.py similarity index 100% rename from examples/databricks_plugin/databricks_plugin/__init__.py rename to examples/bigquery_integration/bigquery_integration/__init__.py diff --git a/examples/bigquery_plugin/bigquery_plugin/bigquery.py b/examples/bigquery_integration/bigquery_integration/bigquery.py similarity index 98% rename from examples/bigquery_plugin/bigquery_plugin/bigquery.py rename to examples/bigquery_integration/bigquery_integration/bigquery.py index 767eadc32..ef4b9d099 100644 --- a/examples/bigquery_plugin/bigquery_plugin/bigquery.py +++ b/examples/bigquery_integration/bigquery_integration/bigquery.py @@ -1,5 +1,5 @@ # %% [markdown] -# # BigQuery Query +# # BigQuery example query # # This example shows how to use a Flyte BigQueryTask to execute a query. # %% diff --git a/examples/bigquery_integration/bigquery_integration/bigquery_agent.md b/examples/bigquery_integration/bigquery_integration/bigquery_agent.md new file mode 100644 index 000000000..6822d1378 --- /dev/null +++ b/examples/bigquery_integration/bigquery_integration/bigquery_agent.md @@ -0,0 +1,37 @@ +(bigquery_agent)= + +# BigQuery agent + +## Installation + +To install the BigQuery agent, run the following command: + +```{eval-rst} +.. prompt:: bash + + pip install flytekitplugins-bigquery +``` + +This agent is purely a spec. Since SQL is completely portable, there is no need to build a Docker container. + +## Example usage + +For an example query, see {doc}`BigQuery example query`. + +## Local testing + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. + +``` + +## Flyte deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the BigQuery agent in your Flyte deployment, see the {ref}`BigQuery agent deployment guide`. diff --git a/examples/bigquery_integration/bigquery_integration/bigquery_plugin.md b/examples/bigquery_integration/bigquery_integration/bigquery_plugin.md new file mode 100644 index 000000000..7b922ef87 --- /dev/null +++ b/examples/bigquery_integration/bigquery_integration/bigquery_plugin.md @@ -0,0 +1,32 @@ +(bigquery_plugin)= + +# BigQuery plugin + +```{note} + +This is a legacy implementation of the BigQuery integration. We recommend using the {ref}`BigQuery agent ` instead. + +``` + +## Installation + +To install the BigQuery plugin, run the following command: + +```{eval-rst} +.. prompt:: bash + + pip install flytekitplugins-bigquery +``` + +This plugin is purely a spec. Since SQL is completely portable, there is no need to build a Docker container. + +## Example usage + +For a usage example, see the {doc}`BigQuery example query` page. + +## Flyte deployment configuration + +BigQuery plugins are [enabled in FlytePropeller's config](https://docs.flyte.org/en/latest/deployment/plugin_setup/gcp/bigquery.html#deployment-plugin-setup-gcp-bigquery). + +To run the BigQuery plugin on a Flyte cluster, you must configure it in your Flyte deployment. For more information, see the +{std:ref}`BigQuery plugin setup guide `. diff --git a/examples/bigquery_plugin/requirements.in b/examples/bigquery_integration/requirements.in similarity index 100% rename from examples/bigquery_plugin/requirements.in rename to examples/bigquery_integration/requirements.in diff --git a/examples/bigquery_plugin/requirements.txt b/examples/bigquery_integration/requirements.txt similarity index 100% rename from examples/bigquery_plugin/requirements.txt rename to examples/bigquery_integration/requirements.txt diff --git a/examples/bigquery_plugin/README.md b/examples/bigquery_plugin/README.md deleted file mode 100644 index f7319f6ee..000000000 --- a/examples/bigquery_plugin/README.md +++ /dev/null @@ -1,30 +0,0 @@ -(big-query)= - -# BigQuery - -```{eval-rst} -.. tags:: GCP, Data, Integration, Advanced -``` - -Flyte backend can be connected with BigQuery service. Once enabled, it can allow you to query a BigQuery table. -This section will provide how to use the BigQuery Plugin using flytekit python. - -## Installation - -To use the flytekit bigquery plugin simply run the following: - -```{eval-rst} -.. prompt:: bash - - pip install flytekitplugins-bigquery -``` - -This plugin is purely a spec. Since SQL is completely portable, there is no need to build a Docker container. - -## Configuring the backend to get bigquery working - -BigQuery plugins are [enabled in flytepropeller's config](https://docs.flyte.org/en/latest/deployment/plugin_setup/gcp/bigquery.html#deployment-plugin-setup-gcp-bigquery) - -```{auto-examples-toc} -bigquery -``` diff --git a/examples/databricks_plugin/Dockerfile b/examples/databricks_integration/Dockerfile similarity index 100% rename from examples/databricks_plugin/Dockerfile rename to examples/databricks_integration/Dockerfile diff --git a/examples/databricks_integration/README.md b/examples/databricks_integration/README.md new file mode 100644 index 000000000..0065c0ff8 --- /dev/null +++ b/examples/databricks_integration/README.md @@ -0,0 +1,21 @@ +# Databricks + +```{eval-rst} +.. tags:: Spark, Integration, DistributedComputing, Data, Advanced +``` + +Flyte can be integrated with the [Databricks](https://www.databricks.com/) service, +enabling you to submit Spark jobs to the Databricks platform. + +There are two implementations of the Databricks integration: + +* **{ref}`Databricks agent `:** This implementation uses the [agents framework](https://docs.flyte.org/en/latest/flyte_agents/index.html). We strongly recommend using the Databricks agent rather than the plugin, as agents can be tested locally without running the Flyte backend, are designed to be scalable, can handle large workloads efficiently, and decrease load on FlytePropeller. +* **{ref}`Databricks plugin `:** This implementation uses the legacy plugin framework. Only use this documentation if you have already configured your Flyte deployment to use the plugin. If you are using the plugin, we highly recommend migrating to the agent framework version instead, if possible. + +```{toctree} +:maxdepth: -1 +:hidden: +databricks_agent +databricks_plugin +databricks_job +``` diff --git a/examples/mmcloud_plugin/mmcloud_plugin/__init__.py b/examples/databricks_integration/databricks_integration/__init__.py similarity index 100% rename from examples/mmcloud_plugin/mmcloud_plugin/__init__.py rename to examples/databricks_integration/databricks_integration/__init__.py diff --git a/examples/databricks_integration/databricks_integration/databricks_agent.md b/examples/databricks_integration/databricks_integration/databricks_agent.md new file mode 100644 index 000000000..d548c8150 --- /dev/null +++ b/examples/databricks_integration/databricks_integration/databricks_agent.md @@ -0,0 +1,34 @@ +(databricks_agent)= + +# Databricks agent + +## Installation + +The Databricks agent comes bundled with the Spark plugin. To install the Spark plugin, run the following command: + +``` +pip install flytekitplugins-spark + +``` + +## Example usage + +For a usage example, see the {doc}`Databricks job ` page. + +## Local testing + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. + +``` + +## Flyte deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the Databricks agent in your Flyte deployment, see the {ref}`Databricks agent setup guide `. diff --git a/examples/databricks_plugin/databricks_plugin/databricks_job.py b/examples/databricks_integration/databricks_integration/databricks_job.py similarity index 100% rename from examples/databricks_plugin/databricks_plugin/databricks_job.py rename to examples/databricks_integration/databricks_integration/databricks_job.py diff --git a/examples/databricks_integration/databricks_integration/databricks_plugin.md b/examples/databricks_integration/databricks_integration/databricks_plugin.md new file mode 100644 index 000000000..7d8e84e99 --- /dev/null +++ b/examples/databricks_integration/databricks_integration/databricks_plugin.md @@ -0,0 +1,49 @@ +(databricks_plugin)= + +# Databricks plugin + +```{note} + +This is a legacy implementation of the Databricks integration. We recommend using the {ref}`Databricks agent ` instead. + +``` + +## Installation + +The Databricks plugin comes bundled with the Spark plugin. To install the Spark plugin, run the following command: + +``` +pip install flytekitplugins-spark + +``` + +## Flyte deployment configuration + +To run the Databricks plugin on a Flyte cluster, you must configure it in your Flyte deployment. For more information, see the +{std:ref}`Databricks plugin setup guide `. + +## Example usage + +For a usage example, see the {doc}`Databricks job ` page. + +### Run the example on the Flyte cluster + +To run the provided example on a Flyte cluster, use the following command: + +``` +pyflyte run --remote \ + --image ghcr.io/flyteorg/flytecookbook:databricks_plugin-latest \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/databricks_integration/databricks_integration/databricks_job.py \ + my_databricks_job +``` + +Using Spark on Databricks allows comprehensive versioning through a +custom-built Spark container. This container also facilitates the execution of standard Spark tasks. + +To use Spark, the image should employ a base image provided by Databricks, +and the workflow code must be copied to `/databricks/driver`. + +```{literalinclude} ../../../examples/databricks_integration/Dockerfile +:language: docker +:emphasize-lines: 1,7-8,20 +``` diff --git a/examples/databricks_plugin/requirements.in b/examples/databricks_integration/requirements.in similarity index 100% rename from examples/databricks_plugin/requirements.in rename to examples/databricks_integration/requirements.in diff --git a/examples/databricks_plugin/requirements.txt b/examples/databricks_integration/requirements.txt similarity index 100% rename from examples/databricks_plugin/requirements.txt rename to examples/databricks_integration/requirements.txt diff --git a/examples/databricks_plugin/README.md b/examples/databricks_plugin/README.md deleted file mode 100644 index 80c2fc01f..000000000 --- a/examples/databricks_plugin/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Databricks - -```{eval-rst} -.. tags:: Spark, Integration, DistributedComputing, Data, Advanced -``` - -Flyte can be seamlessly integrated with the [Databricks](https://www.databricks.com/) service, -enabling you to effortlessly submit Spark jobs to the Databricks platform. - -## Install the plugin - -The Databricks plugin comes bundled with the Spark plugin. -To execute it locally, run the following command: - -``` -pip install flytekitplugins-spark -``` - -If you intend to run the plugin on the Flyte cluster, you must first set it up on the backend. -Please refer to the -{std:ref}`Databricks plugin setup guide ` -for detailed instructions. - -## Run the example on the Flyte cluster - -To run the provided example on the Flyte cluster, use the following command: - -``` -pyflyte run --remote \ - --image ghcr.io/flyteorg/flytecookbook:databricks_plugin-latest \ - https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/databricks_plugin/databricks_plugin/databricks_job.py \ - my_databricks_job -``` - -:::{note} -Using Spark on Databricks is incredibly simple and offers comprehensive versioning through a -custom-built Spark container. This built container also facilitates the execution of standard Spark tasks. - -To utilize Spark, the image should employ a base image provided by Databricks, -and the workflow code must be copied to `/databricks/driver`. - -```{literalinclude} ../../../examples/databricks_plugin/Dockerfile -:language: docker -:emphasize-lines: 1,7-8,20 -``` - -::: - -```{auto-examples-toc} -databricks_job -``` diff --git a/examples/mmcloud_plugin/Dockerfile b/examples/mmcloud_integration/Dockerfile similarity index 100% rename from examples/mmcloud_plugin/Dockerfile rename to examples/mmcloud_integration/Dockerfile diff --git a/examples/mmcloud_integration/README.md b/examples/mmcloud_integration/README.md new file mode 100644 index 000000000..76aab157a --- /dev/null +++ b/examples/mmcloud_integration/README.md @@ -0,0 +1,22 @@ +# Memory Machine Cloud + +```{eval-rst} +.. tags:: AWS, GCP, AliCloud, Integration, Advanced +``` + +[MemVerge](https://memverge.com/) [Memory Machine Cloud](https://www.mmcloud.io/) (MMCloud)—available on AWS, GCP, and AliCloud—empowers users to continuously optimize cloud resources during runtime, safely execute stateful tasks on spot instances, and monitor resource usage in real time. These capabilities make it an excellent fit for long-running batch workloads. Flyte can be integrated with MMCloud, allowing you to execute Flyte tasks using MMCloud. + +There are two implementations of the MMCloud integration: + +* **{ref}`MMCloud agent `:** This implementation uses the [agents framework](https://docs.flyte.org/en/latest/flyte_agents/index.html). We strongly recommend using the MMCloud agent rather than the plugin, as agents can be tested locally without running the Flyte backend, are designed to be scalable, can handle large workloads efficiently, and decrease load on FlytePropeller. +* **{ref}`MMCloud plugin `:** This implementation uses the legacy plugin framework. Only use this documentation if you have already configured your Flyte deployment to use the plugin. If you are using the plugin, we highly recommend migrating to the agent framework version instead, if possible. + + +```{toctree} +:maxdepth: -1 +:hidden: + +mmcloud_agent +mmcloud_plugin +mmcloud_example +``` diff --git a/examples/snowflake_plugin/snowflake_plugin/__init__.py b/examples/mmcloud_integration/mmcloud_integration/__init__.py similarity index 100% rename from examples/snowflake_plugin/snowflake_plugin/__init__.py rename to examples/mmcloud_integration/mmcloud_integration/__init__.py diff --git a/examples/mmcloud_integration/mmcloud_integration/mmcloud_agent.md b/examples/mmcloud_integration/mmcloud_integration/mmcloud_agent.md new file mode 100644 index 000000000..5445963ff --- /dev/null +++ b/examples/mmcloud_integration/mmcloud_integration/mmcloud_agent.md @@ -0,0 +1,37 @@ +(mmcloud_agent)= + +# Memory Machine Cloud agent + +## Installation + +To install the agent, run the following command: + +```{eval-rst} +.. prompt:: bash + + pip install flytekitplugins-mmcloud +``` + +To get started with Memory Machine Cloud, see the [Memory Machine Cloud user guide](https://docs.memverge.com/mmce/current/userguide/olh/index.html). + +## Example usage + +For a usage example, see the {ref}`Memory Machine Cloud example ` page. + +## Local testing + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. + +``` + +## Flyte deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the Memory Machine Cloud agent in your Flyte deployment, see the {ref}`MMCloud agent setup guide `. diff --git a/examples/mmcloud_plugin/mmcloud_plugin/example.py b/examples/mmcloud_integration/mmcloud_integration/mmcloud_example.py similarity index 98% rename from examples/mmcloud_plugin/mmcloud_plugin/example.py rename to examples/mmcloud_integration/mmcloud_integration/mmcloud_example.py index f0f89f21d..f9a96f2be 100644 --- a/examples/mmcloud_plugin/mmcloud_plugin/example.py +++ b/examples/mmcloud_integration/mmcloud_integration/mmcloud_example.py @@ -1,4 +1,5 @@ # %% [markdown] +# (mmcloud_example)= # # Memory Machine Cloud # # This example shows how to use the MMCloud plugin to execute tasks on MemVerge Memory Machine Cloud. diff --git a/examples/mmcloud_plugin/README.md b/examples/mmcloud_integration/mmcloud_integration/mmcloud_plugin.md similarity index 52% rename from examples/mmcloud_plugin/README.md rename to examples/mmcloud_integration/mmcloud_integration/mmcloud_plugin.md index 21279fe58..f7858fab3 100644 --- a/examples/mmcloud_plugin/README.md +++ b/examples/mmcloud_integration/mmcloud_integration/mmcloud_plugin.md @@ -1,12 +1,12 @@ -# Memory Machine Cloud +(mmcloud_plugin)= -```{eval-rst} -.. tags:: AWS, GCP, AliCloud, Integration, Advanced -``` +# Memory Machine Cloud plugin + +```{note} -[MemVerge](https://memverge.com/) [Memory Machine Cloud](https://www.mmcloud.io/) (MMCloud)—available on AWS, GCP, and AliCloud—empowers users to continuously optimize cloud resources during runtime, safely execute stateful tasks on spot instances, and monitor resource usage in real time. These capabilities make it an excellent fit for long-running batch workloads. +This is a legacy implementation of the Memory Machine Cloud integration. We recommend using the {ref}`Memory Machine Cloud agent ` instead. -Flyte can be integrated with MMCloud, allowing you to execute Flyte tasks using MMCloud. +``` ## Installation @@ -18,9 +18,9 @@ To install the plugin, run the following command: pip install flytekitplugins-mmcloud ``` -To get started with MMCloud, refer to the [MMCloud User Guide](https://docs.memverge.com/mmce/current/userguide/olh/index.html). +To get started with MMCloud, see the [MMCloud user guide](https://docs.memverge.com/mmce/current/userguide/olh/index.html). -## Configuring the backend to get MMCloud working +## Flyte deployment configuration The MMCloud plugin is [enabled in FlytePropeller's configuration](https://docs.flyte.org/en/latest/deployment/plugins/memverge/mmcloud.html). @@ -37,13 +37,13 @@ This plugin allows executing `PythonFunctionTask` using MMCloud without changing return str(i) ``` -[Resource](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/customizing_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/multi_images.html) images, and [environment](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html) variable specifications are supported. +[Resource](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/customizing_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/en/latest/flytesnacks/examples/customizing_dependencies/multi_images.html) images, and environment variable specifications are supported. -[ImageSpec](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/image_spec.html) may be used to define images to run tasks. +[ImageSpec](https://docs.flyte.org/en/latest/flytesnacks/examples/customizing_dependencies/image_spec.html#image-spec-example) may be used to define images to run tasks. ### Credentials -The following [secrets](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/use_secrets.html) are required to be defined for the agent server: +The following [secrets](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/use_secrets.html) are required to be defined for the agent server: * `mmc_address`: MMCloud OpCenter address * `mmc_username`: MMCloud OpCenter username * `mmc_password`: MMCloud OpCenter password @@ -78,6 +78,6 @@ COPY float /usr/local/bin/float CMD pyflyte serve agent --port 8000 ``` -```{auto-examples-toc} -example -``` +## Example usage + +For a usage example, see the {ref}`Memory Machine Cloud example ` page. diff --git a/examples/mmcloud_plugin/requirements.in b/examples/mmcloud_integration/requirements.in similarity index 100% rename from examples/mmcloud_plugin/requirements.in rename to examples/mmcloud_integration/requirements.in diff --git a/examples/mmcloud_plugin/requirements.txt b/examples/mmcloud_integration/requirements.txt similarity index 100% rename from examples/mmcloud_plugin/requirements.txt rename to examples/mmcloud_integration/requirements.txt diff --git a/examples/sensor/README.md b/examples/sensor/README.md index 9bbcb5f81..f337a34a6 100644 --- a/examples/sensor/README.md +++ b/examples/sensor/README.md @@ -6,15 +6,29 @@ .. tags:: Data, Basic ``` -## Run the example on the Flyte cluster +## Usage -To run the provided example on the Flyte cluster, use the following command: +For an example of detecting a file with the `FileSensor`, see the {doc}`file sensor example `. + +### Run the file senseor example on a Flyte cluster + +To run the provided example on a Flyte cluster, use the following command: ``` pyflyte run --remote \ https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/sensor/sensor/file_sensor_example.py wf ``` -```{auto-examples-toc} +## Deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the sensor agent in your Flyte deployment, see the {ref}`sensor agent deployment guide`. + +```{toctree} +:maxdepth: -1 +:hidden: file_sensor_example ``` diff --git a/examples/snowflake_plugin/Dockerfile b/examples/snowflake_integration/Dockerfile similarity index 100% rename from examples/snowflake_plugin/Dockerfile rename to examples/snowflake_integration/Dockerfile diff --git a/examples/snowflake_integration/README.md b/examples/snowflake_integration/README.md new file mode 100644 index 000000000..4a6c01298 --- /dev/null +++ b/examples/snowflake_integration/README.md @@ -0,0 +1,22 @@ +(snowflake_integration)= +# Snowflake + +```{eval-rst} +.. tags:: AWS, GCP, AliCloud, Integration, Advanced +``` +Flyte can be seamlessly integrated with the [Snowflake](https://www.snowflake.com) service, +providing you with a straightforward means to query data in Snowflake. + +There are two implementations of the Snowflake integration: + +* **{ref}`Snowflake agent `:** This implementation uses the [agents framework](https://docs.flyte.org/en/latest/flyte_agents/index.html). We strongly recommend using the Snowflake agent rather than the plugin, as agents can be tested locally without running the Flyte backend, are designed to be scalable, can handle large workloads efficiently, and decrease load on FlytePropeller. +* **{ref}`Snowflake plugin `:** This implementation uses the legacy plugin framework. Only use this documentation if you have already configured your Flyte deployment to use the plugin. If you are using the plugin, we highly recommend migrating to the agent framework version instead, if possible. + +```{toctree} +:maxdepth: -1 +:hidden: + +snowflake_agent +snowflake_plugin +snowflake +``` diff --git a/examples/snowflake_plugin/requirements.in b/examples/snowflake_integration/requirements.in similarity index 100% rename from examples/snowflake_plugin/requirements.in rename to examples/snowflake_integration/requirements.in diff --git a/examples/snowflake_plugin/requirements.txt b/examples/snowflake_integration/requirements.txt similarity index 100% rename from examples/snowflake_plugin/requirements.txt rename to examples/snowflake_integration/requirements.txt diff --git a/examples/snowflake_integration/snowflake_integration/__init__.py b/examples/snowflake_integration/snowflake_integration/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/snowflake_plugin/snowflake_plugin/snowflake.py b/examples/snowflake_integration/snowflake_integration/snowflake.py similarity index 99% rename from examples/snowflake_plugin/snowflake_plugin/snowflake.py rename to examples/snowflake_integration/snowflake_integration/snowflake.py index 4a466b3f0..a9a87e1b6 100644 --- a/examples/snowflake_plugin/snowflake_plugin/snowflake.py +++ b/examples/snowflake_integration/snowflake_integration/snowflake.py @@ -1,5 +1,5 @@ # %% [markdown] -# # Quering data in Snowflake +# # Querying data in Snowflake # # This example shows how to use the `SnowflakeTask` to execute a query in Snowflake. # diff --git a/examples/snowflake_integration/snowflake_integration/snowflake_agent.md b/examples/snowflake_integration/snowflake_integration/snowflake_agent.md new file mode 100644 index 000000000..a13aa8dbe --- /dev/null +++ b/examples/snowflake_integration/snowflake_integration/snowflake_agent.md @@ -0,0 +1,33 @@ +(snowflake_agent)= + +# Snowflake agent + +## Installation + +To use the Snowflake plugin, run the following command: + +``` +pip install flytekitplugins-snowflake +``` + +## Example usage + +For a usage example, see the {ref}`Snowflake example page `. + +## Local testing + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. + +``` + +## Flyte deployment configuration + +```{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +``` + +To enable the Snowflake agent in your Flyte deployment, see the {ref}`Snowflake agent setup guide `. diff --git a/examples/snowflake_plugin/README.md b/examples/snowflake_integration/snowflake_integration/snowflake_plugin.md similarity index 67% rename from examples/snowflake_plugin/README.md rename to examples/snowflake_integration/snowflake_integration/snowflake_plugin.md index d1539cac2..34f622387 100644 --- a/examples/snowflake_plugin/README.md +++ b/examples/snowflake_integration/snowflake_integration/snowflake_plugin.md @@ -1,15 +1,14 @@ -(plugins-snowflake)= +(snowflake_plugin)= -# Snowflake +# Snowflake plugin -```{eval-rst} -.. tags:: Integration, Data, Advanced, SQL -``` +```{note} + +This is a legacy implementation of the Snowflake integration. We recommend using the {ref}`Snowflake agent ` instead. -Flyte can be seamlessly integrated with the [Snowflake](https://www.snowflake.com) service, -providing you with a straightforward means to query data in Snowflake. +``` -## Install the plugin +## Installation To use the Snowflake plugin, run the following command: @@ -17,6 +16,8 @@ To use the Snowflake plugin, run the following command: pip install flytekitplugins-snowflake ``` +## Flyte deployment configuration + If you intend to run the plugin on the Flyte cluster, you must first set it up on the backend. Please refer to the {std:ref}`Snowflake plugin setup guide ` @@ -31,7 +32,3 @@ pyflyte run --remote \ https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/snowflake_plugin/snowflake_plugin/snowflake.py \ snowflake_wf --nation_key 10 ``` - -```{auto-examples-toc} -snowflake -```