Code

Slurm agent fn task#3150

Code

Merged

Future-Outlier merged 85 commits intoflyteorg:masterflyteorg/flytekit:masterfrom JiangJiaWei1103:slurm-agent-fn-taskJiangJiaWei1103/flytekit:slurm-agent-fn-taskCopy head branch name to clipboardFeb 22, 2025

+512-5

Commits

Commits on Dec 14, 2024

• Add slurm plugin blank components

  Show description for 421d1b8
  JiangJiaWei1103

  committed
  421d1b8

  Copy full SHA for 421d1b8

Commits on Dec 16, 2024

• feat: Add naive slurm agent create and get with rest api

  Show description for 1d1f806
  JiangJiaWei1103

  committed
  1d1f806

  Copy full SHA for 1d1f806

Commits on Dec 17, 2024

• Use asyncssh instead of REST API

  Show description for 5d97126
  JiangJiaWei1103

  committed
  5d97126

  Copy full SHA for 5d97126

Commits on Dec 18, 2024

• Test ssh communication and run sbatch

  Show description for 2e7f0f2
  JiangJiaWei1103

  committed
  2e7f0f2

  Copy full SHA for 2e7f0f2

Commits on Dec 19, 2024

• Add delete method and support slurm job state

  Show description for 9644b99
  JiangJiaWei1103

  committed
  9644b99

  Copy full SHA for 9644b99

Commits on Dec 27, 2024

• feat: Submit and run SlurmTask on a remote Slurm cluster

  Show description for e41b181
  JiangJiaWei1103

  committed
  e41b181

  Copy full SHA for e41b181

Commits on Dec 28, 2024

• refactor: Remove redundant task_module transfer

  Show description for 6db24dc
  JiangJiaWei1103

  committed
  6db24dc

  Copy full SHA for 6db24dc

• refactor: Remove redundant env var

  Show description for 122c7f1
  JiangJiaWei1103

  committed
  122c7f1

  Copy full SHA for 122c7f1

Commits on Dec 30, 2024

• docs: Add env setup guide for local test

  Show description for e9760a7
  JiangJiaWei1103

  committed
  e9760a7

  Copy full SHA for e9760a7

• docs: Add links and figures

  Show description for e68fda9
  JiangJiaWei1103

  committed
  e68fda9

  Copy full SHA for e68fda9

• docs: Fix commit sha

  Show description for 470637c
  JiangJiaWei1103

  committed
  470637c

  Copy full SHA for 470637c

• docs: Fix commit sha for demo guide

  Show description for 1579ab4
  JiangJiaWei1103

  committed
  1579ab4

  Copy full SHA for 1579ab4

• docs: Fix links

  Show description for 0e538f0
  JiangJiaWei1103

  committed
  0e538f0

  Copy full SHA for 0e538f0

Commits on Dec 31, 2024

• feat: Support SSH config in task config

  Show description for 8229418
  JiangJiaWei1103

  committed
  8229418

  Copy full SHA for 8229418

• docs: Include ssh config in demo example

  Show description for 9e6d8a6
  JiangJiaWei1103

  committed
  9e6d8a6

  Copy full SHA for 9e6d8a6

Commits on Jan 1, 2025

• fix: Retain user-specified file format info

  Show description for cdb56c0
  JiangJiaWei1103

  committed
  cdb56c0

  Copy full SHA for cdb56c0

Commits on Jan 3, 2025

• fix: Set sdt format based on user-specified file_format

  Show description for 36ab6e9
  JiangJiaWei1103

  committed
  36ab6e9

  Copy full SHA for 36ab6e9

• Merge branch 'master' into fix-sd-empty-str-file-format

  JiangJiaWei1103

  committed
  58751fb

  Copy full SHA for 58751fb

Commits on Jan 4, 2025

• Remove redundant modification

  Show description for 4af1f00
  JiangJiaWei1103

  committed
  4af1f00

  Copy full SHA for 4af1f00

• test: Test file_format attribute alignment in dc.sd

  Show description for 46b7b62
  JiangJiaWei1103

  committed
  46b7b62

  Copy full SHA for 46b7b62

Commits on Jan 7, 2025

• refactor: Reduce ssh_conf option to slurm_host only

  Show description for e07b09a
  JiangJiaWei1103

  committed
  e07b09a

  Copy full SHA for e07b09a

• feat: Support Slurm agent with ShellTask

  Show description for 3a7eb6d
  JiangJiaWei1103

  committed
  3a7eb6d

  Copy full SHA for 3a7eb6d

Commits on Jan 9, 2025

• feat: Simplify Slurm job submission logic

  Show description for a815fd9
  JiangJiaWei1103

  committed
  a815fd9

  Copy full SHA for a815fd9

Commits on Jan 10, 2025

• Added script args to agent and task

  Show description for a3ea014
  pryce-turner

  committed
  a3ea014

  Copy full SHA for a3ea014

Commits on Jan 11, 2025

• Add asyncssh to dependencies

  Show description for a109bd8
  JiangJiaWei1103

  committed
  a109bd8

  Copy full SHA for a109bd8

• docs: Update setup and demo for a basic use case

  Show description for e5da665
  JiangJiaWei1103

  committed
  e5da665

  Copy full SHA for e5da665

• docs: Update basic arch figure path

  Show description for 0a3d9f1
  JiangJiaWei1103

  committed
  0a3d9f1

  Copy full SHA for 0a3d9f1

• docs: Fix typo and hyperlink

  Show description for 1b0f6df
  JiangJiaWei1103

  committed
  1b0f6df

  Copy full SHA for 1b0f6df

• fix: A tmp workaround to test agent locally without container_image

  Show description for 26cc201
  JiangJiaWei1103

  committed
  26cc201

  Copy full SHA for 26cc201

Commits on Jan 14, 2025

• feat: Support user-defined batch script content with SlurmShellTask

  Show description for 16d953e
  JiangJiaWei1103

  committed
  16d953e

  Copy full SHA for 16d953e

Commits on Jan 15, 2025

• feat: Fall back to PythonTask for naive use cases

  Show description for c743917
  JiangJiaWei1103

  committed
  c743917

  Copy full SHA for c743917

• refactor: Define Slurm as a base task config and extend for remote script

  Show description for e365dee
  JiangJiaWei1103

  committed
  e365dee

  Copy full SHA for e365dee

Commits on Jan 16, 2025

• feat: Support PythonFunctionTask and reorganize agent structure

  Show description for c1064d4
  JiangJiaWei1103

  committed
  c1064d4

  Copy full SHA for c1064d4

Commits on Jan 22, 2025

• Use poetry virtual env to avoid contamination

  Show description for 361fbd1
  JiangJiaWei1103

  committed
  361fbd1

  Copy full SHA for 361fbd1

Commits on Jan 23, 2025

• docs: Complete local test env setup process

  Show description for fc3e34e
  JiangJiaWei1103

  committed
  fc3e34e

  Copy full SHA for fc3e34e

• docs: Add use cases ranging from basic to advanced

  Show description for 5cd58ec
  JiangJiaWei1103

  committed
  5cd58ec

  Copy full SHA for 5cd58ec

Commits on Feb 6, 2025

• feat: Add a script option for the Slurm function task

  Show description for 9985305
  JiangJiaWei1103

  committed
  9985305

  Copy full SHA for 9985305

• fix: Avoid attaching async resource to different event loops

  Show description for e00a5db
  JiangJiaWei1103

  committed
  e00a5db

  Copy full SHA for e00a5db

• Merge branch 'master' into slurm-agent-dev

  Future-Outlier

  committed
  805548c

  Copy full SHA for 805548c

Commits on Feb 12, 2025

• use await self._connect(slurm_host) in slurm agent

  Show description for 34c7c51
  Future-Outlier

  committed
  34c7c51

  Copy full SHA for 34c7c51

• change

  Show description for 4e85b0a
  Future-Outlier

  committed
  4e85b0a

  Copy full SHA for 4e85b0a

Commits on Feb 13, 2025

• print more info

  Show description for 27cdc8d
  Future-Outlier

  committed
  27cdc8d

  Copy full SHA for 27cdc8d

• use logger

  Show description for 1f30e04
  Future-Outlier

  committed
  1f30e04

  Copy full SHA for 1f30e04

• print more infor

  Show description for 079d101
  Future-Outlier

  committed
  079d101

  Copy full SHA for 079d101

• print

  Show description for 1ce5cf8
  Future-Outlier

  committed
  1ce5cf8

  Copy full SHA for 1ce5cf8

• Use sbatch for running Slurm function task

  Show description for 7b9b38f
  JiangJiaWei1103

  committed
  7b9b38f

  Copy full SHA for 7b9b38f

Commits on Feb 14, 2025

• update

  Show description for b23174a
  Future-Outlier

  committed
  b23174a

  Copy full SHA for b23174a

• push

  Show description for 3727f4d
  Future-Outlier

  committed
  3727f4d

  Copy full SHA for 3727f4d

• feat: Show stdout and stderr msg of the Slurm cluster

  Show description for f291e0b
  JiangJiaWei1103

  committed
  f291e0b

  Copy full SHA for f291e0b

• feat: Show stdout and stderr msg of the Slurm cluster for SlurmFunctionTask

  Show description for a465607
  JiangJiaWei1103

  committed
  a465607

  Copy full SHA for a465607

Commits on Feb 16, 2025

• feat: Make an SSH connetion based on client config file or ssh_config

  Show description for 7b75a51
  JiangJiaWei1103

  committed
  7b75a51

  Copy full SHA for 7b75a51

Commits on Feb 17, 2025

• Clarify SSH connection logic

  Show description for d0967bd
  JiangJiaWei1103

  committed
  d0967bd

  Copy full SHA for d0967bd

Commits on Feb 18, 2025

• feat: Interpolate the script with dynamic input values

  Show description for f80ccd4
  JiangJiaWei1103

  committed
  f80ccd4

  Copy full SHA for f80ccd4

• feat: Interpolate the script with dynamic output values

  Show description for ac25446
  JiangJiaWei1103

  committed
  ac25446

  Copy full SHA for ac25446

Commits on Feb 19, 2025

• Merge branch 'master' into slurm-agent-dev

  Future-Outlier

  committed
  aa6e3ae

  Copy full SHA for aa6e3ae

• add assertion

  Show description for 5fcf6c5
  Future-Outlier

  committed
  5fcf6c5

  Copy full SHA for 5fcf6c5

• update

  Show description for 64e9e06
  Future-Outlier

  committed
  64e9e06

  Copy full SHA for 64e9e06

• update

  Show description for 9d06ed1
  Future-Outlier

  committed
  9d06ed1

  Copy full SHA for 9d06ed1

• Fix Script agent bug

  Show description for 6cda6ae
  Future-Outlier

  committed
  6cda6ae

  Copy full SHA for 6cda6ae

Commits on Feb 20, 2025

• agent service for shell task

  Show description for 3b64ddd
  Future-Outlier

  committed
  3b64ddd

  Copy full SHA for 3b64ddd

• Remove remote path to avoid race condition

  Show description for b107b07
  Future-Outlier

  committed
  b107b07

  Copy full SHA for b107b07

• Revert agent server change

  Show description for 37e4ee1
  Future-Outlier

  committed
  37e4ee1

  Copy full SHA for 37e4ee1

• use key val to run ssh config

  Show description for 77f4d61
  Future-Outlier

  committed
  77f4d61

  Copy full SHA for 77f4d61

• update

  Show description for 68ab8a6
  Future-Outlier

  committed
  68ab8a6

  Copy full SHA for 68ab8a6

• use _get_or_create_ssh_connection

  Show description for a968c01
  Future-Outlier

  committed
  a968c01

  Copy full SHA for a968c01

• update

  Show description for 13cd31e
  Future-Outlier

  committed
  13cd31e

  Copy full SHA for 13cd31e

• use SlurmCluster and hash

  Show description for 52d73d2
  Future-Outlier

  committed
  52d73d2

  Copy full SHA for 52d73d2

• updagte

  Show description for 6b27668
  Future-Outlier

  committed
  6b27668

  Copy full SHA for 6b27668

• update

  Show description for 5aa4e87
  Future-Outlier

  committed
  5aa4e87

  Copy full SHA for 5aa4e87

• update

  Show description for 3cc29fe
  Future-Outlier

  committed
  3cc29fe

  Copy full SHA for 3cc29fe

• refactor: Simplify validation process and clean up legacy code

  Show description for 19de56a
  JiangJiaWei1103

  committed
  19de56a

  Copy full SHA for 19de56a

Commits on Feb 21, 2025

• Add Slurm agent function task

  Show description for 31fc564
  JiangJiaWei1103

  committed
  31fc564

  Copy full SHA for 31fc564

• Revert ShellTask behavior

  Show description for 80f3693
  JiangJiaWei1103

  committed
  80f3693

  Copy full SHA for 80f3693

• Remove fix for SlurmShellTask

  Show description for 96756df
  JiangJiaWei1103

  committed
  96756df

  Copy full SHA for 96756df

• Remove blank line

  Show description for 074cf91
  JiangJiaWei1103

  committed
  074cf91

  Copy full SHA for 074cf91

• fix doc string and remove logs

  Show description for 2d0e0d7
  Future-Outlier

  committed
  2d0e0d7

  Copy full SHA for 2d0e0d7

• build plugins

  Show description for acfaa76
  Future-Outlier

  committed
  acfaa76

  Copy full SHA for acfaa76

Commits on Feb 22, 2025

• merge

  Show description for 6227801
  JiangJiaWei1103

  committed
  6227801

  Copy full SHA for 6227801

• Merge branch 'master' into slurm-agent-fn-task

  Future-Outlier

  committed
  860b703

  Copy full SHA for 860b703

• merge master

  Show description for 45aae83
  Future-Outlier

  committed
  45aae83

  Copy full SHA for 45aae83

• fix-sage-maker-test

  Show description for 285a9e1
  Future-Outlier

  committed
  285a9e1

  Copy full SHA for 285a9e1

• add test_slurm_fn_task

  Show description for 6b70901
  Future-Outlier

  committed
  6b70901

  Copy full SHA for 6b70901

• fix

  Show description for cf8eafa
  Future-Outlier

  committed
  cf8eafa

  Copy full SHA for cf8eafa

• update flytebot

  Show description for f0f7930
  Future-Outlier

  committed
  f0f7930

  Copy full SHA for f0f7930

• add know host = None

  Show description for d0d59d3
  Future-Outlier

  committed
  d0d59d3

  Copy full SHA for d0d59d3