Get poll to return task failure if job/log has been removed.

[wxtim]

Closes #6425

Note

Note to reviewers, you will need to deploy this branch onto remote platforms to confirm it works for remote filesystems.

Finally have a replicable example (Thank you @oliver-sanders)

[scheduling]
    cycling mode = integer
    [[graph]]
        R1 = task

[runtime]
    [[task]]
        script = """
            rm ${CYLC_WORKFLOW_RUN_DIR}/.service/contact
            rm -r "${CYLC_WORKFLOW_RUN_DIR}/log/job/${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
        """
        platform = _remote_pbs

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[wxtim]

Only test__job_poll_status_files_deleted_logdir is directly related to the PR. Other tests should increase coverage. 😄

[oliver-sanders]

LGTM.

[oliver-sanders]

Note to reviewers, you will need to deploy this branch onto remote platforms to confirm it works for remote filesystems.

[MetRonnie]

From the original issue:

If you delete the job log directory for an active task, Cylc will preserve its last known status indefinitely. I.e, Cylc will consider the job to be submitted/running forever.

This does not seem to be true. It is only true if the job log dir and the contact file is removed.

This situation should be handled similarly to the job no longer appearing in the queue, i.e, the job is dead, long live the job. Stick it into the failed/submit-failed state as appropriate.

The job can succeed even if its job log dir is removed from under its feet.

There seems to be a problem here where the job log retrieval process keeps retrying indefinitely, preventing shutdown without the --now --now option.

[runtime]
    [[task]]
        script = """
            rm -r "${CYLC_WORKFLOW_RUN_DIR}/log/job/${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
        """
        platform = <remote PBS>
        execution time limit = PT1M
    [[[directives]]]
        -q = shared
        -l ncpus = 1
        -l mem = 100mb