Failures when testing pegasus v5.1.0 release

[spxiwh]

Hi all,

We have installed pegasus v5.1.0 on one of the LIGO condor submission machines but are running into problems when using it to submit a workflow. The errors occur when using pegasus-wms.api to submit workflows from within the python code that creates them. If I try to launch a workflow as I would for v5.0.9 I hit the following failure:

2025.03.04 05:35:05.211 PST: [ERROR]  /usr/lib64/pegasus/python/sqlalchemy/orm/query.py:196: SyntaxWarning: "is not" with a literal. Did you mean "!="?
2025.03.04 05:35:05.217 PST: [ERROR]    if entities is not ():
2025.03.04 05:35:05.331 PST: [ERROR]  Traceback (most recent call last):
2025.03.04 05:35:05.336 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/cli/pegasus-db-admin.py", line 21, in <module>
2025.03.04 05:35:05.341 PST: [ERROR]      from Pegasus.db.admin import main
2025.03.04 05:35:05.346 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/db/admin/__init__.py", line 1, in <module>
2025.03.04 05:35:05.352 PST: [ERROR]      from Pegasus.db.admin.admin_loader import *
2025.03.04 05:35:05.357 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/db/admin/admin_loader.py", line 31, in <module>
2025.03.04 05:35:05.362 PST: [ERROR]      from Pegasus.db import connection
2025.03.04 05:35:05.367 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/db/connection.py", line 31, in <module>
2025.03.04 05:35:05.372 PST: [ERROR]      from Pegasus.tools import properties, utils
2025.03.04 05:35:05.377 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/tools/utils.py", line 36, in <module>
2025.03.04 05:35:05.382 PST: [ERROR]      from Pegasus import braindump
2025.03.04 05:35:05.388 PST: [ERROR]    File "/usr/lib64/pegasus/python/Pegasus/braindump.py", line 32, in <module>
2025.03.04 05:35:05.393 PST: [ERROR]      class Braindump:
2025.03.04 05:35:05.398 PST: [ERROR]    File "/usr/lib64/pegasus/externals/python/dataclasses.py", line 950, in wrap
2025.03.04 05:35:05.403 PST: [ERROR]      return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
2025.03.04 05:35:05.408 PST: [ERROR]    File "/usr/lib64/pegasus/externals/python/dataclasses.py", line 800, in _process_class
2025.03.04 05:35:05.413 PST: [ERROR]      cls_fields = [_get_field(cls, name, type)
2025.03.04 05:35:05.418 PST: [ERROR]    File "/usr/lib64/pegasus/externals/python/dataclasses.py", line 800, in <listcomp>
2025.03.04 05:35:05.424 PST: [ERROR]      cls_fields = [_get_field(cls, name, type)
2025.03.04 05:35:05.429 PST: [ERROR]    File "/usr/lib64/pegasus/externals/python/dataclasses.py", line 659, in _get_field
2025.03.04 05:35:05.434 PST: [ERROR]      if (_is_classvar(a_type, typing)
2025.03.04 05:35:05.439 PST: [ERROR]    File "/usr/lib64/pegasus/externals/python/dataclasses.py", line 550, in _is_classvar
2025.03.04 05:35:05.444 PST: [ERROR]      return type(a_type) is typing._ClassVar
2025.03.04 05:35:05.449 PST: [ERROR]  AttributeError: module 'typing' has no attribute '_ClassVar'. Did you mean: 'ClassVar'?
2025.03.04 05:35:05.455 PST: [FATAL ERROR]  java.lang.RuntimeException: /usr/bin/pegasus-db-admin update -t master -c /local/ian.harry/pycbc-tmp_ffiarxwh/work/pegasus.4396297059204328463.properties failed with status 1 . Pegasus was unable to update the the worflow database file found at ~/.pegasus/workflow.db . If this file is corrupted, a solution for problem is to remove the file with the command: rm -f ~/.pegasus/workflow.db  - but note that doing so will remove old workflows from the Pegasus Dashboard.
at edu.isi.pegasus.planner.common.PegasusDBAdmin.checkDatabase(PegasusDBAdmin.java:262)
at edu.isi.pegasus.planner.common.PegasusDBAdmin.checkMasterDatabaseForVersionCompatibility(PegasusDBAdmin.java:142)
at edu.isi.pegasus.planner.client.CPlanner.checkMasterDatabaseForVersionCompatibility(CPlanner.java:1961)
at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:657)
at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:333)
at edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:207)

2025.03.04 05:35:05.918 PST: [INFO] event.pegasus.planner planner.version 5.1.0dev  (2.472 seconds) - FINISHED
.
.
.
    raise PegasusClientError("Pegasus command: {} FAILED".format(cmd), result)
Pegasus.client._client.PegasusClientError: Pegasus command: ['/usr/bin/pegasus-plan', '--conf', '/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/pycbc/workflow/pegasus_files/pegasus-properties.conf', '--cluster', 'label,horizontal', '--sites', 'condorpool_shared,condorpool_symlink,osg,local', '--output-sites', 'local', '--staging-site', 'condorpool_shared=condorpool_shared,condorpool_symlink=local,osg=local,local=local', '--dir', '/local/ian.harry/pycbc-tmp_ffiarxwh', '--relative-dir', 'work', '--cleanup', 'inplace', '-q', '--submit', 'gw.dax', '--json'] FAILED

This appears to be a python version error. I am using python 3.10, but the code in /usr/lib64/pegasus/externals appears to be python3.6. I'm not sure why this doesn't fail using pegasus 5.0.9.

If I use export PEGASUS_UPDATE_PYTHONPATH=0 to try to avoid using any code outside my environment I see:

[WARNING]  Traceback (most recent call last):
[WARNING]    File "/usr/lib64/pegasus/python/Pegasus/cli/pegasus-run.py", line 322, in <module>
[WARNING]      pegasus_run()
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
[WARNING]      return self.main(*args, **kwargs)
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/click/core.py", line 1082, in main
[WARNING]      rv = self.invoke(ctx)
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
[WARNING]      return ctx.invoke(self.callback, **ctx.params)
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/click/core.py", line 788, in invoke
[WARNING]      return __callback(*args, **kwargs)
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
[WARNING]      return f(get_current_context(), *args, **kwargs)
[WARNING]    File "/usr/lib64/pegasus/python/Pegasus/cli/pegasus-run.py", line 196, in pegasus_run
[WARNING]      config = slurp_braindb(submit_dir)
[WARNING]    File "/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/Pegasus/tools/utils.py", line 452, in slurp_braindb
[WARNING]      k: str(v) if isinstance(v, Path) else v for k, v in asdict(cfg).items()
[WARNING]    File "/cvmfs/software.igwn.org/conda/envs/igwn/lib/python3.10/dataclasses.py", line 1237, in asdict
[WARNING]      raise TypeError("asdict() should be called on dataclass instances")
[WARNING]  TypeError: asdict() should be called on dataclass instances
2025.03.04 05:37:42.624 PST: [FATAL ERROR]  java.lang.RuntimeException: Unable to submit the workflow using pegasus-run
at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:672)
at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:333)
at edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:207)

2025.03.04 05:37:43.001 PST: [INFO] event.pegasus.planner planner.version 5.1.0dev  (3.872 seconds) - FINISHED
.
.
.
Pegasus.client._client.PegasusClientError: Pegasus command: ['/usr/bin/pegasus-plan', '--conf', '/home/ian.harry/virtualenvs/pycbc_bbh_prec_test_NEWPEGASUS/lib/python3.10/site-packages/pycbc/workflow/pegasus_files/pegasus-properties.conf', '--cluster', 'label,horizontal', '--sites', 'local,condorpool_symlink,osg,condorpool_shared', '--output-sites', 'local', '--staging-site', 'local=local,condorpool_symlink=local,osg=local,condorpool_shared=condorpool_shared', '--dir', '/local/ian.harry/pycbc-tmp_imuw29x8', '--relative-dir', 'work', '--cleanup', 'inplace', '-q', '--submit', 'gw.dax', '--json'] FAILED

[spxiwh]

@mayani Thanks for the patch. It doesn't seem to address the second problem here though (TypeError: asdict() should be called on dataclass instances) ... I don't have permission to reopen this.

[mayani]

@spxiwh Can you verify the braindump.yml in the submit directory is valid? If yes, please send me the submit dir. as I am unable to recreate the issue locally.