Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[4.1.1] ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process #694

Open
hswong3i opened this issue Dec 12, 2023 · 14 comments · May be fixed by #714

Comments

@hswong3i
Copy link

hswong3i commented Dec 12, 2023

Similar as pyca/cryptography#9016, my ceph-mgr in 18.2.1 get error message once upgrade from bcrypt 4.0.1 to 4.1.1:

root@node12:~# ceph health detail
HEALTH_WARN 9 mgr modules have failed dependencies
[WRN] MGR_MODULE_DEPENDENCY: 9 mgr modules have failed dependencies
   Module 'balancer' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'crash' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'devicehealth' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'orchestrator' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'pg_autoscaler' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'progress' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'rbd_support' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'telemetry' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
   Module 'volumes' has failed dependency: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

root@node12:~# journalctl -xef -u [email protected] 
Dec 12 03:41:13 node12 systemd[1]: Started Ceph cluster manager daemon.
░░ Subject: A start job for unit [email protected] has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit [email protected] has finished successfully.
░░ 
░░ The job identifier is 37868.
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.178+0000 7fc8359f5280 -1 mgr[py] Module status has missing NOTIFY_TYPES member
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.314+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.314+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.314+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'mirroring'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.314+0000 7fc8359f5280 -1 mgr[py] Error loading module 'mirroring': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.446+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.446+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.446+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'orchestrator'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.446+0000 7fc8359f5280 -1 mgr[py] Error loading module 'orchestrator': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.542+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.542+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.542+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'nfs'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.542+0000 7fc8359f5280 -1 mgr[py] Error loading module 'nfs': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.642+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.642+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.642+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'prometheus'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.642+0000 7fc8359f5280 -1 mgr[py] Error loading module 'prometheus': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.738+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.738+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.738+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'volumes'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.738+0000 7fc8359f5280 -1 mgr[py] Error loading module 'volumes': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.834+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.834+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.834+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'iostat'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.834+0000 7fc8359f5280 -1 mgr[py] Error loading module 'iostat': (22) Invalid argument
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.930+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.930+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:14 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:14 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:14 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.930+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'crash'
Dec 12 03:41:14 node12 ceph-mgr[3265527]: 2023-12-12T03:41:14.930+0000 7fc8359f5280 -1 mgr[py] Error loading module 'crash': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.026+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.026+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.026+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'devicehealth'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.026+0000 7fc8359f5280 -1 mgr[py] Error loading module 'devicehealth': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.122+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.122+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.122+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'osd_support'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.122+0000 7fc8359f5280 -1 mgr[py] Error loading module 'osd_support': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.218+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.218+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.218+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'restful'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.218+0000 7fc8359f5280 -1 mgr[py] Error loading module 'restful': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.318+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.318+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.318+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'localpool'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.318+0000 7fc8359f5280 -1 mgr[py] Error loading module 'localpool': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.418+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.418+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.418+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'pg_autoscaler'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.418+0000 7fc8359f5280 -1 mgr[py] Error loading module 'pg_autoscaler': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.518+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.518+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.518+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'zabbix'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.518+0000 7fc8359f5280 -1 mgr[py] Error loading module 'zabbix': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.618+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.618+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.622+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'snap_schedule'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.622+0000 7fc8359f5280 -1 mgr[py] Error loading module 'snap_schedule': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.718+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.718+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.718+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'test_orchestrator'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.722+0000 7fc8359f5280 -1 mgr[py] Error loading module 'test_orchestrator': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.838+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.838+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.838+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'stats'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.838+0000 7fc8359f5280 -1 mgr[py] Error loading module 'stats': (22) Invalid argument
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.950+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.950+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:15 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:15 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:15 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.950+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'progress'
Dec 12 03:41:15 node12 ceph-mgr[3265527]: 2023-12-12T03:41:15.950+0000 7fc8359f5280 -1 mgr[py] Error loading module 'progress': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.050+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.050+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.050+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'telemetry'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.050+0000 7fc8359f5280 -1 mgr[py] Error loading module 'telemetry': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.150+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.150+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.154+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'influx'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.154+0000 7fc8359f5280 -1 mgr[py] Error loading module 'influx': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.254+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.254+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.254+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'telegraf'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.254+0000 7fc8359f5280 -1 mgr[py] Error loading module 'telegraf': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.358+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.358+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.358+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'osd_perf_query'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.358+0000 7fc8359f5280 -1 mgr[py] Error loading module 'osd_perf_query': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.458+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.458+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.458+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'selftest'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.458+0000 7fc8359f5280 -1 mgr[py] Error loading module 'selftest': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.554+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.554+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.558+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'insights'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.558+0000 7fc8359f5280 -1 mgr[py] Error loading module 'insights': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.650+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.650+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.654+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'alerts'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.654+0000 7fc8359f5280 -1 mgr[py] Error loading module 'alerts': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.746+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.746+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.746+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'balancer'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.746+0000 7fc8359f5280 -1 mgr[py] Error loading module 'balancer': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.842+0000 7fc8359f5280 -1 mgr[py] Module not found: 'mgr_module'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.842+0000 7fc8359f5280 -1 mgr[py] Traceback (most recent call last):
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from mgr_util import profile_method
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     import bcrypt
Dec 12 03:41:16 node12 ceph-mgr[3265527]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 12 03:41:16 node12 ceph-mgr[3265527]:     from ._bcrypt import (
Dec 12 03:41:16 node12 ceph-mgr[3265527]: ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.842+0000 7fc8359f5280 -1 mgr[py] Class not found in module 'rbd_support'
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.842+0000 7fc8359f5280 -1 mgr[py] Error loading module 'rbd_support': (22) Invalid argument
Dec 12 03:41:16 node12 ceph-mgr[3265527]: 2023-12-12T03:41:16.842+0000 7fc8359f5280 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: mirroring, orchestrator, nfs, prometheus, volumes, iostat, crash, devicehealth, osd_support, restful, localpool, pg_autoscaler, zabbix, snap_schedule, test_orchestrator, stats, progress, telemetry, influx, telegraf, osd_perf_query, selftest, insights, alerts, balancer, rbd_support

This was initially introduced by:

  • 5505d32, bump pyo3 from 0.15.2 to 0.18.3
  • PyO3/pyo3@78ba70d, only allow each #[pymodule] to be initialized once, when pyo3 >= 0.17.0

My only workaround for now is, revert back to bcrypt == 4.0.1, which still depends on pyo3 = { version = "0.15.2" }

@alex
Copy link
Member

alex commented Dec 12, 2023

I'm not familiar with how ceph is implemented, is there an obvious reason why it'd be initializing modules multiple times per process? Does it use subinterpreters?

@hswong3i
Copy link
Author

hswong3i commented Dec 12, 2023

From my point of view, case should as similar as pyca/cryptography#9016 (comment).

BTW now I could combine both Ceph 18.2.1 + python3-cryptography 41.0.7 + python3-bcrypt 4.0.1 without error; the error only happen when running with bcrypt >= 4.1.0.

Therefore I guess bcrypt should reference some handling from cryptography between 41.0.1..41.0.7 for preventing multiple initialization?

@alex
Copy link
Member

alex commented Dec 12, 2023 via email

@hswong3i
Copy link
Author

hswong3i commented Dec 12, 2023

The case with Ceph 18.2.1 are:

  • OK: cryptography 40.0.2 + bcrypt 4.0.1
  • OK: cryptography 41.0.7 + bcrypt 4.0.1
  • FAILED: cryptography 40.0.2 + bcrypt 4.1.1
  • FAILED: cryptography 41.0.7 + bcrypt 4.1.1

Original report for Ceph 18.2.1 + cryptography 41.0.x could be found from:

BTW as above cross check shown, now cryptography no longer become the root cause, but coming from bcrypt ;-(

@alex
Copy link
Member

alex commented Dec 12, 2023

Is it straightforward for me to reproduce this locally, or do I need a full ceph installation?

@hswong3i
Copy link
Author

I have a Vagrant Box for Ceph 18.2: https://app.vagrantup.com/alvistack/boxes/ceph-18.2

It is now rebuilding with my latest functional combination (i.e. cryptography 41.0.7 + bcrypt 4.0.1): https://gitlab.com/alvistack/vagrant-ceph/-/pipelines/1103864469

If you are using VirtualBox (see https://github.com/alvistack/vagrant-ceph#quick-start):

# Initialize Vagrant
cat > Vagrantfile <<-EOF
Vagrant.configure('2') do |config|
  config.vm.hostname = 'ceph-18.2'
  config.vm.box = 'alvistack/ceph-18.2'

  config.vm.provider :virtualbox do |virtualbox|
    config.vm.disk :disk, name: 'sdb', size: '10GB'
    virtualbox.cpus = 2
    virtualbox.customize ['modifyvm', :id, '--cpu-profile', 'host']
    virtualbox.customize ['modifyvm', :id, '--nested-hw-virt', 'on']
    virtualbox.memory = 8192
  end
end
EOF

# Start the virtual machine
export VAGRANT_EXPERIMENTAL='1'
vagrant up

# SSH into this machine
vagrant ssh

Once the box is up and running:

  • vagrant ssh then sudo su - as root
  • Check with ceph -s, now should showing HEALTH_OK
  • Manually upgrade the preinstalled bcrypt 4.0.1 under /usr/lib/python3/dist-packages/bcrypt into 4.1.1
  • Reboot the box
  • Once login as root again and check with ceph -s, now it should show HEALTH_WARN 9 mgr modules have failed dependencies
  • With journalctl -xef -u ceph-mgr@* you should able to get the error log as show above

@alex
Copy link
Member

alex commented Dec 12, 2023

If I built a custom wheel, would you be able to test that out and see if you can reproduce?

@hswong3i
Copy link
Author

If I built a custom wheel, would you be able to test that out and see if you can reproduce?

Or you may provide your forked GitHub branch, so I could keep rebuilding and testing with it in my local dev env?

@alex
Copy link
Member

alex commented Dec 12, 2023

#695 -- you can either download a wheel from the wheel-builder job, or you can build yourself from that branch.

@hswong3i
Copy link
Author

hswong3i commented Dec 13, 2023

@alex my quick check and report:

For the last case, the new error message is now:

Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.339+0000 7f3d7e793280 -1 mgr[py] Module status has missing NOTIFY_TYPES member
Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.479+0000 7f3d7e793280 -1 mgr[py] Module not found: 'mgr_module'
Dec 13 06:41:15 node12 ceph-mgr[17264]: 2023-12-13T06:41:15.479+0000 7f3d7e793280 -1 mgr[py] Traceback (most recent call last):
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/usr/share/ceph/mgr/mgr_module.py", line 28, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     from mgr_util import profile_method
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/usr/share/ceph/mgr/mgr_util.py", line 6, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     import bcrypt
Dec 13 06:41:15 node12 ceph-mgr[17264]:   File "/lib/python3/dist-packages/bcrypt/__init__.py", line 13, in <module>
Dec 13 06:41:15 node12 ceph-mgr[17264]:     from ._bcrypt import (
Dec 13 06:41:15 node12 ceph-mgr[17264]: ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576

See PyO3/pyo3#576

@alex
Copy link
Member

alex commented Dec 13, 2023

Hmm, on the one hand, if ceph really is using subinterpreters, there's nothing we can do.

On the other hand, the fact that this error is showing up only with certain version combinations suggestions there's a deeper bug here. I'm afraid I don't have a suggestion other than that we really need a minimal reproducer here, something smaller than all of ceph.

hswong3i added a commit to alvistack/pyca-bcrypt that referenced this issue Jan 3, 2024
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or
older may only be initialized once per interpreter process` introduced
since pyo3 >= 0.17.0, which only allow each #[pymodule] to be
initialized once, this PR temporarily rollback to pyo3 = 0.16.6.

It also remove the use of `PyUserWarning` which introduced since pyo3 >
0.18.0.

See PyO3/pyo3@78ba70d
See PyO3/pyo3@1d20f2a

Fixes pyca#694

Signed-off-by: Wong Hoi Sing Edison <[email protected]>
hswong3i added a commit to alvistack/pyca-bcrypt that referenced this issue Jan 3, 2024
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or
older may only be initialized once per interpreter process` introduced
since pyo3 >= 0.17.0, which only allow each #[pymodule] to be
initialized once, this PR temporarily rollback to pyo3 = 0.16.6.

It also remove the use of `PyUserWarning` which introduced since pyo3 >
0.18.0.

See PyO3/pyo3@78ba70d
See PyO3/pyo3@1d20f2a

Fixes pyca#694

Signed-off-by: Wong Hoi Sing Edison <[email protected]>
hswong3i added a commit to alvistack/pyca-bcrypt that referenced this issue Jan 3, 2024
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or
older may only be initialized once per interpreter process` introduced
since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be
initialized once, this PR temporarily rollback to pyo3 = 0.16.6.

It also remove the use of `PyUserWarning` which introduced since pyo3 >
0.18.0.

See PyO3/pyo3@78ba70d
See PyO3/pyo3@1d20f2a

Fixes pyca#694

Signed-off-by: Wong Hoi Sing Edison <[email protected]>
hswong3i added a commit to alvistack/pyca-bcrypt that referenced this issue Jan 3, 2024
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or
older may only be initialized once per interpreter process` introduced
since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be
initialized once, this PR temporarily rollback to pyo3 = 0.16.6.

It also remove the use of `PyUserWarning` which introduced since pyo3 >
0.18.0.

See PyO3/pyo3@78ba70d
See PyO3/pyo3@1d20f2a

Fixes pyca#694

Signed-off-by: Wong Hoi Sing Edison <[email protected]>
hswong3i added a commit to alvistack/pyca-bcrypt that referenced this issue May 4, 2024
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or
older may only be initialized once per interpreter process` introduced
since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be
initialized once, this PR temporarily rollback to pyo3 = 0.16.6.

It also remove the use of `PyUserWarning` which introduced since pyo3 >
0.18.0.

See PyO3/pyo3@78ba70d
See PyO3/pyo3@1d20f2a

Fixes pyca#694

Signed-off-by: Wong Hoi Sing Edison <[email protected]>
@marcan
Copy link

marcan commented May 14, 2024

Ceph really does use subinterpreters. It also used to work on my system (F39) after a patch to remove cryptography as a dependency, but now it's failing again with a larger update.

My suspicion is the check for multiple instances itself is/was somehow flawed and doesn't always go off (or used not to), but Ceph has always been using subinterpretrs.

Ceph bug: https://tracker.ceph.com/issues/63529
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=2255688

@bazaah
Copy link

bazaah commented May 22, 2024

Yes. I have, in effect, vendored a rebuild of python-bcrypt with:

diff --git a/src/_bcrypt/Cargo.toml b/src/_bcrypt/Cargo.toml
index a9c7f7c..02317c8 100644
--- a/src/_bcrypt/Cargo.toml
+++ b/src/_bcrypt/Cargo.toml
@@ -6,7 +6,7 @@ edition = "2018"
 publish = false

 [dependencies]
-pyo3 = { version = "0.20.0", features = ["abi3"] }
+pyo3 = { git = "https://git.st8l.com/luxolus/pyo3", tag = "v0.20.3-subint+1", features = ["abi3", "unsafe-allow-subinterpreters"] }
 bcrypt = "0.15"
 bcrypt-pbkdf = "0.10.0"
 base64 = "0.21.5"

https://git.st8l.com/luxolus/pyo3/commit/338c71d0ad10f7ae38b7b44e576d49b91ed20d99

Which is possible because this python module doesn't store Py objects in Rust statics. The dashboard remains broken, because other packages still use pyO3 modules, but the mgr's core functionality is operational.

@Aequitosh
Copy link

For reference, in case anyone's stumbling over this issue, adding sub-interpreter support in PyO3 is being worked on over here: PyO3/pyo3#3451

The Ceph folks that commented here (hi again!) are already aware of this; just wanted to leave some breadcrumbs for others to follow in case anyone's interested in helping out as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

5 participants