Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable abi3 wheel tag for PyPy #626

Merged
merged 1 commit into from
Aug 17, 2024

Conversation

lpsinger
Copy link
Contributor

@lpsinger lpsinger commented May 3, 2024

Pip does not support installing PyPy wheels with the abi3 tag. If you build a wheel for PyPy and set the ABI tag to abi3, pip will not be able to install it.

$ docker run --rm -it quay.io/pypa/manylinux2014_x86_64
[root@cd7b2d465170 /]# /opt/python/pp39-pypy39_pp73/bin/python
Python 3.9.19 (a2113ea87262, Apr 21 2024, 05:40:24)
[PyPy 7.3.16 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> from pip._internal.utils.compatibility_tags import get_supported
>>>> for tag in get_supported():
....     print(tag)
....
pp39-pypy39_pp73-manylinux_2_17_x86_64
pp39-pypy39_pp73-manylinux2014_x86_64

Full output:

>>>> from pip._internal.utils.compatibility_tags import get_supported
>>>> for tag in get_supported():
....     print(tag)
....
pp39-pypy39_pp73-manylinux_2_17_x86_64
pp39-pypy39_pp73-manylinux2014_x86_64
pp39-pypy39_pp73-manylinux_2_16_x86_64
pp39-pypy39_pp73-manylinux_2_15_x86_64
pp39-pypy39_pp73-manylinux_2_14_x86_64
pp39-pypy39_pp73-manylinux_2_13_x86_64
pp39-pypy39_pp73-manylinux_2_12_x86_64
pp39-pypy39_pp73-manylinux2010_x86_64
pp39-pypy39_pp73-manylinux_2_11_x86_64
pp39-pypy39_pp73-manylinux_2_10_x86_64
pp39-pypy39_pp73-manylinux_2_9_x86_64
pp39-pypy39_pp73-manylinux_2_8_x86_64
pp39-pypy39_pp73-manylinux_2_7_x86_64
pp39-pypy39_pp73-manylinux_2_6_x86_64
pp39-pypy39_pp73-manylinux_2_5_x86_64
pp39-pypy39_pp73-manylinux1_x86_64
pp39-pypy39_pp73-linux_x86_64
pp39-none-manylinux_2_17_x86_64
pp39-none-manylinux2014_x86_64
pp39-none-manylinux_2_16_x86_64
pp39-none-manylinux_2_15_x86_64
pp39-none-manylinux_2_14_x86_64
pp39-none-manylinux_2_13_x86_64
pp39-none-manylinux_2_12_x86_64
pp39-none-manylinux2010_x86_64
pp39-none-manylinux_2_11_x86_64
pp39-none-manylinux_2_10_x86_64
pp39-none-manylinux_2_9_x86_64
pp39-none-manylinux_2_8_x86_64
pp39-none-manylinux_2_7_x86_64
pp39-none-manylinux_2_6_x86_64
pp39-none-manylinux_2_5_x86_64
pp39-none-manylinux1_x86_64
pp39-none-linux_x86_64
py39-none-manylinux_2_17_x86_64
py39-none-manylinux2014_x86_64
py39-none-manylinux_2_16_x86_64
py39-none-manylinux_2_15_x86_64
py39-none-manylinux_2_14_x86_64
py39-none-manylinux_2_13_x86_64
py39-none-manylinux_2_12_x86_64
py39-none-manylinux2010_x86_64
py39-none-manylinux_2_11_x86_64
py39-none-manylinux_2_10_x86_64
py39-none-manylinux_2_9_x86_64
py39-none-manylinux_2_8_x86_64
py39-none-manylinux_2_7_x86_64
py39-none-manylinux_2_6_x86_64
py39-none-manylinux_2_5_x86_64
py39-none-manylinux1_x86_64
py39-none-linux_x86_64
py3-none-manylinux_2_17_x86_64
py3-none-manylinux2014_x86_64
py3-none-manylinux_2_16_x86_64
py3-none-manylinux_2_15_x86_64
py3-none-manylinux_2_14_x86_64
py3-none-manylinux_2_13_x86_64
py3-none-manylinux_2_12_x86_64
py3-none-manylinux2010_x86_64
py3-none-manylinux_2_11_x86_64
py3-none-manylinux_2_10_x86_64
py3-none-manylinux_2_9_x86_64
py3-none-manylinux_2_8_x86_64
py3-none-manylinux_2_7_x86_64
py3-none-manylinux_2_6_x86_64
py3-none-manylinux_2_5_x86_64
py3-none-manylinux1_x86_64
py3-none-linux_x86_64
py38-none-manylinux_2_17_x86_64
py38-none-manylinux2014_x86_64
py38-none-manylinux_2_16_x86_64
py38-none-manylinux_2_15_x86_64
py38-none-manylinux_2_14_x86_64
py38-none-manylinux_2_13_x86_64
py38-none-manylinux_2_12_x86_64
py38-none-manylinux2010_x86_64
py38-none-manylinux_2_11_x86_64
py38-none-manylinux_2_10_x86_64
py38-none-manylinux_2_9_x86_64
py38-none-manylinux_2_8_x86_64
py38-none-manylinux_2_7_x86_64
py38-none-manylinux_2_6_x86_64
py38-none-manylinux_2_5_x86_64
py38-none-manylinux1_x86_64
py38-none-linux_x86_64
py37-none-manylinux_2_17_x86_64
py37-none-manylinux2014_x86_64
py37-none-manylinux_2_16_x86_64
py37-none-manylinux_2_15_x86_64
py37-none-manylinux_2_14_x86_64
py37-none-manylinux_2_13_x86_64
py37-none-manylinux_2_12_x86_64
py37-none-manylinux2010_x86_64
py37-none-manylinux_2_11_x86_64
py37-none-manylinux_2_10_x86_64
py37-none-manylinux_2_9_x86_64
py37-none-manylinux_2_8_x86_64
py37-none-manylinux_2_7_x86_64
py37-none-manylinux_2_6_x86_64
py37-none-manylinux_2_5_x86_64
py37-none-manylinux1_x86_64
py37-none-linux_x86_64
py36-none-manylinux_2_17_x86_64
py36-none-manylinux2014_x86_64
py36-none-manylinux_2_16_x86_64
py36-none-manylinux_2_15_x86_64
py36-none-manylinux_2_14_x86_64
py36-none-manylinux_2_13_x86_64
py36-none-manylinux_2_12_x86_64
py36-none-manylinux2010_x86_64
py36-none-manylinux_2_11_x86_64
py36-none-manylinux_2_10_x86_64
py36-none-manylinux_2_9_x86_64
py36-none-manylinux_2_8_x86_64
py36-none-manylinux_2_7_x86_64
py36-none-manylinux_2_6_x86_64
py36-none-manylinux_2_5_x86_64
py36-none-manylinux1_x86_64
py36-none-linux_x86_64
py35-none-manylinux_2_17_x86_64
py35-none-manylinux2014_x86_64
py35-none-manylinux_2_16_x86_64
py35-none-manylinux_2_15_x86_64
py35-none-manylinux_2_14_x86_64
py35-none-manylinux_2_13_x86_64
py35-none-manylinux_2_12_x86_64
py35-none-manylinux2010_x86_64
py35-none-manylinux_2_11_x86_64
py35-none-manylinux_2_10_x86_64
py35-none-manylinux_2_9_x86_64
py35-none-manylinux_2_8_x86_64
py35-none-manylinux_2_7_x86_64
py35-none-manylinux_2_6_x86_64
py35-none-manylinux_2_5_x86_64
py35-none-manylinux1_x86_64
py35-none-linux_x86_64
py34-none-manylinux_2_17_x86_64
py34-none-manylinux2014_x86_64
py34-none-manylinux_2_16_x86_64
py34-none-manylinux_2_15_x86_64
py34-none-manylinux_2_14_x86_64
py34-none-manylinux_2_13_x86_64
py34-none-manylinux_2_12_x86_64
py34-none-manylinux2010_x86_64
py34-none-manylinux_2_11_x86_64
py34-none-manylinux_2_10_x86_64
py34-none-manylinux_2_9_x86_64
py34-none-manylinux_2_8_x86_64
py34-none-manylinux_2_7_x86_64
py34-none-manylinux_2_6_x86_64
py34-none-manylinux_2_5_x86_64
py34-none-manylinux1_x86_64
py34-none-linux_x86_64
py33-none-manylinux_2_17_x86_64
py33-none-manylinux2014_x86_64
py33-none-manylinux_2_16_x86_64
py33-none-manylinux_2_15_x86_64
py33-none-manylinux_2_14_x86_64
py33-none-manylinux_2_13_x86_64
py33-none-manylinux_2_12_x86_64
py33-none-manylinux2010_x86_64
py33-none-manylinux_2_11_x86_64
py33-none-manylinux_2_10_x86_64
py33-none-manylinux_2_9_x86_64
py33-none-manylinux_2_8_x86_64
py33-none-manylinux_2_7_x86_64
py33-none-manylinux_2_6_x86_64
py33-none-manylinux_2_5_x86_64
py33-none-manylinux1_x86_64
py33-none-linux_x86_64
py32-none-manylinux_2_17_x86_64
py32-none-manylinux2014_x86_64
py32-none-manylinux_2_16_x86_64
py32-none-manylinux_2_15_x86_64
py32-none-manylinux_2_14_x86_64
py32-none-manylinux_2_13_x86_64
py32-none-manylinux_2_12_x86_64
py32-none-manylinux2010_x86_64
py32-none-manylinux_2_11_x86_64
py32-none-manylinux_2_10_x86_64
py32-none-manylinux_2_9_x86_64
py32-none-manylinux_2_8_x86_64
py32-none-manylinux_2_7_x86_64
py32-none-manylinux_2_6_x86_64
py32-none-manylinux_2_5_x86_64
py32-none-manylinux1_x86_64
py32-none-linux_x86_64
py31-none-manylinux_2_17_x86_64
py31-none-manylinux2014_x86_64
py31-none-manylinux_2_16_x86_64
py31-none-manylinux_2_15_x86_64
py31-none-manylinux_2_14_x86_64
py31-none-manylinux_2_13_x86_64
py31-none-manylinux_2_12_x86_64
py31-none-manylinux2010_x86_64
py31-none-manylinux_2_11_x86_64
py31-none-manylinux_2_10_x86_64
py31-none-manylinux_2_9_x86_64
py31-none-manylinux_2_8_x86_64
py31-none-manylinux_2_7_x86_64
py31-none-manylinux_2_6_x86_64
py31-none-manylinux_2_5_x86_64
py31-none-manylinux1_x86_64
py31-none-linux_x86_64
py30-none-manylinux_2_17_x86_64
py30-none-manylinux2014_x86_64
py30-none-manylinux_2_16_x86_64
py30-none-manylinux_2_15_x86_64
py30-none-manylinux_2_14_x86_64
py30-none-manylinux_2_13_x86_64
py30-none-manylinux_2_12_x86_64
py30-none-manylinux2010_x86_64
py30-none-manylinux_2_11_x86_64
py30-none-manylinux_2_10_x86_64
py30-none-manylinux_2_9_x86_64
py30-none-manylinux_2_8_x86_64
py30-none-manylinux_2_7_x86_64
py30-none-manylinux_2_6_x86_64
py30-none-manylinux_2_5_x86_64
py30-none-manylinux1_x86_64
py30-none-linux_x86_64
pp39-none-any
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

@lpsinger lpsinger force-pushed the pypy-no-abi3-tag branch from ec72108 to 228d3e9 Compare May 3, 2024 21:25
@rgommers rgommers added the maintenance Regular code improvements that are not new features nor end-user-visible bugs label May 10, 2024
@rgommers
Copy link
Contributor

Thanks for working on this @lpsinger. The current behavior certainly doesn't seem quite right - we should never be producing something with a tag that pip won't understand.

Related to gh-624. If I understand correctly, with the build option discussed in gh-624 you can disable asking for limited API usage, which will then produce a tag like pp310-pypy310_pp73-linux_x86_64 that can be installed. This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like pp310-pypy310_pp73-linux_x86_64. Is that right? If so, that seems reasonable to me, since it reduced friction and the end result should be the same as erroring out and forcing the user to specify -C-Dpython.allow_limited_api=false.

CI is a little unhappy, the tests need updating to match:

FAILED tests/test_tags.py::test_tag_stable_abi - AssertionError: assert 'pp310-pypy310_pp73-linux_x86_64' == 'pp310-abi3-linux_x86_64'
  
  - pp310-abi3-linux_x86_64
  + pp310-pypy310_pp73-linux_x86_64
FAILED tests/test_wheel.py::test_limited_api - AssertionError: assert 'pypy310_pp73' == 'abi3'

@lpsinger
Copy link
Contributor Author

This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like pp310-pypy310_pp73-linux_x86_64. Is that right?

That's correct. I'm looking to avoid having to add this line to projects that use the limited API and cibuildwheel: https://github.com/nasa-gcn/hpx/blob/main/pyproject.toml#L81-L84

@lpsinger lpsinger force-pushed the pypy-no-abi3-tag branch from 228d3e9 to 0e1e1c6 Compare May 10, 2024 14:36
@lpsinger
Copy link
Contributor Author

CI is a little unhappy, the tests need updating to match:

I pushed a change for that; let's see if it fixes the CI.

@dnicolodi
Copy link
Member

My understanding is that PyPy supports the limited API but does not use a dedicated filename suffix for extension modules compiled for the limited API. The Python support in Meson implies the same understanding https://github.com/mesonbuild/meson/blob/cfd57180eef9036c7167c5682b9f3055a540fccc/mesonbuild/scripts/python_info.py#L103-L107

What I am not sure about is whether, when using the limited API, PyPy implements a stable ABI, and thus the resulting wheels should be taggeg abi3 or not. If this is the case, the problem being solved here is a bug in pip and not something we need to work-around in meson-python.

@mattip Can you confirm that my understanding is correct?

I added the tool.meson-python.limited-api setting based on the assumption that PyPy supports the limited API without a dedicated module filename suffix and thus it is not sufficient to look at the filenames of the installed extension modules to determine whether a package uses the limited API only. If the assumption is wrong and PyPy does not support the limited API or supports it without exporting a stable ABI, the setting could in principle be removed.

@mattip
Copy link

mattip commented May 14, 2024

PyPy does not implement a stable ABI. Wheels for PyPy should not be tagged abi3. The output of pip's get_supported is correct.

@dnicolodi
Copy link
Member

Thanks for the clarification @mattip. My reasoning for implementing the tool.meson-python.limited-api setting is wrong. I think we should remove this setting and rely on the filename suffix of the installed extension modules to decide the ABI tag for the created wheel. We would still accept the parameter for backward compatibility but emit a deprecation warning when we encounter it. @rgommers, what do you think?

@dnicolodi
Copy link
Member

One of the reasons to remove the tool.meson-python.limited-api setting is that it is misnamed. When I introduced it, I thought that using the limited API always results in a stable ABI, and thus in wheels to be tagged with the abi3 tag. However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea. To my defense, the CPython documentation does not make this distinction at all.

@henryiii
Copy link
Contributor

henryiii commented May 14, 2024

You can’t do that on windows. No suffix there.

In scikit-build-core, I use wheel.py-api, which is the wheel tag and you can get both limited api and pythonless wheels from that, see https://scikit-build-core.readthedocs.io/en/latest/configuration.html#customizing-the-output-wheel.

Setting something like cp311 is ignored on CPython < 3.11, PyPy, and free-threaded Python.

@dnicolodi
Copy link
Member

IIRC, on windows there is no suffix specific to abi3 extension modules but there is a Python ABI specific suffix and the general .pyd suffix. I was thinking of mapping the .pyd suffix to the abi3 wheel tag. This is not fool proof, but I don't expect it to be a problem in practice. I need to go back and check my reasoning from when I implemented the tool.meson-python.limited-api setting.

@dnicolodi
Copy link
Member

@henryiii Thanks for the pointer to how scikit-build-core solves this. However, it is not clear to me how you handle the case when the user requires the stable ABI tag abi3 but the package is built for PyPy. Do you simply override the user choice in this case?

@rgommers
Copy link
Contributor

My impression is that the current build option is fine, since it works as designed for CPython. For PyPy, proceeding with the regular minor-version-specific tag as done in this PR would seem reasonable to me. The question I have is whether we should be emitting a warning when doing that.

However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea.

PyPy could implement it in the future in principle (it's just a missing feature now), so I wouldn't attach too much value to this since conceptually it does make sense.

Moreover, PyPy is probably going to fade away pretty quickly over the next 1-2 years - there is unlikely to be a 3.11 release for PyPy, so next year when Python 3.10 starts being dropped that's the end of it. Conda-forge is also starting to drop PyPy builds because of maintenance issues and no real future. Hence deprecating because of PyPy isn't needed imho.

@henryiii
Copy link
Contributor

The way selecting cp311 works is that it's only applicable on CPython with GIL 3.11+. Any older version or other interpreter (including PyPy) ignores this choice. (It wasn't quite right for PyPy until the most recent release, when I was working on free-threading I cleaned it up a little). A user is expected to skip requesting it CMake like this:

if(NOT "${SKBUILD_SABI_COMPONENT}" STREQUAL "")
  python_add_library(some_ext MODULE WITH_SOABI USE_SABI 3.11 ...)
else()
  python_add_library(some_ext MODULE WITH_SOABI ...)
endif()

(USE_SABI also sets the Limited API define) Though I haven't checked to see exactly how CMake's FindPython module handles it, it might do the right thing (IMO which is to ignore this setting if it doesn't apply).

Not setting the define just means you are not limited to the Limited API, so shouldn't hurt user code.

@lpsinger
Copy link
Contributor Author

Moreover, PyPy is probably going to fade away pretty quickly over the next 1-2 years - there is unlikely to be a 3.11 release for PyPy, so next year when Python 3.10 starts being dropped that's the end of it. Conda-forge is also starting to drop PyPy builds because of maintenance issues and no real future. Hence deprecating because of PyPy isn't needed imho.

Does that mean that this is a "won't fix?"

@rgommers
Copy link
Contributor

Does that mean that this is a "won't fix?"

I don't think so. This is a minor change that to me seems strictly an improvement over the status quo. So I'd be in favor of merging it as is.

The larger change @dnicolodi was thinking about (this comment above) is orthogonal, and may or may not happen. I'd suggest that it shouldn't be blocking for this PR.

lpsinger added a commit to lpsinger/astropy-healpix that referenced this pull request Jul 15, 2024
PyPy is probably not long for this world; see
mesonbuild/meson-python#626 (comment).

Fixes astropy#216.
lpsinger added a commit to astropy/astropy-healpix that referenced this pull request Jul 15, 2024
PyPy is probably not long for this world; see
mesonbuild/meson-python#626 (comment).

Fixes #216.
@dnicolodi dnicolodi force-pushed the pypy-no-abi3-tag branch 2 times, most recently from 50550f1 to d192fe0 Compare August 17, 2024 09:42
PyPy supports the limited API but not the stable ABI.
@dnicolodi dnicolodi merged commit e5a180c into mesonbuild:main Aug 17, 2024
39 checks passed
@lpsinger lpsinger deleted the pypy-no-abi3-tag branch August 19, 2024 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Regular code improvements that are not new features nor end-user-visible bugs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants